Move BucketWriter and BucketReader to storage/bucket.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Move code around and add new directories for cloud backend merge.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add dependency on our fork of txAWS (0.2.1.post5).
Add 'six' to ignorable package list because it is a dependency of txAWS via python-dateutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

.gitignore: changes to facilitate cloud backend merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add documentation for each storage backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add test_leasedb.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb: use a semantic primary key (storage_index, shnum, account_id), rather than an integer, for the leases table.
Take advantage of this to simplify add_or_renew_leases.
Fix a bug in add_starter_lease (which is not used yet) when a starter lease already exists.
Clean up leftover accesses to self._dirty.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Rename 'buckets' to 'sharesets' on storage status page.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove unused files storage/lease.py and storage/expirer.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Main leasedb changes (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove the 'original-*' and 'configured-*' lease crawler output that won't be supported by the accounting crawler.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cosmetics.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: ss -> server for cases that will remain a server after the server/account split.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Asyncify crawlers. Note that this breaks tests for the LeaseCrawler
(which is going away, to be replaced by the AccountingCrawler).

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

scripts/debug.py: remove display of lease information and secrets.
This version replaces the expiration field with '-' instead of '0', per Zooko's comments.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove the [storage]expire.{mutable,immutable} options.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove support for [storage]debug_discard option.
(BucketWriter.throw_out_all_data is kept because it might still be useful.)

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add new files for leasedb.

Authors: Brian Warner <warner@lothar.com> and David-Sarah Hopwood
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to specification of add_lease and renew_lease in RIStorageServer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/garbage-collection.rst: update text for leasedb changes.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Use "PRAGMA synchronous = OFF" for dbutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add util/dbutil.py: open/create/update sqlite databases given some schema.

Author: Brian Warner <warner@lothar.com>
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

util/fileutil.py: add get_used_space. This version does not use FilePath.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Some useful Deferred utilities. Addresses Zooko's review comments.
HookMixin and WaitForDelayedCallsMixin have been added for leasedb,
the rest are from the cloud backend branch, with minor improvements.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

misc/simulators/hashbasedsig.py: revert last patch; the original code was correct.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: fix minor miscalculation bug.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: the cost of labels was being double-counted.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: fix an incorrect comment.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Use bigger random one-time keys in timing_safe_compare. fixes #2165

Follows this advice from Marsh Ray and Solar Designer: https://twitter.com/zooko/status/431105294777597952

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Rename 'constant_time_compare' to 'timing_safe_compare'. refs #2165

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #85 from leif/isolate-keygen-test

Isolate test_keygen in its own base directory. This also causes KeyGeneratorService to create its base directory if it does not exist.

isolate test_keygen data

This makes test_keygen use its own basedir, to avoid polluting the _trial_temp
directory for later tests.

Merge pull request #84 from kpreid/patch-1

Change kpreid's email address.

Change kpreid's email address.

Merge pull request #82 from multikatt/typo

Small typo: "stil" -> "still"

Small typo: "stil" -> "still"

Merge pull request #81 from multikatt/right_align

Right-align the services div on the WUI Welcome page, to address overlap. refs #1931

Right aligning services div

docs/about.rst: Reword and remove redundancy. Also avoid "(S)FTP".

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #73 from pataquets/patch-1

Warn about ambiguities on "client" and "gateway" terms

Warn about ambiguities on "client" and "gateway" terms

Merge pull request #69 from zooko/1948-fix-warnings-from-make-doc-checks-3

1948 fix warnings from make doc checks 3

change more links to footnote-style

update link to Maildir tips (thanks MK_FG)

fix warnings from rst2html

Apparently the in-line link syntax with "<>" in them causes these warnings. I
don't know why. I changed them all to a slightly more verbose syntax.

Thanks to Mike Kazantzsev's review comment
(https://github.com/tahoe-lafs/tahoe-lafs/pull/67#commitcomment-4561370), I
moved the links to the end of each section.

magic first line tells emacs to use utf8+bom

Add ".. -*- coding: utf-8-with-signature -*-" to the first line of each .rst
file. This tells emacs to treat the file contents as utf-8, and also to prepend
a so-called utf-8 "bom" marker at the beginning of the file. This patch also
prepends those markers to each of those files.

Fix typo pointed out by CptPlastic.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Makefile: update comment for 'make clean'.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #64 from daira/2092-distclean

Add 'make distclean'.

Add 'make distclean'.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #63 from zooko/2092-make-clean-shouldnt-remove-allmydata-tahoe-egg-info

remove tahoe egg-info dir from the "clean" rule

remove tahoe egg-info dir from the "clean" rule

fixes #2092

Add quotes around command lines in quickstart.rst.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #62 from zooko/master

fix a couple more "file:" links

Merge pull request #61 from zooko/fix-small-formatting-issues-in-docs

fix small issues in formatting

fix a couple more "file:" links

Making them be just links to "$FILENAME" instead of to "file:$FILENAME" causes
them to be working links on trac and on github. Yay!

fix small issues in formatting

* fix warning from "rst2html --verbose" by not using the in-line hyperlink
  syntax. The warning said::

    known_issues.rst:261: (INFO/1) Hyperlink target "google chart api" is not referenced.

* remove dead code (no-longer-referenced link), thus fixing another warning

* change links to relative files to be just $FILENAME instead of
  file:$FILENAME, which I think will fix the problem of broken links on the
  trac and github, e.g. the link in
  https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/known_issues.rst ,
  which ought to go to
  https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/cautions.rst , but
  instead goes to file:///cautions.rst

misc/build_helpers/gen-package-table.py: show Python 2.6 as unsupported on Windows.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Tweak Travis IRC notification settings.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Customize Travis notifications.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add Travis configuration.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

tahoe debug trial: print a warning message if testing uncommitted code. fixes #1992

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Merge pull request #57 from markberger/1842-lease-tradeoff.svg-overlapping

Fixes the overlapping text in docs/lease-tradeoffs.svg

Merge pull request #48 from markberger/740-cli-accept-multiple-aliases

740 cli accept multiple aliases

Fixes the overlapping text in docs/lease-tradeoffs.svg

This fixes ticket #1842. Patch courtesy of PRabahy.

Added test_check_with_multiple_aliases in test_cli

Allows check and deep-check to take multiple arguments (ticket #740)

Fix a pyflakes import warning.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

src/allmydata/uri.py: Remove unused 'human encoding' methods, and associated tests. fixes #1807

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Ignore .*.kate-swp and /*.patch files.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Ignore .DS_Store files

Add a link in accounting-overview.txt to ticket #666.

test_iputil.py: fix and improve tests on Windows.
Test all platform variants (Unix, Windows, Cygwin) on each platform.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

test_iputil.py: repair a test for cygwin (which is intended to behave differently).

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: simplify address finding code on cygwin.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

test_iputil.py: use more realistic error for 'command not found' in mock.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

test_iputil.py: repair a test by mocking 'get_local_ip_for'.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: add tests for recent changes. refs #1381, #1988, #982, #1064, #1536, #1935, #898, #1707, #1918

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: fix an error handling bug. refs #1381

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: minor cleanup to avoid dropping a Deferred.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: refactor address-finding code to treat all Unices the same.
refs #1988, #982, #1064, #1536, #1935
fixes #898, #1707, #1918

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: retry address queries on EINTR. fixes #1381

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Makefile: remove 'install' target. fixes #1717

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

WUI: add time page was rendered to client and introducer welcome pages. closes #1972

Author: Leif Ryge <leif@synthesize.us>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

properly describe saving incident reports. closes #1974

Improvements to test refs #1742

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add a test for raising NotEnoughSharesError when a Retrieve object is created with an insufficient servermap. refs #1742

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Include the number of remaining shares of the right version in NotEnoughSharesError messages.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

mutable/retrieve.py: improve comments. refs #1742

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Raise NotEnoughSharesError if there are initially not enough shares for a mutable retrieve. refs #1742

We also change the order of setting up attributes in Retrieve so that _raise_notenoughshareserror() can be called from _setup_download.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

mutable/retrieve.py: inline the single-use function _remove_reader.

A simple refactoring. Doesn't even require a new or updated unit test.

Author: Zooko O'Whielacronx <zooko@zooko.com>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add dummy __init__.py files for directories containing data. fixes #1960

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

setup.py: add allmydata/web/static/img/*.png to package_data. refs #1969

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

minor changes to hush newer pyflakes

apparently it can now detect aliasing from inside list comprehensions

update release how-to to be closer to current reality

Update terminology in RAIC design doc.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

docs/proposed/leasedb.rst: Add design doc for leasedb.
This version refs #1834.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Show git branch in version output. fixes #1953
Also make sure strings in _version.py are correctly escaped, and repair a test.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

update docs for 1.10 release

.gitignore: ignore the generated zope.interface egg too

Some dependencies are installed in the top-level directory, maybe
because they're required by the build process itself? I'm not sure.
Twisted is one of these, if the system version is not new enough, and
our .gitignore has a clause to ignore the resulting .egg directory. Now
that we've bumped our zope.interface dependency, it sometimes builds a
z.i egg too (even if it doesn't build a Twisted one). So ignore both.