Add new files for cloud merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Move BucketWriter and BucketReader to storage/bucket.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Move code around and add new directories for cloud backend merge.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add dependency on our fork of txAWS (0.2.1.post5).
Add 'six' to ignorable package list because it is a dependency of txAWS via python-dateutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

.gitignore: changes to facilitate cloud backend merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add documentation for each storage backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add test_leasedb.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb: use a semantic primary key (storage_index, shnum, account_id), rather than an integer, for the leases table.
Take advantage of this to simplify add_or_renew_leases.
Fix a bug in add_starter_lease (which is not used yet) when a starter lease already exists.
Clean up leftover accesses to self._dirty.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Rename 'buckets' to 'sharesets' on storage status page.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove unused files storage/lease.py and storage/expirer.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Main leasedb changes (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove the 'original-*' and 'configured-*' lease crawler output that won't be supported by the accounting crawler.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cosmetics.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: ss -> server for cases that will remain a server after the server/account split.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Asyncify crawlers. Note that this breaks tests for the LeaseCrawler
(which is going away, to be replaced by the AccountingCrawler).

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

scripts/debug.py: remove display of lease information and secrets.
This version replaces the expiration field with '-' instead of '0', per Zooko's comments.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove the [storage]expire.{mutable,immutable} options.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove support for [storage]debug_discard option.
(BucketWriter.throw_out_all_data is kept because it might still be useful.)

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add new files for leasedb.

Authors: Brian Warner <warner@lothar.com> and David-Sarah Hopwood
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to specification of add_lease and renew_lease in RIStorageServer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/garbage-collection.rst: update text for leasedb changes.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Use "PRAGMA synchronous = OFF" for dbutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add util/dbutil.py: open/create/update sqlite databases given some schema.

Author: Brian Warner <warner@lothar.com>
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

util/fileutil.py: add get_used_space. This version does not use FilePath.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Some useful Deferred utilities. Addresses Zooko's review comments.
HookMixin and WaitForDelayedCallsMixin have been added for leasedb,
the rest are from the cloud backend branch, with minor improvements.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

misc/simulators/hashbasedsig.py: revert last patch; the original code was correct.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: fix minor miscalculation bug.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: the cost of labels was being double-counted.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: fix an incorrect comment.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Use bigger random one-time keys in timing_safe_compare. fixes #2165

Follows this advice from Marsh Ray and Solar Designer: https://twitter.com/zooko/status/431105294777597952

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Rename 'constant_time_compare' to 'timing_safe_compare'. refs #2165

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #85 from leif/isolate-keygen-test

Isolate test_keygen in its own base directory. This also causes KeyGeneratorService to create its base directory if it does not exist.

isolate test_keygen data

This makes test_keygen use its own basedir, to avoid polluting the _trial_temp
directory for later tests.

Merge pull request #84 from kpreid/patch-1

Change kpreid's email address.

Change kpreid's email address.

Merge pull request #82 from multikatt/typo

Small typo: "stil" -> "still"

Small typo: "stil" -> "still"

Merge pull request #81 from multikatt/right_align

Right-align the services div on the WUI Welcome page, to address overlap. refs #1931

Right aligning services div

docs/about.rst: Reword and remove redundancy. Also avoid "(S)FTP".

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #73 from pataquets/patch-1

Warn about ambiguities on "client" and "gateway" terms

Warn about ambiguities on "client" and "gateway" terms

Merge pull request #69 from zooko/1948-fix-warnings-from-make-doc-checks-3

1948 fix warnings from make doc checks 3

change more links to footnote-style

update link to Maildir tips (thanks MK_FG)

fix warnings from rst2html

Apparently the in-line link syntax with "<>" in them causes these warnings. I
don't know why. I changed them all to a slightly more verbose syntax.

Thanks to Mike Kazantzsev's review comment
(https://github.com/tahoe-lafs/tahoe-lafs/pull/67#commitcomment-4561370), I
moved the links to the end of each section.

magic first line tells emacs to use utf8+bom

Add ".. -*- coding: utf-8-with-signature -*-" to the first line of each .rst
file. This tells emacs to treat the file contents as utf-8, and also to prepend
a so-called utf-8 "bom" marker at the beginning of the file. This patch also
prepends those markers to each of those files.

Fix typo pointed out by CptPlastic.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Makefile: update comment for 'make clean'.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #64 from daira/2092-distclean

Add 'make distclean'.

Add 'make distclean'.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #63 from zooko/2092-make-clean-shouldnt-remove-allmydata-tahoe-egg-info

remove tahoe egg-info dir from the "clean" rule

remove tahoe egg-info dir from the "clean" rule

fixes #2092

Add quotes around command lines in quickstart.rst.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #62 from zooko/master

fix a couple more "file:" links

Merge pull request #61 from zooko/fix-small-formatting-issues-in-docs

fix small issues in formatting

fix a couple more "file:" links

Making them be just links to "$FILENAME" instead of to "file:$FILENAME" causes
them to be working links on trac and on github. Yay!

fix small issues in formatting

* fix warning from "rst2html --verbose" by not using the in-line hyperlink
  syntax. The warning said::

    known_issues.rst:261: (INFO/1) Hyperlink target "google chart api" is not referenced.

* remove dead code (no-longer-referenced link), thus fixing another warning

* change links to relative files to be just $FILENAME instead of
  file:$FILENAME, which I think will fix the problem of broken links on the
  trac and github, e.g. the link in
  https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/known_issues.rst ,
  which ought to go to
  https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/cautions.rst , but
  instead goes to file:///cautions.rst

misc/build_helpers/gen-package-table.py: show Python 2.6 as unsupported on Windows.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Tweak Travis IRC notification settings.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Customize Travis notifications.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add Travis configuration.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

tahoe debug trial: print a warning message if testing uncommitted code. fixes #1992

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Merge pull request #57 from markberger/1842-lease-tradeoff.svg-overlapping

Fixes the overlapping text in docs/lease-tradeoffs.svg

Merge pull request #48 from markberger/740-cli-accept-multiple-aliases

740 cli accept multiple aliases

Fixes the overlapping text in docs/lease-tradeoffs.svg

This fixes ticket #1842. Patch courtesy of PRabahy.

Added test_check_with_multiple_aliases in test_cli

Allows check and deep-check to take multiple arguments (ticket #740)

Fix a pyflakes import warning.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

src/allmydata/uri.py: Remove unused 'human encoding' methods, and associated tests. fixes #1807

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Ignore .*.kate-swp and /*.patch files.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Ignore .DS_Store files

Add a link in accounting-overview.txt to ticket #666.

test_iputil.py: fix and improve tests on Windows.
Test all platform variants (Unix, Windows, Cygwin) on each platform.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

test_iputil.py: repair a test for cygwin (which is intended to behave differently).

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: simplify address finding code on cygwin.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

test_iputil.py: use more realistic error for 'command not found' in mock.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

test_iputil.py: repair a test by mocking 'get_local_ip_for'.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: add tests for recent changes. refs #1381, #1988, #982, #1064, #1536, #1935, #898, #1707, #1918

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: fix an error handling bug. refs #1381

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: minor cleanup to avoid dropping a Deferred.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: refactor address-finding code to treat all Unices the same.
refs #1988, #982, #1064, #1536, #1935
fixes #898, #1707, #1918

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

iputil.py: retry address queries on EINTR. fixes #1381

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Makefile: remove 'install' target. fixes #1717

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

WUI: add time page was rendered to client and introducer welcome pages. closes #1972

Author: Leif Ryge <leif@synthesize.us>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

properly describe saving incident reports. closes #1974

Improvements to test refs #1742

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add a test for raising NotEnoughSharesError when a Retrieve object is created with an insufficient servermap. refs #1742

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Include the number of remaining shares of the right version in NotEnoughSharesError messages.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

mutable/retrieve.py: improve comments. refs #1742

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Raise NotEnoughSharesError if there are initially not enough shares for a mutable retrieve. refs #1742

We also change the order of setting up attributes in Retrieve so that _raise_notenoughshareserror() can be called from _setup_download.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

mutable/retrieve.py: inline the single-use function _remove_reader.

A simple refactoring. Doesn't even require a new or updated unit test.

Author: Zooko O'Whielacronx <zooko@zooko.com>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add dummy __init__.py files for directories containing data. fixes #1960

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

setup.py: add allmydata/web/static/img/*.png to package_data. refs #1969

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

minor changes to hush newer pyflakes

apparently it can now detect aliasing from inside list comprehensions

update release how-to to be closer to current reality

Update terminology in RAIC design doc.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

docs/proposed/leasedb.rst: Add design doc for leasedb.
This version refs #1834.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Show git branch in version output. fixes #1953
Also make sure strings in _version.py are correctly escaped, and repair a test.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

update docs for 1.10 release