Start of tests for the Google Storage container.

Tests for googlestorage_container.AuthenticationClient.

Author: Itamar Turner-Trauring <itamar@futurefoundries.com>

Sketch of working Google Cloud Storage authentication, with some demo code.

docs/backends/cloud.rst: clarify how to get to API Access in the Rackspace console.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: fix a type error introduced by the fix to #1921.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: support HP Cloud Object Storage.
Also make the choice of auth protocol for Rackspace configurable via
openstack.provider, and change the reauth period to 11 hours.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb/accounting crawler: only treat stable shares as disappeared or unleased.
fixes #1921

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cloud backend: move potentially reusable HTTP request utilities to cloud_common.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: if we get a 401 Unauthorized response, reauthenticate immediately.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

cloud_common.py: generalize ContainerRetryMixin to allow the container class to specify what to retry.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: remove a superfluous argument to get_auth_info_locked.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

accounting_crawler.py: disable removing leasedb entries for disappeared shares.
This works around ticket #1921 for now.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: avoid logging secrets in request headers.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs: add references to OpenStack/cloud backend in configuration.rst and running.rst.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: generalize to support multiple auth protocols, and add V2 protocol.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: add _http_request helper.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: factor out HTTP response code checking.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/backends/cloud.rst: add documentation for OpenStack config parameters.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: improve name of _auth_lock; simplify by using DeferredLock.run.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: change provider names to rackspace.com and rackspace.co.uk.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: add tests for OpenStackContainer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: disable or remove debug prints.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: fix a bug in type of ContainerListing.is_truncated.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Move classes common to mock and OpenStack cloud services, to cloud_common.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: refactor OpenStackCloudBackend to make it easier to add new tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: mostly complete implementation of OpenStackContainer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: improve logging in openstack_container.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: add openstack.container config parameter.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add test for OpenStack authentication client.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: add shutdown() to avoid unclean reactor errors in tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: add _ prefix to private attributes.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

setup.py: add allmydata.storage.backends.cloud.openstack module.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_client.py: add OpenStack config tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_client.py: cleanups to S3 config tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack service: add AuthenticationClient.
Configure using properties relevant to OpenStack.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add stub OpenStack container impl. as a copy of S3 container impl.
Generalize the container instantiation to work for either.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Fix interface violations introduced in cloud merge.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Make backupdb use dbutil.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Comment changes for ticket ref #1784

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Makefile: have 'make tmpfstest' unmount and remove stale temp directories.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Makefile: the timing for 'make tmpfstest' should exclude filesystem
mounting/unmounting and entering the password if needed.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Makefile: allow tmpfs size to be more easily overridden, and use 500 MiB by default (rebased).

(The kernel will only allocate space that is used; the limit is just in case
tests write more than expected.)

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

test_runner.py: add test for 'tahoe debug trial'.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Fixes to tests. Some tests are applied to multiple backends.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fixes to test infrastructure.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Miscellaneous corrections and additions.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to crawler classes (ShareCrawler and AccountingCrawler).
Pass in a Clock to allow (in theory) deterministic testing, although this isn't used yet by tests.
Simplify the generic ShareCrawler code by not attempting to track state during processing
of a single prefix.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to fileutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Fix bugs in Accountant.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Simplify Account.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to debug.py.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes to node classes (Node, Client and StorageServer).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes to Bucket{Reader,Writer} and disk backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes and additions to interface documentation.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add new files for cloud merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Move BucketWriter and BucketReader to storage/bucket.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Move code around and add new directories for cloud backend merge.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add dependency on our fork of txAWS (0.2.1.post5).
Add 'six' to ignorable package list because it is a dependency of txAWS via python-dateutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

.gitignore: changes to facilitate cloud backend merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add documentation for each storage backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add test_leasedb.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb: use a semantic primary key (storage_index, shnum, account_id), rather than an integer, for the leases table.
Take advantage of this to simplify add_or_renew_leases.
Fix a bug in add_starter_lease (which is not used yet) when a starter lease already exists.
Clean up leftover accesses to self._dirty.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Rename 'buckets' to 'sharesets' on storage status page.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove unused files storage/lease.py and storage/expirer.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Main leasedb changes (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove the 'original-*' and 'configured-*' lease crawler output that won't be supported by the accounting crawler.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cosmetics.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: ss -> server for cases that will remain a server after the server/account split.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Asyncify crawlers. Note that this breaks tests for the LeaseCrawler
(which is going away, to be replaced by the AccountingCrawler).

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

scripts/debug.py: remove display of lease information and secrets.
This version replaces the expiration field with '-' instead of '0', per Zooko's comments.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove the [storage]expire.{mutable,immutable} options.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove support for [storage]debug_discard option.
(BucketWriter.throw_out_all_data is kept because it might still be useful.)

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add new files for leasedb.

Authors: Brian Warner <warner@lothar.com> and David-Sarah Hopwood
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to specification of add_lease and renew_lease in RIStorageServer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/garbage-collection.rst: update text for leasedb changes.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Use "PRAGMA synchronous = OFF" for dbutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add util/dbutil.py: open/create/update sqlite databases given some schema.

Author: Brian Warner <warner@lothar.com>
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

util/fileutil.py: add get_used_space. This version does not use FilePath.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Some useful Deferred utilities. Addresses Zooko's review comments.
HookMixin and WaitForDelayedCallsMixin have been added for leasedb,
the rest are from the cloud backend branch, with minor improvements.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

misc/simulators/hashbasedsig.py: revert last patch; the original code was correct.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: fix minor miscalculation bug.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: the cost of labels was being double-counted.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

misc/simulators/hashbasedsig.py: fix an incorrect comment.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Use bigger random one-time keys in timing_safe_compare. fixes #2165

Follows this advice from Marsh Ray and Solar Designer: https://twitter.com/zooko/status/431105294777597952

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Rename 'constant_time_compare' to 'timing_safe_compare'. refs #2165

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #85 from leif/isolate-keygen-test

Isolate test_keygen in its own base directory. This also causes KeyGeneratorService to create its base directory if it does not exist.

isolate test_keygen data

This makes test_keygen use its own basedir, to avoid polluting the _trial_temp
directory for later tests.

Merge pull request #84 from kpreid/patch-1

Change kpreid's email address.

Change kpreid's email address.

Merge pull request #82 from multikatt/typo

Small typo: "stil" -> "still"

Small typo: "stil" -> "still"

Merge pull request #81 from multikatt/right_align

Right-align the services div on the WUI Welcome page, to address overlap. refs #1931

Right aligning services div

docs/about.rst: Reword and remove redundancy. Also avoid "(S)FTP".

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #73 from pataquets/patch-1

Warn about ambiguities on "client" and "gateway" terms

Warn about ambiguities on "client" and "gateway" terms

Merge pull request #69 from zooko/1948-fix-warnings-from-make-doc-checks-3

1948 fix warnings from make doc checks 3

change more links to footnote-style

update link to Maildir tips (thanks MK_FG)

fix warnings from rst2html

Apparently the in-line link syntax with "<>" in them causes these warnings. I
don't know why. I changed them all to a slightly more verbose syntax.

Thanks to Mike Kazantzsev's review comment
(https://github.com/tahoe-lafs/tahoe-lafs/pull/67#commitcomment-4561370), I
moved the links to the end of each section.