Some improvements and bug fixes.

1. Discard body even if response code indicates a problem, when doing cloud backend HTTP requests. I believe this was triggering a bug in Twistd.
2. Google backend retries on 403 and other 4xx codes, not just 401.
3. More logging.

Fix bug when using oauth2client 1.1 instead of 1.0 (returned HTTP header was unicode rather than the expected bytes).

Fix pyflakes warnings.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Cleanup to declare not_import_versionable and ignorable packages in _auto_deps.py

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix version check warnings for httplib2 and python-gflags (used by oauth2client).

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add oauth2client to requirements.

Retry cloud HTTP requests on *any* exception (the list is long, and hard to make complete, so easier to just handle all exceptions).

Retry on timeouts, and increase number of persistent HTTP connections.

Author: Itamar Turner-Trauring <itamar@futurefoundries.com>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix PUTs.

Fix prefix inclusion, so authentication works.

Configuration for MS Azure.

Author: Itamar Turner-Trauring <itamar@futurefoundries.com>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

First pass at implementing the Azure GET/PUT/DELETE.

Address review comments from Daira.

1. Fix typo.
2. Rename config item googlestorage.bucket_name to googlestorage.bucket for
   consistency.

Add documentation for Google Cloud Storage backend.

msazure_container.py: Implement authentication signature scheme.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix pyflakes errors.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Unconditionally use HTTPConnectionPool, and depend on a Twisted that provides it.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

googlestorage_container.py: Use Amazon S3 namespace, since Google insists on using it.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

If oauth2client isn't present, skip Google Storage tests rather than blowing up.

Configuration support for Google Cloud Storage backend.

googlestorage_container.py: Implement PUT and listing of bucket contents.

googlestorage_container.py: Implement DELETE object.

googlestorage_container.py: Implement GET object.

Refactor useful functionality out of OpenStackContainer and into utility class.

More tests for the Google Storage container, and fixes to the tests.

Start of tests for the Google Storage container.

Tests for googlestorage_container.AuthenticationClient.

Author: Itamar Turner-Trauring <itamar@futurefoundries.com>

Sketch of working Google Cloud Storage authentication, with some demo code.

docs/backends/cloud.rst: clarify how to get to API Access in the Rackspace console.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: fix a type error introduced by the fix to #1921.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: support HP Cloud Object Storage.
Also make the choice of auth protocol for Rackspace configurable via
openstack.provider, and change the reauth period to 11 hours.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb/accounting crawler: only treat stable shares as disappeared or unleased.
fixes #1921

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cloud backend: move potentially reusable HTTP request utilities to cloud_common.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: if we get a 401 Unauthorized response, reauthenticate immediately.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

cloud_common.py: generalize ContainerRetryMixin to allow the container class to specify what to retry.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: remove a superfluous argument to get_auth_info_locked.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

accounting_crawler.py: disable removing leasedb entries for disappeared shares.
This works around ticket #1921 for now.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: avoid logging secrets in request headers.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs: add references to OpenStack/cloud backend in configuration.rst and running.rst.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: generalize to support multiple auth protocols, and add V2 protocol.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: add _http_request helper.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: factor out HTTP response code checking.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/backends/cloud.rst: add documentation for OpenStack config parameters.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: improve name of _auth_lock; simplify by using DeferredLock.run.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: change provider names to rackspace.com and rackspace.co.uk.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: add tests for OpenStackContainer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: disable or remove debug prints.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: fix a bug in type of ContainerListing.is_truncated.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Move classes common to mock and OpenStack cloud services, to cloud_common.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: refactor OpenStackCloudBackend to make it easier to add new tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: mostly complete implementation of OpenStackContainer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: improve logging in openstack_container.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: add openstack.container config parameter.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add test for OpenStack authentication client.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: add shutdown() to avoid unclean reactor errors in tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: add _ prefix to private attributes.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

setup.py: add allmydata.storage.backends.cloud.openstack module.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_client.py: add OpenStack config tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_client.py: cleanups to S3 config tests.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack service: add AuthenticationClient.
Configure using properties relevant to OpenStack.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add stub OpenStack container impl. as a copy of S3 container impl.
Generalize the container instantiation to work for either.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Fix interface violations introduced in cloud merge.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Make backupdb use dbutil.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Comment changes for ticket ref #1784

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Makefile: have 'make tmpfstest' unmount and remove stale temp directories.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Makefile: the timing for 'make tmpfstest' should exclude filesystem
mounting/unmounting and entering the password if needed.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Makefile: allow tmpfs size to be more easily overridden, and use 500 MiB by default (rebased).

(The kernel will only allocate space that is used; the limit is just in case
tests write more than expected.)

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

test_runner.py: add test for 'tahoe debug trial'.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Fixes to tests. Some tests are applied to multiple backends.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fixes to test infrastructure.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Miscellaneous corrections and additions.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to crawler classes (ShareCrawler and AccountingCrawler).
Pass in a Clock to allow (in theory) deterministic testing, although this isn't used yet by tests.
Simplify the generic ShareCrawler code by not attempting to track state during processing
of a single prefix.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to fileutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Fix bugs in Accountant.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Simplify Account.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to debug.py.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes to node classes (Node, Client and StorageServer).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes to Bucket{Reader,Writer} and disk backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes and additions to interface documentation.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add new files for cloud merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Move BucketWriter and BucketReader to storage/bucket.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Move code around and add new directories for cloud backend merge.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add dependency on our fork of txAWS (0.2.1.post5).
Add 'six' to ignorable package list because it is a dependency of txAWS via python-dateutil.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

.gitignore: changes to facilitate cloud backend merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add documentation for each storage backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add test_leasedb.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb: use a semantic primary key (storage_index, shnum, account_id), rather than an integer, for the leases table.
Take advantage of this to simplify add_or_renew_leases.
Fix a bug in add_starter_lease (which is not used yet) when a starter lease already exists.
Clean up leftover accesses to self._dirty.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Rename 'buckets' to 'sharesets' on storage status page.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove unused files storage/lease.py and storage/expirer.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Main leasedb changes (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove the 'original-*' and 'configured-*' lease crawler output that won't be supported by the accounting crawler.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cosmetics.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: ss -> server for cases that will remain a server after the server/account split.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Asyncify crawlers. Note that this breaks tests for the LeaseCrawler
(which is going away, to be replaced by the AccountingCrawler).

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

scripts/debug.py: remove display of lease information and secrets.
This version replaces the expiration field with '-' instead of '0', per Zooko's comments.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove the [storage]expire.{mutable,immutable} options.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove support for [storage]debug_discard option.
(BucketWriter.throw_out_all_data is kept because it might still be useful.)

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add new files for leasedb.

Authors: Brian Warner <warner@lothar.com> and David-Sarah Hopwood
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to specification of add_lease and renew_lease in RIStorageServer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/garbage-collection.rst: update text for leasedb changes.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>