Turn off bridging to Twisted log.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

introweb: fix connection hints for server announcements

A long time ago, the introducer's status web page would show the
advertised IP addresses for all published services, by parsing their
FURL's connection hints. This hasn't worked since about 12-Aug-2014 when
foolscap-0.6.5 changed the internal format of these hints (the column
has been empty this whole time).

This removes the "Advertised IPs" column from the Service Announcements
table. Instead, the service's full connection hints (not just the IP
address) is displayed in a tooltip/popup on the "Announced" timestamp
column.

The code that pulls these connection hints is now tolerant of all three
foolscap styles:

* foolscap<=0.6.4 : tuples of ("ipv4",host,port)
* 0.6.5 .. 0.8.0  : tuples of ("tcp",host,port)
* foolscap>=0.9.0 : strings

fixes ticket:2510

remove "announcement_distinct_hosts" from introweb JSON

The machine-parseable JSON output for the introducer status web page
used to include a key named "announcement_distinct_hosts", which counted
the number of distinct IP addresses advertised by all connected storage
servers. This hasn't worked since Aug-2014 when foolscap-0.6.5 change
the internal hints format.

This removes that field.

introducer: stop tracking hints for subscribed clients

A long time ago, the introducer's status web page would show the
advertised IP addresses for all subscribers, by parsing their
RemoteReference's FURL's connection hints. This hasn't worked since
about 12-Aug-2014 when foolscap-0.6.5 changed the internal format of
these hints.

This removes the feature: we no longer attempt to show advertised IP
addresses of subscribed clients. It also removes the code that looked
inside foolscap internals for this information.

Bump pyasn1 requirement to >= 0.1.8.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Update CREDITS.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

add print_py_pkg_ver for all deps

Including probable-near-future-transitive deps due to the new pyOpenSSL…

remove lzip, add cffi, cryptography.io

remove darcs tools from show-tool-versions

Update my CREDITS entry.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Blacklist mock > 1.0.1.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Require foolscap >= 0.8.0 and pyOpenSSL >= 0.13.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add total size output to 'tahoe admin ls-container'.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add Namespace class.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Leasedb and cloud backend merge is scheduled for v1.12.0.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

tests: rename *WithMockCloudBackend to *WithCloudBackendAndMockContainer.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add the 'tahoe admin ls-container' command and tests. fixes #1759

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add backend support for listing container contents. refs #1759

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Refactoring to move ContainerItem and ContainerListing.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Move a utility function into allmydata.util.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

S3Container: fix an oversight in the parent constructor call.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fix #2206. Includes refactoring of ContainerRetryMixin into CommonContainerMixin, and rearrangement of the code to initialize HTTPClientMixin.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

S3 container: treat an empty GET bucket response as an error.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fix wrong arguments to CorruptStoredShareError constructor in immutable share classes.
(Bug pointed out by Mark_B.)

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fixes fd leak in leasedb (ticket #2015)

Brian correctly diagnozed this issue and suggested the fix in Weekly Dev
Chat of 2013-07-09. This patch has been manually tested by Zooko using his
fdleakfinder and a unit test has been added to ensure that database connections
are properly closed. Without this patch, the 1819-cloud-merge branch uses
more than 1020 fds and will start failing on operating systems with a low fds
limit.

The portions affecting leasedb.py were written by Zooko and Daira.

Accounting crawler: re-enable share deletion. refs #1987

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Cloud backend: log URLs for HTTP responses, as well as requests.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Cloud backend: suppress unhandled TaskStopped exceptions from FileBodyProducer.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Cloud backend: change the ChunkCache's replacement policy to LRU. fixes #1885

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

test_storage.py: clean up arguments to FakeAccount methods.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Accounting crawler: make share deletion conditional (defaulting to False for now). refs #1987, #1921

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Azure: change 'container_name' config entry to 'container' for consistency with OpenStack.
Also fix a hidden bug in a test.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

docs/backends/cloud.rst: cosmetics and caveat about support for 'create-container'. refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add tests for 'tahoe admin create-container'. refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Refactor mock_cloud methods to simplify and make them more like CommonContainerMixin.
Also make sure that stripping of data arguments for logging is correct for all containers.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Move failure handling for create_container to make it more testable. refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix incorrect calls to create and delete S3 buckets in S3Container. refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add and document "tahoe admin create-container" command (rebased). refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

A missing basedir should cause an error if we try to read the config. refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Refactoring to make node config accessible without actually creating a Node. refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Refactoring to make backend configuration accessible from outside Client. refs #1971

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Help for admin commands: cosmetics and new tests.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Actually run the creation code against real Azure service, and corresponding bug fix.

Implement method to create containers in Microsoft Azure storage backend.

Add documentation for Microsoft Azure Blob Service cloud storage backend config.

Implement dumping of chunked shares and fix system tests. fixes #1959
Also, change 'assert' to '_assert' in debug.py.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Cleanup: improve error reporting for DataTooLargeError
(and fix an off-by-one error in MutableDiskShare._write_share_data).

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Make the cloud backend report corrupted shares correctly. fixes #1566

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add CorruptStoredShareError as a superclass for all corrupt share errors raised by a storage server (rebased). refs #1566

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Improve skip messages for Azure tests.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Use 'fail*' rather than 'assert*' methods for consistency with other tests.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Test that share methods are called with the shareset locked. refs #1869

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix a pyflakes warning.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Lock remote operations on sharesets. fixes #1869

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Cosmetic: fix trailing whitespace.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add Google and Azure backends to setup.py.

Some improvements and bug fixes.

1. Discard body even if response code indicates a problem, when doing cloud backend HTTP requests. I believe this was triggering a bug in Twistd.
2. Google backend retries on 403 and other 4xx codes, not just 401.
3. More logging.

Fix bug when using oauth2client 1.1 instead of 1.0 (returned HTTP header was unicode rather than the expected bytes).

Fix pyflakes warnings.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Cleanup to declare not_import_versionable and ignorable packages in _auto_deps.py

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix version check warnings for httplib2 and python-gflags (used by oauth2client).

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Add oauth2client to requirements.

Retry cloud HTTP requests on *any* exception (the list is long, and hard to make complete, so easier to just handle all exceptions).

Retry on timeouts, and increase number of persistent HTTP connections.

Author: Itamar Turner-Trauring <itamar@futurefoundries.com>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix PUTs.

Fix prefix inclusion, so authentication works.

Configuration for MS Azure.

Author: Itamar Turner-Trauring <itamar@futurefoundries.com>
Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

First pass at implementing the Azure GET/PUT/DELETE.

Address review comments from Daira.

1. Fix typo.
2. Rename config item googlestorage.bucket_name to googlestorage.bucket for
   consistency.

Add documentation for Google Cloud Storage backend.

msazure_container.py: Implement authentication signature scheme.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Fix pyflakes errors.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

Unconditionally use HTTPConnectionPool, and depend on a Twisted that provides it.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

googlestorage_container.py: Use Amazon S3 namespace, since Google insists on using it.

Signed-off-by: Daira Hopwood <david-sarah@jacaranda.org>

If oauth2client isn't present, skip Google Storage tests rather than blowing up.

Configuration support for Google Cloud Storage backend.

googlestorage_container.py: Implement PUT and listing of bucket contents.

googlestorage_container.py: Implement DELETE object.

googlestorage_container.py: Implement GET object.

Refactor useful functionality out of OpenStackContainer and into utility class.

More tests for the Google Storage container, and fixes to the tests.

Start of tests for the Google Storage container.

Tests for googlestorage_container.AuthenticationClient.

Author: Itamar Turner-Trauring <itamar@futurefoundries.com>

Sketch of working Google Cloud Storage authentication, with some demo code.

docs/backends/cloud.rst: clarify how to get to API Access in the Rackspace console.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: fix a type error introduced by the fix to #1921.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: support HP Cloud Object Storage.
Also make the choice of auth protocol for Rackspace configurable via
openstack.provider, and change the reauth period to 11 hours.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb/accounting crawler: only treat stable shares as disappeared or unleased.
fixes #1921

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cloud backend: move potentially reusable HTTP request utilities to cloud_common.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: if we get a 401 Unauthorized response, reauthenticate immediately.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

cloud_common.py: generalize ContainerRetryMixin to allow the container class to specify what to retry.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: remove a superfluous argument to get_auth_info_locked.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

accounting_crawler.py: disable removing leasedb entries for disappeared shares.
This works around ticket #1921 for now.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: avoid logging secrets in request headers.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs: add references to OpenStack/cloud backend in configuration.rst and running.rst.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: generalize to support multiple auth protocols, and add V2 protocol.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

OpenStack: add _http_request helper.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: factor out HTTP response code checking.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/backends/cloud.rst: add documentation for OpenStack config parameters.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

openstack_container.py: improve name of _auth_lock; simplify by using DeferredLock.run.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>