David-Sarah Hopwood [Fri, 22 Feb 2013 03:59:48 +0000 (03:59 +0000)]
leasedb/accounting crawler: only treat stable shares as disappeared or unleased.
fixes #1921
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 21 Feb 2013 22:04:16 +0000 (22:04 +0000)]
Cloud backend: move potentially reusable HTTP request utilities to cloud_common.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 21 Feb 2013 22:17:06 +0000 (22:17 +0000)]
OpenStack: if we get a 401 Unauthorized response, reauthenticate immediately.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 21 Feb 2013 22:14:27 +0000 (22:14 +0000)]
cloud_common.py: generalize ContainerRetryMixin to allow the container class to specify what to retry.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 21 Feb 2013 21:37:54 +0000 (21:37 +0000)]
openstack_container.py: remove a superfluous argument to get_auth_info_locked.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 21 Feb 2013 01:59:48 +0000 (01:59 +0000)]
accounting_crawler.py: disable removing leasedb entries for disappeared shares.
This works around ticket #1921 for now.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 21 Feb 2013 01:58:56 +0000 (01:58 +0000)]
openstack_container.py: avoid logging secrets in request headers.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 20 Feb 2013 23:26:15 +0000 (23:26 +0000)]
docs: add references to OpenStack/cloud backend in configuration.rst and running.rst.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 20 Feb 2013 06:21:25 +0000 (06:21 +0000)]
OpenStack: generalize to support multiple auth protocols, and add V2 protocol.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 19 Feb 2013 18:47:14 +0000 (18:47 +0000)]
OpenStack: add _http_request helper.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Mon, 18 Feb 2013 22:33:30 +0000 (22:33 +0000)]
openstack_container.py: factor out HTTP response code checking.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Fri, 15 Feb 2013 23:05:20 +0000 (23:05 +0000)]
docs/backends/cloud.rst: add documentation for OpenStack config parameters.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Fri, 15 Feb 2013 20:59:06 +0000 (20:59 +0000)]
openstack_container.py: improve name of _auth_lock; simplify by using DeferredLock.run.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Fri, 15 Feb 2013 03:13:13 +0000 (03:13 +0000)]
OpenStack: change provider names to rackspace.com and rackspace.co.uk.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 14 Feb 2013 22:30:51 +0000 (22:30 +0000)]
test_storage.py: add tests for OpenStackContainer.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 14 Feb 2013 22:30:04 +0000 (22:30 +0000)]
openstack_container.py: disable or remove debug prints.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 14 Feb 2013 22:29:17 +0000 (22:29 +0000)]
openstack_container.py: fix a bug in type of ContainerListing.is_truncated.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 14 Feb 2013 22:24:08 +0000 (22:24 +0000)]
Move classes common to mock and OpenStack cloud services, to cloud_common.py.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 14 Feb 2013 22:19:34 +0000 (22:19 +0000)]
test_storage.py: refactor OpenStackCloudBackend to make it easier to add new tests.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 13 Feb 2013 22:08:46 +0000 (22:08 +0000)]
OpenStack: mostly complete implementation of OpenStackContainer.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 13 Feb 2013 22:07:55 +0000 (22:07 +0000)]
OpenStack: improve logging in openstack_container.py.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 12 Feb 2013 01:58:51 +0000 (01:58 +0000)]
OpenStack: add openstack.container config parameter.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 12 Feb 2013 01:44:43 +0000 (01:44 +0000)]
Add test for OpenStack authentication client.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Mon, 11 Feb 2013 02:43:36 +0000 (02:43 +0000)]
openstack_container.py: add shutdown() to avoid unclean reactor errors in tests.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Mon, 11 Feb 2013 02:42:23 +0000 (02:42 +0000)]
openstack_container.py: add _ prefix to private attributes.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 5 Feb 2013 19:58:32 +0000 (19:58 +0000)]
setup.py: add allmydata.storage.backends.cloud.openstack module.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 5 Feb 2013 19:42:17 +0000 (19:42 +0000)]
test_client.py: add OpenStack config tests.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 5 Feb 2013 19:42:01 +0000 (19:42 +0000)]
test_client.py: cleanups to S3 config tests.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 5 Feb 2013 19:41:31 +0000 (19:41 +0000)]
OpenStack service: add AuthenticationClient.
Configure using properties relevant to OpenStack.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 29 Jan 2013 23:17:31 +0000 (23:17 +0000)]
Add stub OpenStack container impl. as a copy of S3 container impl.
Generalize the container instantiation to work for either.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 20 Feb 2013 06:25:32 +0000 (06:25 +0000)]
Fix interface violations introduced in cloud merge.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Mon, 15 Apr 2013 19:50:40 +0000 (20:50 +0100)]
Make backupdb use dbutil.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 07:19:47 +0000 (07:19 +0000)]
Makefile: have 'make tmpfstest' unmount and remove stale temp directories.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 07:18:55 +0000 (07:18 +0000)]
Makefile: the timing for 'make tmpfstest' should exclude filesystem
mounting/unmounting and entering the password if needed.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 04:23:01 +0000 (05:23 +0100)]
Makefile: allow tmpfs size to be more easily overridden, and use 500 MiB by default (rebased).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 07:12:05 +0000 (07:12 +0000)]
test_runner.py: add test for 'tahoe debug trial'.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Tue, 4 Aug 2015 17:20:35 +0000 (18:20 +0100)]
Fixes to tests. Some tests are applied to multiple backends.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 03:43:07 +0000 (04:43 +0100)]
Fixes to test infrastructure.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 07:04:31 +0000 (07:04 +0000)]
Miscellaneous corrections and additions.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 07:01:59 +0000 (07:01 +0000)]
Changes to crawler classes (ShareCrawler and AccountingCrawler).
Pass in a Clock to allow (in theory) deterministic testing, although this isn't used yet by tests.
Simplify the generic ShareCrawler code by not attempting to track state during processing
of a single prefix.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 03:35:27 +0000 (04:35 +0100)]
Changes to fileutil.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 06:54:02 +0000 (06:54 +0000)]
Fix bugs in Accountant.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 06:53:24 +0000 (06:53 +0000)]
Simplify Account.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 03:34:00 +0000 (04:34 +0100)]
Changes to debug.py.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 4 Aug 2015 17:02:50 +0000 (18:02 +0100)]
Changes to node classes (Node, Client and StorageServer).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 8 Apr 2014 23:56:58 +0000 (00:56 +0100)]
Changes to Bucket{Reader,Writer} and disk backend (rebased).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 06:23:16 +0000 (06:23 +0000)]
Changes and additions to interface documentation.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Wed, 9 Apr 2014 00:33:33 +0000 (01:33 +0100)]
Add new files for cloud merge (rebased).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Thu, 22 Nov 2012 06:43:49 +0000 (06:43 +0000)]
Move BucketWriter and BucketReader to storage/bucket.py.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Thu, 22 Nov 2012 05:33:35 +0000 (05:33 +0000)]
Move code around and add new directories for cloud backend merge.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 03:20:08 +0000 (04:20 +0100)]
Add dependency on our fork of txAWS (0.2.1.post5).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 8 Apr 2014 23:46:57 +0000 (00:46 +0100)]
.gitignore: changes to facilitate cloud backend merge (rebased).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 8 Apr 2014 23:43:30 +0000 (00:43 +0100)]
Add documentation for each storage backend (rebased).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Sat, 15 Dec 2012 06:27:21 +0000 (06:27 +0000)]
Add test_leasedb.py.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Sat, 15 Dec 2012 06:24:41 +0000 (06:24 +0000)]
leasedb: use a semantic primary key (storage_index, shnum, account_id), rather than an integer, for the leases table.
Take advantage of this to simplify add_or_renew_leases.
Fix a bug in add_starter_lease (which is not used yet) when a starter lease already exists.
Clean up leftover accesses to self._dirty.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Sun, 18 Nov 2012 06:22:18 +0000 (06:22 +0000)]
Rename 'buckets' to 'sharesets' on storage status page.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 03:10:53 +0000 (04:10 +0100)]
Remove unused files storage/lease.py and storage/expirer.py.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 4 Aug 2015 17:00:16 +0000 (18:00 +0100)]
Main leasedb changes (rebased).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Sun, 18 Nov 2012 05:00:13 +0000 (05:00 +0000)]
Remove the 'original-*' and 'configured-*' lease crawler output that won't be supported by the accounting crawler.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Sun, 18 Nov 2012 05:04:48 +0000 (05:04 +0000)]
Cosmetics.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Tue, 4 Aug 2015 16:58:37 +0000 (17:58 +0100)]
test_storage.py: ss -> server for cases that will remain a server after the server/account split.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Sun, 18 Nov 2012 02:52:40 +0000 (02:52 +0000)]
Asyncify crawlers. Note that this breaks tests for the LeaseCrawler
(which is going away, to be replaced by the AccountingCrawler).
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Sun, 18 Nov 2012 02:19:14 +0000 (02:19 +0000)]
scripts/debug.py: remove display of lease information and secrets.
This version replaces the expiration field with '-' instead of '0', per Zooko's comments.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 03:06:08 +0000 (04:06 +0100)]
Remove the [storage]expire.{mutable,immutable} options.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Fri, 10 Jul 2015 03:04:58 +0000 (04:04 +0100)]
Remove support for [storage]debug_discard option.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
David-Sarah Hopwood [Wed, 12 Dec 2012 05:17:11 +0000 (05:17 +0000)]
Add new files for leasedb.
Authors: Brian Warner <warner@lothar.com> and David-Sarah Hopwood
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Sat, 17 Nov 2012 00:32:24 +0000 (00:32 +0000)]
Changes to specification of add_lease and renew_lease in RIStorageServer.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Tue, 30 Oct 2012 20:34:30 +0000 (20:34 +0000)]
docs/garbage-collection.rst: update text for leasedb changes.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
David-Sarah Hopwood [Fri, 16 Nov 2012 23:26:41 +0000 (23:26 +0000)]
util/fileutil.py: add get_used_space. This version does not use FilePath.
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
Daira Hopwood [Fri, 31 Jul 2015 17:21:47 +0000 (18:21 +0100)]
Change some instances of "filesystem" that were missed to "file store".
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Brian Warner [Fri, 31 Jul 2015 02:35:00 +0000 (19:35 -0700)]
update docs for 1.10.2 release
HawkOwl [Wed, 29 Jul 2015 02:21:31 +0000 (12:21 +1000)]
add a pypy that is allowed to fail
Brian Warner [Wed, 29 Jul 2015 01:10:40 +0000 (18:10 -0700)]
update NEWS with recent changes
Brian Warner [Mon, 15 Jun 2015 04:21:22 +0000 (21:21 -0700)]
quickstart: Mention python-dev package for Python.h
Closes ticket:2328.
Brian Warner [Wed, 29 Jul 2015 00:39:39 +0000 (17:39 -0700)]
Merge branch '2447-cp-overwrite'
Brian Warner [Wed, 29 Jul 2015 00:36:50 +0000 (17:36 -0700)]
test_cli_cp: minor cleanup
Brian Warner [Wed, 29 Jul 2015 00:19:42 +0000 (17:19 -0700)]
cp: error on target-filename collisions, rather than overwrite
Closes ticket:2447
Zooko [Tue, 16 Jun 2015 17:22:59 +0000 (17:22 +0000)]
unit test for my desired behavior
Brian Warner [Wed, 29 Jul 2015 00:38:09 +0000 (17:38 -0700)]
Merge branch '2436.less-scary.3'
This reverts and improves the earlier fix for ticket:2436
Daira Hopwood [Tue, 28 Jul 2015 22:41:26 +0000 (23:41 +0100)]
test_version.py: minor cleanup.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 28 Jul 2015 22:41:13 +0000 (23:41 +0100)]
Don't show scary diagnostic warnings from --version[-and-path] (corrected). refs ticket:2436
The previous version would incorrectly add to the output of
get_package_versions_string each time it was called.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 28 Jul 2015 16:35:11 +0000 (17:35 +0100)]
Revert "Don't show scary diagnostic warnings from --version[-and-path]"
This reverts commit
431728f8f854e02d9ab2f731675f12ce18cda122.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Brian Warner [Tue, 28 Jul 2015 20:32:46 +0000 (13:32 -0700)]
mutable/retrieve.py: remove all bare assert()s
Replace them with _assert() or precondition(). Closes ticket:2462.
Brian Warner [Tue, 28 Jul 2015 20:28:36 +0000 (13:28 -0700)]
improve HTTP byte-range docs
Brian Warner [Tue, 28 Jul 2015 20:15:50 +0000 (13:15 -0700)]
webapi.rst: document Range: header on GET requests
closes ticket#2460
Brian Warner [Tue, 28 Jul 2015 20:01:15 +0000 (13:01 -0700)]
IReadable.read(): document out-of-range errors
closes ticket#2461
Brian Warner [Tue, 28 Jul 2015 18:02:17 +0000 (11:02 -0700)]
Merge branch 'sdmf-partial-2'
This cleans up the mutable-retrieve code a bit, and should fix some
corner cases where an offset/size combinations that reads the last byte
of the file (but not the first) could cause an assert to fire, making
the download hang. Should address ticket:2459 and ticket:2462.
Brian Warner [Tue, 28 Jul 2015 17:27:24 +0000 (10:27 -0700)]
test_mutable: more enhancements
Brian Warner [Tue, 28 Jul 2015 17:11:47 +0000 (10:11 -0700)]
mutable/retrieve.py: rewrite partial-read handling
This should tolerate offset/size combinations that read the last byte of
the file, something which was broken before. It quits early in the case
of zero-byte reads, to simplify the resulting "which segments do I need"
logic. Probably addresses ticket:2459.
Brian Warner [Tue, 28 Jul 2015 16:56:42 +0000 (09:56 -0700)]
enhance SDMF/MDMF tests: zero-length files, size=None
Daira Hopwood [Tue, 7 Jul 2015 20:28:59 +0000 (21:28 +0100)]
Add tests for SDMF partial reads. refs #2459
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 7 Jul 2015 20:01:33 +0000 (21:01 +0100)]
Remove some bare asserts in retrieve.py (there are still quite a few left). refs #2462
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
zooko [Tue, 21 Jul 2015 17:59:31 +0000 (17:59 +0000)]
Merge pull request #176 from tahoe-lafs/2436.less-scary.0
Don't show scary diagnostic warnings from --version[-and-path] if the only problem is that extra packages are found by pkg_resources.
Daira Hopwood [Tue, 21 Jul 2015 17:55:47 +0000 (18:55 +0100)]
Minor code cleanup in __init__.py.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Tue, 21 Jul 2015 17:51:29 +0000 (18:51 +0100)]
Don't show scary diagnostic warnings from --version[-and-path]
if the only problem is that extra packages are found by pkg_resources.
fixes ticket:2436
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Mon, 20 Jul 2015 23:16:40 +0000 (00:16 +0100)]
Change the separator character in the Magic Folder user interface design doc from '|' to '+'.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Fri, 17 Jul 2015 21:03:53 +0000 (22:03 +0100)]
Eliminate mock dependency.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Daira Hopwood [Fri, 17 Jul 2015 20:16:45 +0000 (21:16 +0100)]
Add Namespace utility class. (This is already used in the leasedb/cloud backend branch.)
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
Brian Warner [Thu, 16 Jul 2015 16:27:26 +0000 (09:27 -0700)]
Makefile upload-osx-pkg: mention where the packages go
Daira Hopwood [Thu, 16 Jul 2015 14:38:06 +0000 (15:38 +0100)]
Bump pyasn1 requirement to >= 0.1.8.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>