Brian Warner [Sat, 1 Oct 2011 23:35:53 +0000 (00:35 +0100)]
MDMF: remove extension fields from caps, tolerate arbitrary ones. Fixes #1526
The filecaps used to be produced with hints for 'k' and segsize, but they
weren't actually used, and doing so had the potential to limit how we change
those filecaps in the future. Also the parsing code had some problems dealing
with other numbers of extensions. Removing the existing fields and making the
parser tolerate (and ignore) extra ones makes MDMF more future-proof.
david-sarah [Tue, 27 Sep 2011 22:53:36 +0000 (15:53 -0700)]
test/test_runner.py: BinTahoe.test_path has rare nondeterministic failures; this patch probably fixes a problem where the actual cause of failure is masked by a string conversion error.
Zooko O'Whielacronx [Mon, 26 Sep 2011 20:38:01 +0000 (13:38 -0700)]
docs/configuration.rst: add section about the types of node, and clarify when setting web.port enables web-API service. fixes #1444
Brian Warner [Sun, 25 Sep 2011 23:48:11 +0000 (16:48 -0700)]
TAG allmydata-tahoe-1.9.0a2
Brian Warner [Sun, 25 Sep 2011 23:35:29 +0000 (16:35 -0700)]
NEWS: tidy up a little bit, reprioritize some items, hide some non-user-visible items
david-sarah [Sun, 25 Sep 2011 05:10:01 +0000 (22:10 -0700)]
docs: fix some broken .rst links. refs #1542
david-sarah [Sun, 25 Sep 2011 05:22:06 +0000 (22:22 -0700)]
mutable/publish.py: fix an unused import. refs #1542
david-sarah [Sun, 25 Sep 2011 05:01:19 +0000 (22:01 -0700)]
NEWS: fix .rst formatting.
david-sarah [Sun, 25 Sep 2011 04:53:43 +0000 (21:53 -0700)]
NEWS: updates for 1.9alpha2.
david-sarah [Sun, 25 Sep 2011 02:36:51 +0000 (19:36 -0700)]
mutable/layout.py: make unpack_sdmf_checkstring and unpack_mdmf_checkstring more similar, and change an assert to give a more useful message if it fails. refs #1540
kevan [Sun, 25 Sep 2011 00:43:05 +0000 (17:43 -0700)]
mutable/publish: handle unknown mutable share formats when handling errors
kevan [Sun, 25 Sep 2011 00:41:34 +0000 (17:41 -0700)]
mutable/layout: break unpack_checkstring into unpack_mdmf_checkstring and unpack_sdmf_checkstring, add distinguisher function for checkstrings
kevan [Sat, 24 Sep 2011 23:54:15 +0000 (16:54 -0700)]
test/test_mutable: reenable mdmf publish surprise test
kevan [Sat, 24 Sep 2011 23:51:37 +0000 (16:51 -0700)]
mutable/publish: use unpack_mdmf_checkstring and unpack_sdmf_checkstring instead of unpack_checkstring. fixes #1540
david-sarah [Fri, 16 Sep 2011 22:34:50 +0000 (15:34 -0700)]
misc/coding_tools/check_interfaces.py: report all violations rather than only one for a given class, by including a forked version of verifyClass. refs #1474
david-sarah [Fri, 16 Sep 2011 21:26:33 +0000 (14:26 -0700)]
misc/coding_tools/check_interfaces.py: use os.walk instead of FilePath, since this script shouldn't really depend on Twisted. refs #1474
david-sarah [Fri, 16 Sep 2011 21:26:24 +0000 (14:26 -0700)]
misc/coding_tools/check-interfaces.py: reduce false-positives by adding Dummy* to the set of excluded classnames, and bench-* to the set of excluded basenames. refs #1474
david-sarah [Thu, 15 Sep 2011 16:15:32 +0000 (09:15 -0700)]
Add a script 'misc/coding_tools/check-interfaces.py' that checks whether zope interfaces are enforced. Also add 'check-interfaces', 'version-and-path', and 'code-checks' targets to the Makefile. fixes #1474
david-sarah [Sat, 24 Sep 2011 21:12:08 +0000 (14:12 -0700)]
mutable/publish.py: copy the self.writers dict before iterating over it, since we remove elements from it during the iteration. refs #393
david-sarah [Sat, 24 Sep 2011 20:50:04 +0000 (13:50 -0700)]
mutable/publish.py: simplify by refactoring self.outstanding to self.num_outstanding. refs #393
david-sarah [Fri, 23 Sep 2011 21:16:22 +0000 (14:16 -0700)]
test_mutable.py: update SkipTest message for test_publish_surprise_mdmf to reference the right ticket number. refs #1540.
Brian Warner [Fri, 23 Sep 2011 07:37:48 +0000 (00:37 -0700)]
control.py: unbreak speed-test: overwrite() wants a MutableData, not str
Really, all the upload/modify APIs should take a string or a filehandle, and
internally wrap it as needed. Callers should not need to be aware of
Uploadable() or MutableData() classes.
david-sarah [Tue, 20 Sep 2011 18:33:19 +0000 (11:33 -0700)]
test_mutable.py: skip test_publish_surprise_mdmf, which is causing an error. refs #1534, #393
kevan [Sun, 18 Sep 2011 00:36:57 +0000 (17:36 -0700)]
test/test_mutable: write publish surprise test for MDMF, rename existing test_publish_surprise to clarify that it is for SDMF
kevan [Sun, 18 Sep 2011 00:35:33 +0000 (17:35 -0700)]
test/test_mutable: refactor publish surprise test into common test fixture, rewrite test_publish_surprise to use test fixture
kevan [Sat, 17 Sep 2011 23:47:08 +0000 (16:47 -0700)]
mutable/publish: add errback immediately after write, don't consume errors from other parts of the publisher
david-sarah [Tue, 20 Sep 2011 15:49:18 +0000 (08:49 -0700)]
.darcs-boringfile: minor cleanups.
david-sarah [Tue, 20 Sep 2011 03:01:56 +0000 (20:01 -0700)]
uri.py: fix two interface violations in verifier URI classes. refs #1474
Brian Warner [Thu, 15 Sep 2011 20:26:20 +0000 (13:26 -0700)]
Make platform-detection code tolerate linux-3.0, patch by zooko.
Otherwise address-autodetection can't find ifconfig. refs #1536
david-sarah [Thu, 15 Sep 2011 18:51:26 +0000 (11:51 -0700)]
test_web.py: fix a bug in _count_leases that was causing us to check only the lease count of one share file, not of all share files as intended.
Zooko O'Whielacronx [Wed, 14 Sep 2011 06:47:28 +0000 (23:47 -0700)]
docs: insert a newline at the beginning of known_issues.rst to see if this makes it render more nicely in trac
Zooko O'Whielacronx [Wed, 14 Sep 2011 05:57:13 +0000 (22:57 -0700)]
docs: remove the coding: utf-8 declaration at the to of known_issues.rst, since the trac rendering doesn't hide it
Zooko O'Whielacronx [Wed, 14 Sep 2011 05:54:19 +0000 (22:54 -0700)]
docs: more cleanup of known_issues.rst -- now it passes "rst2html --verbose" without comment
Zooko O'Whielacronx [Wed, 14 Sep 2011 05:16:39 +0000 (22:16 -0700)]
docs: more formatting improvements to known_issues.rst
Zooko O'Whielacronx [Wed, 14 Sep 2011 05:02:40 +0000 (22:02 -0700)]
docs: reformatting of known_issues.rst
Zooko O'Whielacronx [Wed, 14 Sep 2011 04:59:09 +0000 (21:59 -0700)]
docs: fix formatting error in docs/known_issues.rst
Zooko O'Whielacronx [Tue, 13 Sep 2011 21:05:44 +0000 (14:05 -0700)]
merge Tahoe-LAFS v1.8.3 release announcement with trunk
Zooko O'Whielacronx [Tue, 13 Sep 2011 16:58:26 +0000 (09:58 -0700)]
docs: release notes for Tahoe-LAFS v1.8.3
Zooko O'Whielacronx [Tue, 13 Sep 2011 02:42:55 +0000 (19:42 -0700)]
tests: bump up the timeout in this test that fails on FreeStorm's CentOS in order to see if it is just very slow
david-sarah [Tue, 13 Sep 2011 00:28:43 +0000 (17:28 -0700)]
interfaces: document that the 'fills-holes-with-zero-bytes' key should be used to detect whether a storage server has that behavior. refs #1528
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:33:57 +0000 (15:33 -0700)]
CREDITS: more CREDITS for Kevan and David-Sarah
Zooko O'Whielacronx [Tue, 13 Sep 2011 20:55:21 +0000 (13:55 -0700)]
merge NEWS about the mutable file bounds fixes with NEWS about work-in-progress
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:33:29 +0000 (15:33 -0700)]
doc: add NEWS item about fixes to potential palimpsest issues in mutable files
ref. #1528
Zooko O'Whielacronx [Tue, 13 Sep 2011 20:51:53 +0000 (13:51 -0700)]
merge the NEWS about the security fix (#1528) with the work-in-progress NEWS
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:32:46 +0000 (15:32 -0700)]
doc: add NEWS entry about the issue which allows unauthorized deletion of shares
ref. #1528
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:31:35 +0000 (15:31 -0700)]
doc: add entry in known_issues.rst about the issue which allows unauthorized deletion of shares
ref. #1528
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:26:55 +0000 (15:26 -0700)]
storage: more paranoid handling of bounds and palimpsests in mutable share files
* storage server ignores requests to extend shares by sending a new_length
* storage server fills exposed holes (created by sending a write vector whose offset begins after the end of the current data) with 0 to avoid "palimpsest" exposure of previous contents
* storage server zeroes out lease info at the old location when moving it to a new location
ref. #1528
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:25:54 +0000 (15:25 -0700)]
storage: test that the storage server ignores requests to extend shares by sending a new_length, and that the storage server fills exposed holes with 0 to avoid "palimpsest" exposure of previous contents
ref. #1528
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:24:58 +0000 (15:24 -0700)]
immutable: prevent clients from reading past the end of share data, which would allow them to learn the cancellation secret
Declare explicitly that we prevent this problem in the server's version dict.
fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:23:31 +0000 (15:23 -0700)]
storage: remove the storage server's "remote_cancel_lease" function
We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:23:24 +0000 (15:23 -0700)]
storage: test that the storage server does *not* have a "remote_cancel_lease" function
We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
ref. #1528
Zooko O'Whielacronx [Mon, 12 Sep 2011 22:12:01 +0000 (15:12 -0700)]
immutable: test whether the server allows clients to read past the end of share data, which would allow them to learn the cancellation secret
Also test whether the server explicitly declares that it prevents this problem.
ref #1528
Brian Warner [Mon, 5 Sep 2011 19:36:26 +0000 (12:36 -0700)]
Retrieve._activate_enough_peers: rewrite Verify logic
Brian Warner [Mon, 5 Sep 2011 19:02:42 +0000 (12:02 -0700)]
Retrieve: implement/test stopProducing
Brian Warner [Sun, 4 Sep 2011 02:59:06 +0000 (19:59 -0700)]
move DownloadStopped from download.common to interfaces
Brian Warner [Mon, 5 Sep 2011 18:54:04 +0000 (11:54 -0700)]
retrieve.py: remove vestigal self._validated_readers
Brian Warner [Mon, 5 Sep 2011 18:38:44 +0000 (11:38 -0700)]
Retrieve: rewrite flow-control: use a top-level loop() to catch all errors
This ought to close the potential for dropped errors and hanging downloads.
Verify needs to be examined, I may have broken it, although all tests pass.
Brian Warner [Mon, 5 Sep 2011 08:31:18 +0000 (01:31 -0700)]
Retrieve: merge _validate_active_prefixes into _add_active_peers
Brian Warner [Mon, 5 Sep 2011 08:26:33 +0000 (01:26 -0700)]
Retrieve: remove the initial prefix-is-still-good check
This check needs to be done with each fetch from the storage server, to
detect when someone has changed the share (i.e. our servermap goes stale).
Doing it just once at the beginning of retrieve isn't enough: a write might
occur after the first segment but before the second, etc.
_try_to_validate_prefix() was not removed: it will be used by the future
check-with-each-fetch code.
test_mutable.Roundtrip.test_corrupt_all_seqnum_late was disabled, since it
fails until this check is brought back. (the corruption it applies only
touches the prefix, not the block data, so the check-less retrieve actually
tolerates it). Don't forget to re-enable it once the check is brought back.
Brian Warner [Mon, 5 Sep 2011 19:04:08 +0000 (12:04 -0700)]
MDMFSlotReadProxy: remove the queue
This is a neat trick to reduce Foolscap overhead, but the need for an
explicit flush() complicates the Retrieve path and makes it prone to
lost-progress bugs.
Also change test_mutable.FakeStorageServer to tolerate multiple reads of the
same share in a row, a limitation exposed by turning off the queue.
Brian Warner [Sun, 4 Sep 2011 03:21:18 +0000 (20:21 -0700)]
rearrange Retrieve: first step, shouldn't change order of execution
david-sarah [Tue, 6 Sep 2011 18:37:30 +0000 (11:37 -0700)]
CLI: test_cli.py -- remove an unnecessary call in test_mkdir_mutable_type. refs #1527
david-sarah [Tue, 6 Sep 2011 18:30:20 +0000 (11:30 -0700)]
CLI: improve test for 'tahoe mkdir --mutable-type='. refs #1527
david-sarah [Mon, 5 Sep 2011 02:09:22 +0000 (19:09 -0700)]
CLI: make the --mutable-type option value for 'tahoe put' and 'tahoe mkdir' case-insensitive, and change --help for these commands accordingly. fixes #1527
Kevan Carstensen [Sat, 3 Sep 2011 19:09:20 +0000 (12:09 -0700)]
cli: make --mutable-type imply --mutable in 'tahoe put'
david-sarah [Sat, 3 Sep 2011 22:23:04 +0000 (15:23 -0700)]
SFTP: add a comment about a subtle interaction between OverwriteableFileConsumer and GeneralSFTPFile, and test the case it is commenting on.
Brian Warner [Thu, 1 Sep 2011 16:05:43 +0000 (09:05 -0700)]
improve the storage/mutable.py asserts even more
wilcoxjg [Thu, 1 Sep 2011 08:41:44 +0000 (01:41 -0700)]
storage/mutable.py: special characters in struct.foo arguments indicate standard as opposed to native sizes, we should be using these characters in these asserts
david-sarah [Wed, 31 Aug 2011 23:21:48 +0000 (16:21 -0700)]
docs/write_coordination.rst: fix formatting and add more specific warning about access via sshfs.
Brian Warner [Wed, 31 Aug 2011 05:04:51 +0000 (22:04 -0700)]
test_mutable.Version: consolidate some tests, reduce runtime from 19s to 15s
Kevan Carstensen [Tue, 30 Aug 2011 21:01:41 +0000 (14:01 -0700)]
mutable/retrieve: handle the case where self._read_length is 0.
Note that the downloader will still fetch a segment for a zero-length
read, which is wasteful. Fixing that isn't specifically required to fix
#1512, but it should probably be fixed before 1.9.
Brian Warner [Tue, 30 Aug 2011 16:32:05 +0000 (09:32 -0700)]
NEWS: added summary of all changes since 1.8.2. Needs editing.
Brian Warner [Mon, 29 Aug 2011 07:27:17 +0000 (00:27 -0700)]
test_mutable.Update: only upload the files needed for each test. refs #1500
This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
It also fixes a couple of places where a Deferred was being dropped, which
would cause two tests to run in parallel and also confuse error reporting.
Brian Warner [Mon, 29 Aug 2011 06:32:46 +0000 (23:32 -0700)]
Let Uploader retain History instead of passing it into upload(). Fixes #1079.
This consistently records all immutable uploads in the Recent Uploads And
Downloads page, regardless of code path. Previously, certain webapi upload
operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
object and were left out.
Brian Warner [Sun, 28 Aug 2011 23:22:21 +0000 (16:22 -0700)]
Fix mutable publish/retrieve timing status displays. Fixes #1505.
publish:
* encrypt and encode times are cumulative, not just current-segment
retrieve:
* same for decrypt and decode times
* update "current status" to include segment number
* set status to Finished/Failed when download is complete
* set progress to 1.0 when complete
More improvements to consider:
* progress is currently 0% or 100%: should calculate how many segments are
involved (remembering retrieve can be less than the whole file) and set it
to a fraction
* "fetch" time is fuzzy: what we want is to know how much of the delay is not
our own fault, but since we do decode/decrypt work while waiting for more
shares, it's not straightforward
Brian Warner [Sun, 28 Aug 2011 08:09:31 +0000 (01:09 -0700)]
Teach 'tahoe debug catalog-shares about MDMF. Closes #1507.
Brian Warner [Sun, 28 Aug 2011 07:45:56 +0000 (00:45 -0700)]
debug.py: remove some dead comments
Brian Warner [Sun, 28 Aug 2011 07:42:54 +0000 (00:42 -0700)]
hush pyflakes
Brian Warner [Sun, 28 Aug 2011 07:41:03 +0000 (00:41 -0700)]
MutableFileNode.set_downloader_hints: never depend upon order of dict.values()
The old code was calculating the "extension parameters" (a list) from the
downloader hints (a dictionary) with hints.values(), which is not stable, and
would result in corrupted filecaps (with the 'k' and 'segsize' hints
occasionally swapped). The new code always uses [k,segsize].
Brian Warner [Sun, 28 Aug 2011 07:39:21 +0000 (00:39 -0700)]
layout.py: fix MDMF share layout documentation
Brian Warner [Sun, 28 Aug 2011 07:38:34 +0000 (00:38 -0700)]
teach 'tahoe debug dump-share' about MDMF and offsets. refs #1507
Brian Warner [Sun, 28 Aug 2011 06:47:28 +0000 (23:47 -0700)]
test_mutable.Version.test_debug: use splitlines() to fix buildslaves
Any slave running in a directory with spaces in the name was miscounting
shares, causing the test to fail.
Brian Warner [Sun, 28 Aug 2011 00:55:42 +0000 (17:55 -0700)]
test_mutable.Version: exercise 'tahoe debug find-shares' on MDMF. refs #1507
Also changes NoNetworkGrid to put shares in storage/shares/ .
Brian Warner [Sun, 28 Aug 2011 00:21:18 +0000 (17:21 -0700)]
test_mutable.py: oops, missed a .todo
Brian Warner [Sat, 27 Aug 2011 23:57:07 +0000 (16:57 -0700)]
test_mutable: merge davidsarah's patch with my Version refactorings
david-sarah [Tue, 23 Aug 2011 01:27:20 +0000 (18:27 -0700)]
Make the immutable/read-only constraint checking for MDMF URIs identical to that for SSK URIs. refs #393
david-sarah [Tue, 23 Aug 2011 01:15:32 +0000 (18:15 -0700)]
Additional tests for MDMF URIs and for zero-length files. refs #393
david-sarah [Mon, 22 Aug 2011 01:41:11 +0000 (18:41 -0700)]
Additional tests for zero-length partial reads and updates to mutable versions. refs #393
Brian Warner [Sat, 27 Aug 2011 23:27:37 +0000 (16:27 -0700)]
test_mutable.Version: factor out some expensive uploads, save 25% runtime
Brian Warner [Sat, 27 Aug 2011 22:50:31 +0000 (15:50 -0700)]
SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.
Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).
The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.
Brian Warner [Sat, 27 Aug 2011 19:52:53 +0000 (12:52 -0700)]
layout.py: annotate assertion to figure out 'tahoe backup' failure
Brian Warner [Sat, 27 Aug 2011 19:50:48 +0000 (12:50 -0700)]
Add 'tahoe debug dump-cap' support for MDMF, DIR2-CHK, DIR2-MDMF. refs #1507.
This also adds tests for all those cases, and fixes an omission in uri.py
that broke parsing of DIR2-MDMF-Verifier and DIR2-CHK-Verifier.
Brian Warner [Sat, 27 Aug 2011 19:06:02 +0000 (12:06 -0700)]
MDMF: more writable/writeable consistentifications
Brian Warner [Sat, 27 Aug 2011 18:33:57 +0000 (11:33 -0700)]
MDMF: s/Writable/Writeable/g, for consistency with existing SDMF code
david-sarah [Fri, 26 Aug 2011 23:03:45 +0000 (16:03 -0700)]
setup.cfg: remove no-longer-supported test_mac_diskimage alias. refs #1479
Brian Warner [Thu, 25 Aug 2011 23:01:40 +0000 (16:01 -0700)]
test_mutable.Update: increase timeout from 120s to 400s, slaves are failing
Zooko O'Whielacronx [Thu, 25 Aug 2011 20:11:16 +0000 (13:11 -0700)]
tests: fix check_memory test
fixes #1503
Brian Warner [Thu, 25 Aug 2011 16:11:22 +0000 (09:11 -0700)]
TAG allmydata-tahoe-1.9.0a1
Brian Warner [Thu, 25 Aug 2011 16:10:26 +0000 (09:10 -0700)]
touch NEWS to trigger buildslaves
Brian Warner [Tue, 23 Aug 2011 21:19:04 +0000 (14:19 -0700)]
test_mutable.Update: remove .timeout overrides, otherwise tests ERROR