Brian Warner [Thu, 1 Sep 2011 16:05:43 +0000 (09:05 -0700)]
improve the storage/mutable.py asserts even more
wilcoxjg [Thu, 1 Sep 2011 08:41:44 +0000 (01:41 -0700)]
storage/mutable.py: special characters in struct.foo arguments indicate standard as opposed to native sizes, we should be using these characters in these asserts
david-sarah [Wed, 31 Aug 2011 23:21:48 +0000 (16:21 -0700)]
docs/write_coordination.rst: fix formatting and add more specific warning about access via sshfs.
Brian Warner [Wed, 31 Aug 2011 05:04:51 +0000 (22:04 -0700)]
test_mutable.Version: consolidate some tests, reduce runtime from 19s to 15s
Kevan Carstensen [Tue, 30 Aug 2011 21:01:41 +0000 (14:01 -0700)]
mutable/retrieve: handle the case where self._read_length is 0.
Note that the downloader will still fetch a segment for a zero-length
read, which is wasteful. Fixing that isn't specifically required to fix
#1512, but it should probably be fixed before 1.9.
Brian Warner [Tue, 30 Aug 2011 16:32:05 +0000 (09:32 -0700)]
NEWS: added summary of all changes since 1.8.2. Needs editing.
Brian Warner [Mon, 29 Aug 2011 07:27:17 +0000 (00:27 -0700)]
test_mutable.Update: only upload the files needed for each test. refs #1500
This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
It also fixes a couple of places where a Deferred was being dropped, which
would cause two tests to run in parallel and also confuse error reporting.
Brian Warner [Mon, 29 Aug 2011 06:32:46 +0000 (23:32 -0700)]
Let Uploader retain History instead of passing it into upload(). Fixes #1079.
This consistently records all immutable uploads in the Recent Uploads And
Downloads page, regardless of code path. Previously, certain webapi upload
operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
object and were left out.
Brian Warner [Sun, 28 Aug 2011 23:22:21 +0000 (16:22 -0700)]
Fix mutable publish/retrieve timing status displays. Fixes #1505.
publish:
* encrypt and encode times are cumulative, not just current-segment
retrieve:
* same for decrypt and decode times
* update "current status" to include segment number
* set status to Finished/Failed when download is complete
* set progress to 1.0 when complete
More improvements to consider:
* progress is currently 0% or 100%: should calculate how many segments are
involved (remembering retrieve can be less than the whole file) and set it
to a fraction
* "fetch" time is fuzzy: what we want is to know how much of the delay is not
our own fault, but since we do decode/decrypt work while waiting for more
shares, it's not straightforward
Brian Warner [Sun, 28 Aug 2011 08:09:31 +0000 (01:09 -0700)]
Teach 'tahoe debug catalog-shares about MDMF. Closes #1507.
Brian Warner [Sun, 28 Aug 2011 07:45:56 +0000 (00:45 -0700)]
debug.py: remove some dead comments
Brian Warner [Sun, 28 Aug 2011 07:42:54 +0000 (00:42 -0700)]
hush pyflakes
Brian Warner [Sun, 28 Aug 2011 07:41:03 +0000 (00:41 -0700)]
MutableFileNode.set_downloader_hints: never depend upon order of dict.values()
The old code was calculating the "extension parameters" (a list) from the
downloader hints (a dictionary) with hints.values(), which is not stable, and
would result in corrupted filecaps (with the 'k' and 'segsize' hints
occasionally swapped). The new code always uses [k,segsize].
Brian Warner [Sun, 28 Aug 2011 07:39:21 +0000 (00:39 -0700)]
layout.py: fix MDMF share layout documentation
Brian Warner [Sun, 28 Aug 2011 07:38:34 +0000 (00:38 -0700)]
teach 'tahoe debug dump-share' about MDMF and offsets. refs #1507
Brian Warner [Sun, 28 Aug 2011 06:47:28 +0000 (23:47 -0700)]
test_mutable.Version.test_debug: use splitlines() to fix buildslaves
Any slave running in a directory with spaces in the name was miscounting
shares, causing the test to fail.
Brian Warner [Sun, 28 Aug 2011 00:55:42 +0000 (17:55 -0700)]
test_mutable.Version: exercise 'tahoe debug find-shares' on MDMF. refs #1507
Also changes NoNetworkGrid to put shares in storage/shares/ .
Brian Warner [Sun, 28 Aug 2011 00:21:18 +0000 (17:21 -0700)]
test_mutable.py: oops, missed a .todo
Brian Warner [Sat, 27 Aug 2011 23:57:07 +0000 (16:57 -0700)]
test_mutable: merge davidsarah's patch with my Version refactorings
david-sarah [Tue, 23 Aug 2011 01:27:20 +0000 (18:27 -0700)]
Make the immutable/read-only constraint checking for MDMF URIs identical to that for SSK URIs. refs #393
david-sarah [Tue, 23 Aug 2011 01:15:32 +0000 (18:15 -0700)]
Additional tests for MDMF URIs and for zero-length files. refs #393
david-sarah [Mon, 22 Aug 2011 01:41:11 +0000 (18:41 -0700)]
Additional tests for zero-length partial reads and updates to mutable versions. refs #393
Brian Warner [Sat, 27 Aug 2011 23:27:37 +0000 (16:27 -0700)]
test_mutable.Version: factor out some expensive uploads, save 25% runtime
Brian Warner [Sat, 27 Aug 2011 22:50:31 +0000 (15:50 -0700)]
SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.
Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).
The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.
Brian Warner [Sat, 27 Aug 2011 19:52:53 +0000 (12:52 -0700)]
layout.py: annotate assertion to figure out 'tahoe backup' failure
Brian Warner [Sat, 27 Aug 2011 19:50:48 +0000 (12:50 -0700)]
Add 'tahoe debug dump-cap' support for MDMF, DIR2-CHK, DIR2-MDMF. refs #1507.
This also adds tests for all those cases, and fixes an omission in uri.py
that broke parsing of DIR2-MDMF-Verifier and DIR2-CHK-Verifier.
Brian Warner [Sat, 27 Aug 2011 19:06:02 +0000 (12:06 -0700)]
MDMF: more writable/writeable consistentifications
Brian Warner [Sat, 27 Aug 2011 18:33:57 +0000 (11:33 -0700)]
MDMF: s/Writable/Writeable/g, for consistency with existing SDMF code
david-sarah [Fri, 26 Aug 2011 23:03:45 +0000 (16:03 -0700)]
setup.cfg: remove no-longer-supported test_mac_diskimage alias. refs #1479
Brian Warner [Thu, 25 Aug 2011 23:01:40 +0000 (16:01 -0700)]
test_mutable.Update: increase timeout from 120s to 400s, slaves are failing
Zooko O'Whielacronx [Thu, 25 Aug 2011 20:11:16 +0000 (13:11 -0700)]
tests: fix check_memory test
fixes #1503
Brian Warner [Thu, 25 Aug 2011 16:11:22 +0000 (09:11 -0700)]
TAG allmydata-tahoe-1.9.0a1
Brian Warner [Thu, 25 Aug 2011 16:10:26 +0000 (09:10 -0700)]
touch NEWS to trigger buildslaves
Brian Warner [Tue, 23 Aug 2011 21:19:04 +0000 (14:19 -0700)]
test_mutable.Update: remove .timeout overrides, otherwise tests ERROR
Brian Warner [Thu, 25 Aug 2011 02:19:02 +0000 (19:19 -0700)]
blacklist.py: add read() method too, for completeness
david-sarah [Wed, 24 Aug 2011 15:59:28 +0000 (08:59 -0700)]
Implementation, tests and docs for blacklists. This version allows listing directories containing a blacklisted child. Inclusion of blacklist.py fixed. fixes #1425
david-sarah [Tue, 16 Aug 2011 22:50:43 +0000 (15:50 -0700)]
mutable/layout.py: fix unused import. refs #393
david-sarah [Tue, 16 Aug 2011 21:46:12 +0000 (14:46 -0700)]
mutable/retrieve.py: cosmetics and remove a stale comment. refs #393
Kevan Carstensen [Sat, 13 Aug 2011 21:00:05 +0000 (14:00 -0700)]
mutable/filenode.py: don't fetch more segments than necesasry to update the file
Kevan Carstensen [Sat, 13 Aug 2011 18:39:36 +0000 (11:39 -0700)]
test/test_mutable: test for incorrect div_ceil equations
Kevan Carstensen [Sat, 13 Aug 2011 18:38:33 +0000 (11:38 -0700)]
mutable/retrieve.py: use floor division to calculate segment boundaries, don't fetch more segments than necessary
Kevan Carstensen [Sat, 13 Aug 2011 18:37:10 +0000 (11:37 -0700)]
mdmf: clean up boolean expressions, correct typos, remove self._paused, and don't unconditionally initialize block hash trees, asll as suggested by davidsarahs' review comments
Brian Warner [Thu, 11 Aug 2011 17:54:18 +0000 (10:54 -0700)]
now that tests pass with full-size keys, return test-keys to normal (522bit)
Brian Warner [Thu, 11 Aug 2011 17:53:50 +0000 (10:53 -0700)]
fix SHARE_HASH_CHAIN_SIZE computation
david-sarah [Wed, 10 Aug 2011 20:29:42 +0000 (13:29 -0700)]
More idiomatic resolution of the conflict between ticket393-MDMF-2 and trunk. refs #393
david-sarah [Wed, 10 Aug 2011 20:23:10 +0000 (13:23 -0700)]
Replace the hard-coded 522-bit RSA key size used for tests with a TEST_RSA_KEY_SIZE constant defined in test/common.py (part 2). refs #393
david-sarah [Wed, 10 Aug 2011 20:22:43 +0000 (13:22 -0700)]
Replace the hard-coded 522-bit RSA key size used for tests with a TEST_RSA_KEY_SIZE constant defined in test/common.py (part 1). refs #393
Zooko O'Whielacronx [Wed, 10 Aug 2011 17:21:39 +0000 (10:21 -0700)]
merge some minor conflicts in test code from the 393-2 branch and trunk
Zooko O'Whielacronx [Tue, 9 Aug 2011 13:26:01 +0000 (06:26 -0700)]
doc: eliminate the phrase "rootcap" from doc/frontends/FTP-and-SFTP.rst
Two different people have asked me for help, saying they couldn't figure out what a "rootcap" is. Hopefully just calling it a "cap" will make it easier for them to find out from the other docs what it is.
david-sarah [Mon, 8 Aug 2011 21:39:25 +0000 (14:39 -0700)]
test_web.py: fix a test failure dependent on whether simplejson.loads returns a unicode or str object.
Kevan Carstensen [Sun, 7 Aug 2011 00:45:14 +0000 (17:45 -0700)]
immutable/filenode: fix pyflakes warnings
Kevan Carstensen [Sun, 7 Aug 2011 00:44:59 +0000 (17:44 -0700)]
test: fix assorted tests broken by MDMF changes
Kevan Carstensen [Sun, 7 Aug 2011 00:44:36 +0000 (17:44 -0700)]
uri: add MDMF and MDMF directory caps, add extension hint support
Kevan Carstensen [Sun, 7 Aug 2011 00:44:14 +0000 (17:44 -0700)]
test/test_mutable: tests for MDMF
These are their own patch because they cut across a lot of the changes
I've made in implementing MDMF in such a way as to make it difficult to
split them up into the other patches.
Kevan Carstensen [Sun, 7 Aug 2011 00:43:48 +0000 (17:43 -0700)]
webapi changes for MDMF
- Learn how to create MDMF files and directories through the
mutable-type argument.
- Operate with the interface changes associated with MDMF and #993.
- Learn how to do partial updates of mutable files.
Kevan Carstensen [Sun, 7 Aug 2011 00:42:59 +0000 (17:42 -0700)]
mutable/servermap: Rework the servermap to work with MDMF mutable files
Kevan Carstensen [Sun, 7 Aug 2011 00:42:24 +0000 (17:42 -0700)]
dirnode: teach dirnode to make MDMF directories
david-sarah [Tue, 2 Aug 2011 03:28:10 +0000 (20:28 -0700)]
Fix some test failures caused by #393 patch.
Kevan Carstensen [Tue, 2 Aug 2011 02:20:56 +0000 (19:20 -0700)]
docs: amend configuration, webapi documentation to talk about MDMF
Kevan Carstensen [Tue, 2 Aug 2011 02:16:13 +0000 (19:16 -0700)]
cli: teach CLI how to create MDMF mutable files
Specifically, 'tahoe mkdir' and 'tahoe put' now take a --mutable-type
argument.
Kevan Carstensen [Tue, 2 Aug 2011 02:12:07 +0000 (19:12 -0700)]
frontends/sftpd: Resolve incompatibilities between SFTP frontend and MDMF changes
Kevan Carstensen [Tue, 2 Aug 2011 02:11:20 +0000 (19:11 -0700)]
mutable/layout: Define MDMF share format, write tools for working with MDMF share format
The changes in layout.py are mostly concerned with the MDMF share
format. In particular, we define read and write proxy objects used by
retrieval, publishing, and other code to write and read the MDMF share
format. We create equivalent proxies for SDMF objects so that these
objects can be suitably general.
Kevan Carstensen [Tue, 2 Aug 2011 02:09:05 +0000 (19:09 -0700)]
immutable/filenode: implement unified filenode interface
Kevan Carstensen [Tue, 2 Aug 2011 02:08:14 +0000 (19:08 -0700)]
immutable/literal.py: Implement interface changes in literal nodes.
Kevan Carstensen [Tue, 2 Aug 2011 01:56:43 +0000 (18:56 -0700)]
test/common: Alter common test code to work with MDMF.
This mostly has to do with making the test code implement the new
unified filenode interfaces.
Kevan Carstensen [Tue, 2 Aug 2011 01:51:40 +0000 (18:51 -0700)]
mutable: train checker and repairer to work with MDMF mutable files
Kevan Carstensen [Tue, 2 Aug 2011 01:49:26 +0000 (18:49 -0700)]
nodemaker: teach nodemaker about MDMF caps
Kevan Carstensen [Tue, 2 Aug 2011 01:48:11 +0000 (18:48 -0700)]
client: teach client how to create and work with MDMF files
Kevan Carstensen [Tue, 2 Aug 2011 01:45:01 +0000 (18:45 -0700)]
mutable/filenode: Modify mutable filenodes for use with MDMF
In particular:
- Break MutableFileNode and MutableFileVersion into distinct classes.
- Implement the interface modifications made for MDMF.
- Be aware of MDMF caps.
- Learn how to create and work with MDMF files.
Kevan Carstensen [Tue, 2 Aug 2011 01:42:58 +0000 (18:42 -0700)]
nodemaker: teach nodemaker how to create MDMF mutable files
Kevan Carstensen [Tue, 2 Aug 2011 01:41:19 +0000 (18:41 -0700)]
interfaces: change interfaces to work with MDMF
A lot of this work concerns #993, in that it unifies (to an extent) the
interfaces of mutable and immutable files.
Kevan Carstensen [Tue, 2 Aug 2011 01:39:31 +0000 (18:39 -0700)]
mutable/publish: teach the publisher how to publish MDMF mutable files
Like the downloader, the publisher needs some substantial changes to handle multiple segment mutable files.
Kevan Carstensen [Tue, 2 Aug 2011 01:35:24 +0000 (18:35 -0700)]
mutable/retrieve: rework the mutable downloader to handle multiple-segment files
The downloader needs substantial reworking to handle multiple segment
mutable files, which it needs to handle for MDMF.
david-sarah [Fri, 19 Aug 2011 20:48:36 +0000 (13:48 -0700)]
Fix repeated 'the' in license text.
david-sarah [Fri, 19 Aug 2011 20:47:42 +0000 (13:47 -0700)]
Remove Non-Profit Open Software License from the set of 'added permission' licenses. Although it actually does qualify as an Open Source license (because it allows relicensing under plain OSL), its wording is unclear and could easily be misunderstood, and it contributes to incompatible license proliferation.
Zooko O'Whielacronx [Fri, 19 Aug 2011 06:01:42 +0000 (23:01 -0700)]
docs: change links that pointed to COPYING.TGPPL.html to point to COPYING.TGPPL.rst instead
Zooko O'Whielacronx [Fri, 19 Aug 2011 06:01:10 +0000 (23:01 -0700)]
docs: formatting: reflow to fill-column 77
Zooko O'Whielacronx [Fri, 19 Aug 2011 06:00:41 +0000 (23:00 -0700)]
docs: formatting: M-x whitespace-cleanup
Zooko O'Whielacronx [Fri, 19 Aug 2011 05:46:56 +0000 (22:46 -0700)]
licensing: add to the list of licenses that we grant the added permission for
Added: most of the ones listed on the FSF's "List of Free Software, GPL Incompatible Licenses", plus the Non-Profit Open Software License.
Zooko O'Whielacronx [Fri, 19 Aug 2011 05:30:59 +0000 (22:30 -0700)]
docs: reflow the added text at the top of COPYING.GPL to fill-column 77
Zooko O'Whielacronx [Fri, 19 Aug 2011 05:27:53 +0000 (22:27 -0700)]
docs: reformat COPYING.TGPPL.html to COPYING.TGPPL.rst
Zooko O'Whielacronx [Fri, 19 Aug 2011 04:41:03 +0000 (21:41 -0700)]
docs: reflow docs/logging.rst to fill-column 77
Zooko O'Whielacronx [Fri, 19 Aug 2011 04:39:46 +0000 (21:39 -0700)]
doc: fix formatting error in docs/logging.rst
david-sarah [Thu, 18 Aug 2011 04:07:49 +0000 (21:07 -0700)]
Cleanups for suppression of UserWarnings. refs #1435
Zooko O'Whielacronx [Wed, 17 Aug 2011 20:31:34 +0000 (13:31 -0700)]
suppress warning emitted by newer zope.interface with Nevow 0.10
refs #1435
Zooko O'Whielacronx [Tue, 9 Aug 2011 13:25:10 +0000 (06:25 -0700)]
doc: formatting: reflow to fill-column=77
david-sarah [Mon, 15 Aug 2011 02:53:47 +0000 (19:53 -0700)]
_auto_deps.py: change the requirement for zope.interface to <= 3.6.2, >= 3.6.6. fixes #1435
david-sarah [Mon, 15 Aug 2011 03:51:53 +0000 (20:51 -0700)]
allmydata/__init__.py, test_version.py: make version parsing understand '<=', with test. refs #1435
david-sarah [Mon, 1 Aug 2011 03:19:52 +0000 (20:19 -0700)]
Makefile and setup.py: remove setup.py commands that we no longer need, and their uses in the Makefile. Delete a stale and incorrect comment about updating _version.py. Also fix some coding style checks in the Makefile to operate on all source files.
Brian Warner [Thu, 11 Aug 2011 18:27:05 +0000 (11:27 -0700)]
remove misc/debian[_helpers], rely upon official packaging instead. fixes #1454
david-sarah [Mon, 1 Aug 2011 03:18:57 +0000 (20:18 -0700)]
Makefile: remove targets that used misc/debian[_helpers] which no longer exist. Also change docs/debian.rst to reflect the fact that we no longer support building .debs using those targets. refs #1454
Brian Warner [Thu, 11 Aug 2011 17:37:04 +0000 (10:37 -0700)]
replace tabs with spaces in the #1441 'tahoe debug' synopsis
david-sarah [Sun, 24 Jul 2011 16:25:30 +0000 (09:25 -0700)]
Correct the information printed by '/usr/bin/tahoe debug --help' on Debian/Ubuntu. fixes #1441
Zooko O'Whielacronx [Thu, 4 Aug 2011 19:34:09 +0000 (12:34 -0700)]
doc: edit the explanation of K-of-N tradeoffs
Zooko O'Whielacronx [Thu, 4 Aug 2011 19:27:22 +0000 (12:27 -0700)]
doc: clean up formatting of doc/configuration.rst
reflow to 77 chars line width, M-x white-space cleanup, blank link between name and definition
david-sarah [Sun, 24 Jul 2011 21:16:59 +0000 (14:16 -0700)]
Add test for webopen. fixes #1149
david-sarah [Wed, 10 Aug 2011 05:25:04 +0000 (22:25 -0700)]
test_client.py: relax a check in test_create_drop_uploader so that it should pass on Python 2.4.x. refs #1429
david-sarah [Wed, 10 Aug 2011 05:19:06 +0000 (22:19 -0700)]
test/common_util.py: correct fix to mkdir_nonascii. refs #1472
david-sarah [Wed, 10 Aug 2011 04:42:35 +0000 (21:42 -0700)]
test/common_util.py: fix a typo. refs #1472
david-sarah [Wed, 10 Aug 2011 03:45:05 +0000 (20:45 -0700)]
test_client.py, test_drop_upload.py: fix pyflakes warnings.