Zooko O'Whielacronx [Wed, 17 Dec 2008 01:04:50 +0000 (18:04 -0700)]
immutable: use new logging mixins to simplify logging
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:53:25 +0000 (17:53 -0700)]
immutable: refactor ReadBucketProxy a little
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:51:45 +0000 (17:51 -0700)]
debug: pass empty optional arguments to ReadBucketProxy
because those arguments are about to become non-optional (for other code than test/debug code)
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:49:30 +0000 (17:49 -0700)]
uri: generalize regexp that recognizes tahoe URLs to work for any host and port
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:38:07 +0000 (17:38 -0700)]
util: logging: refactor some common logging behavior into mixins
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:37:45 +0000 (17:37 -0700)]
pyutil: assertutil: copy in simplified assertutil from pyutil
Zooko O'Whielacronx [Wed, 10 Dec 2008 14:10:57 +0000 (07:10 -0700)]
pyutil: assertutil: simplify handling of exception during formatting of precondition message, and reduce dependency to just the Python Standard Library's logging module
Zooko O'Whielacronx [Tue, 9 Dec 2008 00:04:00 +0000 (17:04 -0700)]
client: add get_servers()
Brian Warner [Wed, 10 Dec 2008 05:44:49 +0000 (22:44 -0700)]
mutable publish: if we are surprised by shares that match what we would have written anyways, don't be surprised. This should fix one of the two #546 problems, in which we re-use a server and forget that we already sent them a share.
Brian Warner [Wed, 10 Dec 2008 00:11:46 +0000 (17:11 -0700)]
NEWS: updated to most recent user-visible changes, including the 8123-to-3456 change
Zooko O'Whielacronx [Tue, 9 Dec 2008 23:45:46 +0000 (16:45 -0700)]
immutable: remove unused code to produce plaintext hashes
Zooko O'Whielacronx [Tue, 9 Dec 2008 23:33:18 +0000 (16:33 -0700)]
finish renaming 'subshare' to 'block' in immutable/encode.py and in docs/
Zooko O'Whielacronx [Tue, 9 Dec 2008 00:16:34 +0000 (17:16 -0700)]
introducer: fix bug in recent simplification caught by Brian's sharp code-reviewing eye
Zooko O'Whielacronx [Mon, 8 Dec 2008 23:57:25 +0000 (16:57 -0700)]
introducer: simplify get_permuted_peers() implementation and add get_peers()
Brian Warner [Mon, 8 Dec 2008 22:32:56 +0000 (15:32 -0700)]
webapi.txt: minor edits
Zooko O'Whielacronx [Mon, 8 Dec 2008 19:44:11 +0000 (12:44 -0700)]
rename "get_verifier()" to "get_verify_cap()"
Zooko O'Whielacronx [Mon, 8 Dec 2008 18:47:25 +0000 (11:47 -0700)]
setup: try depending on setuptools >= 0.6c6 instead of >= 0.6c7 at run-time, to be able to use the setuptools that came with Ubuntu Gutsy
Zooko O'Whielacronx [Mon, 8 Dec 2008 15:35:37 +0000 (08:35 -0700)]
setup: loosen requirement on simplejson to >= 1.4
That's the version of simplejson that comes with ubuntu feisty, and the one that we've required for most of our history. Currently the Ubuntu dapper buildslave fails (see issue #534), and setting the simplejson requirement to be >= 2.0 would fix that failure, but I don't understand why.
Zooko O'Whielacronx [Mon, 8 Dec 2008 05:34:12 +0000 (22:34 -0700)]
setup: require simplejson >= 1.7.1
That's the version that comes with gutsy, and we don't really understand why increasing the required version number helped with issue #553.
Zooko O'Whielacronx [Sun, 7 Dec 2008 15:45:19 +0000 (08:45 -0700)]
mutable: merge renaming with test patches
Zooko O'Whielacronx [Sun, 7 Dec 2008 15:20:08 +0000 (08:20 -0700)]
mutable: rename mutable/node.py to mutable/filenode.py and mutable/repair.py to mutable/repairer.py
To be more consistent with the immutable layout that I am working on.
Brian Warner [Sat, 6 Dec 2008 06:14:12 +0000 (23:14 -0700)]
web/directory.py: really really fix #553. Unfortunately it's tricky to simulate the behavior of a brower's relative-url handling in a unit test.
francois [Tue, 18 Nov 2008 14:41:35 +0000 (07:41 -0700)]
filenode.py: Fix partial HTTP Range header handling according to RFC2616
Tahoe webapi was failing on HTTP request containing a partial Range header.
This change allows movies players like mplayer to seek in movie files stored in
tahoe.
Associated tests for GET and HEAD methods are also included
Brian Warner [Sat, 6 Dec 2008 05:49:23 +0000 (22:49 -0700)]
mutable.modify(): after UCWE, publish even if the second invocation of the modifier didn't modify anything. For #551.
Brian Warner [Sat, 6 Dec 2008 05:08:37 +0000 (22:08 -0700)]
dirnode.py: dirnode.delete which hits UCWE should not fail with NoSuchChildError. Fixes #550.
Brian Warner [Sat, 6 Dec 2008 05:07:10 +0000 (22:07 -0700)]
MutableFileNode.modify: pass first_time= and servermap= to the modifier callback
Brian Warner [Fri, 5 Dec 2008 22:54:12 +0000 (15:54 -0700)]
misc/cpu-watcher.tac: tolerate disk-full errors when writing the pickle, and pickle corruption from earlier disk-full errors
Zooko O'Whielacronx [Fri, 5 Dec 2008 22:39:39 +0000 (15:39 -0700)]
web: fix more info links again
Really, *really* closes #553.
Zooko O'Whielacronx [Fri, 5 Dec 2008 22:29:39 +0000 (15:29 -0700)]
web: fix moreinfo link
Zooko O'Whielacronx [Fri, 5 Dec 2008 22:05:02 +0000 (15:05 -0700)]
web: "More Info" link describes the same file that the "file" link points to, rather than to the file under the same name in this directory
It's a subtle but real difference.
Fixes #553 -- "More Info" link should point to a file/dir, not a dir+childname .
Zooko O'Whielacronx [Fri, 5 Dec 2008 20:07:23 +0000 (13:07 -0700)]
minor: fix unused imports -- thanks, pyflakes
Zooko O'Whielacronx [Fri, 5 Dec 2008 15:17:54 +0000 (08:17 -0700)]
download: refactor handling of URI Extension Block and crypttext hash tree, simplify things
Refactor into a class the logic of asking each server in turn until one of them gives an answer
that validates. It is called ValidatedThingObtainer.
Refactor the downloading and verification of the URI Extension Block into a class named
ValidatedExtendedURIProxy.
The new logic of validating UEBs is minimalist: it doesn't require the UEB to contain any
unncessary information, but of course it still accepts such information for backwards
compatibility (so that this new download code is able to download files uploaded with old, and
for that matter with current, upload code).
The new logic of validating UEBs follows the practice of doing all validation up front. This
practice advises one to isolate the validation of incoming data into one place, so that all of
the rest of the code can assume only valid data.
If any redundant information is present in the UEB+URI, the new code cross-checks and asserts
that it is all fully consistent. This closes some issues where the uploader could have
uploaded inconsistent redundant data, which would probably have caused the old downloader to
simply reject that download after getting a Python exception, but perhaps could have caused
greater harm to the old downloader.
I removed the notion of selecting an erasure codec from codec.py based on the string that was
passed in the UEB. Currently "crs" is the only such string that works, so
"_assert(codec_name == 'crs')" is simpler and more explicit. This is also in keeping with the
"validate up front" strategy -- now if someone sets a different string than "crs" in their UEB,
the downloader will reject the download in the "validate this UEB" function instead of in a
separate "select the codec instance" function.
I removed the code to check plaintext hashes and plaintext Merkle Trees. Uploaders do not
produce this information any more (since it potentially exposes confidential information about
the file), and the unit tests for it were disabled. The downloader before this patch would
check that plaintext hash or plaintext merkle tree if they were present, but not complain if
they were absent. The new downloader in this patch complains if they are present and doesn't
check them. (We might in the future re-introduce such hashes over the plaintext, but encrypt
the hashes which are stored in the UEB to preserve confidentiality. This would be a double-
check on the correctness of our own source code -- the current Merkle Tree over the ciphertext
is already sufficient to guarantee the integrity of the download unless there is a bug in our
Merkle Tree or AES implementation.)
This patch increases the lines-of-code count by 8 (from 17,770 to 17,778), and reduces the
uncovered-by-tests lines-of-code count by 24 (from 1408 to 1384). Those numbers would be more
meaningful if we omitted src/allmydata/util/ from the test-coverage statistics.
Brian Warner [Fri, 5 Dec 2008 09:12:10 +0000 (02:12 -0700)]
test_web: add get_permuted_peers, to unbreak recent checker_results change
Brian Warner [Fri, 5 Dec 2008 09:03:09 +0000 (02:03 -0700)]
web checker_results: include a table of servers in permuted order, so you can see the places where new servers have been inserted
Brian Warner [Fri, 5 Dec 2008 00:27:04 +0000 (17:27 -0700)]
test_system.py: assert less about the stats we get, since shares (and thus allocate() calls) are distributed randomly
Brian Warner [Thu, 4 Dec 2008 22:01:24 +0000 (15:01 -0700)]
stats: don't return booleans: it violates the schema. Add a test.
Brian Warner [Thu, 4 Dec 2008 22:00:53 +0000 (15:00 -0700)]
test_system.py: don't ask the stats-gatherer to poll: it tolerates failures, so it isn't really giving us enough test coverage. Removing the call will make it more clear that we need to improve the tests later
Peter Secor [Wed, 3 Dec 2008 03:38:31 +0000 (20:38 -0700)]
confwiz.py - removing hardcoded version number
Brian Warner [Wed, 3 Dec 2008 03:20:22 +0000 (20:20 -0700)]
CLI: check for pre-existing aliases in 'tahoe create-alias' and 'tahoe add-alias'
Brian Warner [Wed, 3 Dec 2008 03:08:28 +0000 (20:08 -0700)]
test_cli: pass rc out of do_cli() too
Zooko O'Whielacronx [Wed, 3 Dec 2008 02:50:40 +0000 (19:50 -0700)]
setup: one more address to send release announcements to
Zooko O'Whielacronx [Wed, 3 Dec 2008 02:44:14 +0000 (19:44 -0700)]
setup: another note about the process of making a tahoe release: mail to duplicity-talk@nongnu.org
Brian Warner [Wed, 3 Dec 2008 02:41:02 +0000 (19:41 -0700)]
test_storage.py: more windows-vs-readonly-storage fixes
Brian Warner [Wed, 3 Dec 2008 02:07:26 +0000 (19:07 -0700)]
docs/webapi.txt: update helper section to discuss tahoe.cfg
Brian Warner [Wed, 3 Dec 2008 02:06:12 +0000 (19:06 -0700)]
docs/webapi.txt: update to discuss tahoe.cfg, not BASEDIR/webport
Brian Warner [Wed, 3 Dec 2008 02:03:17 +0000 (19:03 -0700)]
storage.py: oops, fix windows again, readonly_storage wasn't getting picked up properly
Brian Warner [Wed, 3 Dec 2008 01:31:26 +0000 (18:31 -0700)]
test_download.py: remove extra base32 import
Brian Warner [Wed, 3 Dec 2008 01:30:07 +0000 (18:30 -0700)]
test_download: test both mutable and immutable pre-generated shares
Brian Warner [Wed, 3 Dec 2008 01:22:08 +0000 (18:22 -0700)]
test_download.py: added 'known-answer-tests', to make sure current code can download a file that was created by earlier code
Brian Warner [Tue, 2 Dec 2008 22:51:01 +0000 (15:51 -0700)]
docs/configuration.txt: fix minor typo
Brian Warner [Tue, 2 Dec 2008 02:49:46 +0000 (19:49 -0700)]
storage.py: unbreak readonly_storage=True on windows
Brian Warner [Tue, 2 Dec 2008 00:47:21 +0000 (17:47 -0700)]
#542 'tahoe create-key-generator': fix the .tac file this creates to be compatible with modern code, add a test
Brian Warner [Tue, 2 Dec 2008 00:25:40 +0000 (17:25 -0700)]
storage.py: fix minor typo in comment
Brian Warner [Tue, 2 Dec 2008 00:24:21 +0000 (17:24 -0700)]
storage: replace sizelimit with reserved_space, make the stats 'disk_avail' number incorporate this reservation
Brian Warner [Tue, 2 Dec 2008 00:24:12 +0000 (17:24 -0700)]
util/abbreviate: add abbreviated-size parser
Zooko O'Whielacronx [Wed, 26 Nov 2008 00:57:37 +0000 (17:57 -0700)]
wui/wapi: change the default port number from 8123 to 3456 to avoid conflict with TorButton
See ticket #536 for details.
Zooko O'Whielacronx [Tue, 25 Nov 2008 21:37:51 +0000 (14:37 -0700)]
setup: move the requirement on simplejson from setup.py to _auto_deps.py, and loosen it from >= 2.0.5 to > 1.8.1
We'll see if this fixes the tests on all of our current buildslaves, and if it does then I'll be happy to leave it at "> 1.8.1" for now, even though I don't know exactly what versions of simplejson changed exactly what behavior that interacts with exactly what environment. See http://allmydata.org/trac/tahoe/ticket/534 for uncertainties.
francois [Tue, 25 Nov 2008 18:17:27 +0000 (11:17 -0700)]
setup.py: Require simplejson version >= 2.0.5
Brian Warner [Tue, 25 Nov 2008 21:27:35 +0000 (14:27 -0700)]
mutable publish: reinstate the foolscap-reference-token-bug workaround, both for the original reasons and because of an apparent new foolscap bug that's triggered by reference tokens. See #541 for details.
Zooko O'Whielacronx [Tue, 25 Nov 2008 16:55:28 +0000 (09:55 -0700)]
setup: fix missing import -- thanks, pyflakes
Zooko O'Whielacronx [Tue, 25 Nov 2008 16:51:18 +0000 (09:51 -0700)]
setup: correctly detect Arch Linux in platform description
Brian Warner [Mon, 24 Nov 2008 21:40:46 +0000 (14:40 -0700)]
dirnode manifest: add verifycaps, both to internal API and to webapi. This will give the manual-GC tools more to work with, so they can estimate how much space will be freed.
Brian Warner [Sun, 23 Nov 2008 06:13:23 +0000 (23:13 -0700)]
control.py: use get_buckets() instead of get_version() to measure ping time, because the latter changed recently
Brian Warner [Sat, 22 Nov 2008 03:29:32 +0000 (20:29 -0700)]
upload: when using a Helper, insist that it provide protocols/helper/v1 . Related to #538.
Brian Warner [Sat, 22 Nov 2008 03:28:12 +0000 (20:28 -0700)]
upload: don't use servers which can't support the share size we need. This ought to avoid #439 problems. Some day we'll have a storage server which advertises support for a larger share size. No tests yet.
Brian Warner [Sat, 22 Nov 2008 03:07:27 +0000 (20:07 -0700)]
#538: fetch version and attach to the rref. Make IntroducerClient demand v1 support.
Brian Warner [Sat, 22 Nov 2008 00:43:52 +0000 (17:43 -0700)]
#538: add remote_get_version() to four main Referenceable objects: Introducer Service, Storage Server, Helper, CHK Upload Helper. Remove unused storage-server get_versions().
Zooko O'Whielacronx [Fri, 21 Nov 2008 05:36:45 +0000 (22:36 -0700)]
setup: turn off --multi-version until I can figure out why it breaks test_runner
Zooko O'Whielacronx [Fri, 21 Nov 2008 05:36:11 +0000 (22:36 -0700)]
setup: require setuptools >= 0.6c7 to run
Zooko O'Whielacronx [Thu, 20 Nov 2008 21:15:45 +0000 (14:15 -0700)]
setup: use "setup.py develop --multi-version" so that if there is a too-old version of a dependency installed this doesn't prevent Tahoe's "develop" and run-in-place from working
Zooko O'Whielacronx [Thu, 20 Nov 2008 16:15:03 +0000 (09:15 -0700)]
setup: we require setuptools > 0.6a9 in order to parse requirements that have a dot in them such as "zope.interface"
In the near future we might start actually relying on setuptools's pkg_resources's "require()" function to make modules importable, so we can't just skip zope.interface.
Brian Warner [Wed, 19 Nov 2008 23:02:12 +0000 (16:02 -0700)]
test_dirnode: add an explainError call
Brian Warner [Wed, 19 Nov 2008 23:00:27 +0000 (16:00 -0700)]
manifest: add storage-index strings to the json results
Brian Warner [Wed, 19 Nov 2008 22:03:47 +0000 (15:03 -0700)]
manifest: include stats in results. webapi is unchanged.
Brian Warner [Wed, 19 Nov 2008 21:05:52 +0000 (14:05 -0700)]
misc/spacetime/diskwatcher.tac: remove dead code
Brian Warner [Wed, 19 Nov 2008 21:05:01 +0000 (14:05 -0700)]
mutable: respect the new tahoe.cfg 'shares.needed' and 'shares.total' settings
Brian Warner [Wed, 19 Nov 2008 03:32:59 +0000 (20:32 -0700)]
oops, update tests to match 'tahoe stats' change
Brian Warner [Wed, 19 Nov 2008 03:28:16 +0000 (20:28 -0700)]
cli: tahoe stats: abbreviate total sizes too
Brian Warner [Wed, 19 Nov 2008 03:17:36 +0000 (20:17 -0700)]
cli: 'tahoe stats': add abbreviated size to the histogram. Not sure this actually improves things.
Brian Warner [Wed, 19 Nov 2008 03:11:42 +0000 (20:11 -0700)]
util/abbreviate: little utility to abbreviate seconds and bytes
Brian Warner [Wed, 19 Nov 2008 02:12:10 +0000 (19:12 -0700)]
cli: add 'tahoe check' and 'tahoe deep-check' commands, with primitive reporting code
Brian Warner [Wed, 19 Nov 2008 02:11:13 +0000 (19:11 -0700)]
cli: factor out slow-http-operation to a separate module
Brian Warner [Wed, 19 Nov 2008 01:36:08 +0000 (18:36 -0700)]
cli: tahoe stats/manifest: change --verbose to --raw, since I want -v for --verify for check/deep-check/repair
Brian Warner [Wed, 19 Nov 2008 01:29:50 +0000 (18:29 -0700)]
test_system: make 'where' strings more helpful, to track down test failures better
Brian Warner [Wed, 19 Nov 2008 01:28:26 +0000 (18:28 -0700)]
webapi: add 'summary' string to checker results JSON
Brian Warner [Tue, 18 Nov 2008 22:44:31 +0000 (15:44 -0700)]
munin/tahoe_disktotal: add a 'disk used' line, since it will always be less than disktotal
Brian Warner [Tue, 18 Nov 2008 22:32:38 +0000 (15:32 -0700)]
munin/tahoe_introstats: add line for distinct-storage-hosts (which counts machines instead of nodes)
Brian Warner [Tue, 18 Nov 2008 22:30:15 +0000 (15:30 -0700)]
webapi: introducer stats: add 'announcement_distinct_hosts' to the t=json form, to show how many distinct hosts are providing e.g. storage services
Brian Warner [Tue, 18 Nov 2008 08:47:58 +0000 (01:47 -0700)]
'tahoe create-key-generator': fix help text
Brian Warner [Tue, 18 Nov 2008 08:46:20 +0000 (01:46 -0700)]
#330: convert stats-gatherer into a .tac file service, add 'tahoe create-stats-gatherer'
Brian Warner [Tue, 18 Nov 2008 08:25:25 +0000 (01:25 -0700)]
munin/tahoe_diskused: new plugin to show total disk space used across the grid
Brian Warner [Tue, 18 Nov 2008 07:51:01 +0000 (00:51 -0700)]
munin/tahoe_disktotal: new plugin to show total disk space (used and unused) in the grid
Brian Warner [Tue, 18 Nov 2008 07:29:44 +0000 (00:29 -0700)]
tahoe.cfg: add controls for k and N (and shares-of-happiness)
Brian Warner [Tue, 18 Nov 2008 05:11:14 +0000 (22:11 -0700)]
cli: add tests for 'tahoe stats --verbose'
Brian Warner [Tue, 18 Nov 2008 05:02:19 +0000 (22:02 -0700)]
cli: add --verbose to 'tahoe manifest', to show the raw JSON data
Brian Warner [Tue, 18 Nov 2008 04:45:16 +0000 (21:45 -0700)]
diskwatcher: record total-space (the size of the disk as reported by df) in the db, report it to HTTP clients. This will involve a 50-item-per-second upgrade process when it is first used on old data
Brian Warner [Sat, 15 Nov 2008 05:50:49 +0000 (22:50 -0700)]
dirnode manifest/stats: process more than one LIT file per tree; we were accidentally ignoring all but the first
Brian Warner [Sat, 15 Nov 2008 04:11:44 +0000 (21:11 -0700)]
limiter.py: fix stack blowout by inserting an eventual-send between _done and maybe_start_task. This was causing failures during a 'tahoe manifest' of a large set of directories
francois [Fri, 14 Nov 2008 15:05:48 +0000 (08:05 -0700)]
New credit file entry
francois [Fri, 14 Nov 2008 14:44:58 +0000 (07:44 -0700)]
test_cli.py: Ensure that we can read our uploaded files back