immutable: invent download.BadOrMissingHashError which is raised if either hashtree.BadHashError, hashtree.NotEnoughHashesError, and which is a subclass of IntegrityCheckReject
dirnode: don't check MAC on entries in dirnodes
In an ancient version of directories, we needed a MAC on each entry. In modern times, the entire dirnode comes with a digital signature, so the MAC on each entry is redundant.
With this patch, we no longer check those MACs when reading directories, but we still produce them so that older readers will accept directories that we write.
cli: if response code from wapi server is not 200 then stop instead of proceeding
Also, include the data that failed to json parse in an exception raised by the json parser.
immutable: when downloading an immutable file, use primary shares if they are available
Primary shares require no erasure decoding so the more primary shares you have, the less CPU is used.
try to tidy up uri-as-string vs. uri-as-object
I get confused about whether a given argument or return value is a uri-as-string or uri-as-object. This patch adds a lot of assertions that it is one or the other, and also changes CheckerResults to take objects not strings.
In the future, I hope that we generally use Python objects except when importing into or exporting from the Python interpreter e.g. over the wire, the UI, or a stored file.
pyutil: assertutil: simplify handling of exception during formatting of precondition message, and reduce dependency to just the Python Standard Library's logging module
Brian Warner [Wed, 10 Dec 2008 05:44:49 +0000 (22:44 -0700)]
mutable publish: if we are surprised by shares that match what we would have written anyways, don't be surprised. This should fix one of the two #546 problems, in which we re-use a server and forget that we already sent them a share.
setup: loosen requirement on simplejson to >= 1.4
That's the version of simplejson that comes with ubuntu feisty, and the one that we've required for most of our history. Currently the Ubuntu dapper buildslave fails (see issue #534), and setting the simplejson requirement to be >= 2.0 would fix that failure, but I don't understand why.
setup: require simplejson >= 1.7.1
That's the version that comes with gutsy, and we don't really understand why increasing the required version number helped with issue #553.
mutable: rename mutable/node.py to mutable/filenode.py and mutable/repair.py to mutable/repairer.py
To be more consistent with the immutable layout that I am working on.
francois [Tue, 18 Nov 2008 14:41:35 +0000 (07:41 -0700)]
filenode.py: Fix partial HTTP Range header handling according to RFC2616
Tahoe webapi was failing on HTTP request containing a partial Range header.
This change allows movies players like mplayer to seek in movie files stored in
tahoe.
Associated tests for GET and HEAD methods are also included
web: "More Info" link describes the same file that the "file" link points to, rather than to the file under the same name in this directory
It's a subtle but real difference.
Fixes #553 -- "More Info" link should point to a file/dir, not a dir+childname .
download: refactor handling of URI Extension Block and crypttext hash tree, simplify things
Refactor into a class the logic of asking each server in turn until one of them gives an answer
that validates. It is called ValidatedThingObtainer.
Refactor the downloading and verification of the URI Extension Block into a class named
ValidatedExtendedURIProxy.
The new logic of validating UEBs is minimalist: it doesn't require the UEB to contain any
unncessary information, but of course it still accepts such information for backwards
compatibility (so that this new download code is able to download files uploaded with old, and
for that matter with current, upload code).
The new logic of validating UEBs follows the practice of doing all validation up front. This
practice advises one to isolate the validation of incoming data into one place, so that all of
the rest of the code can assume only valid data.
If any redundant information is present in the UEB+URI, the new code cross-checks and asserts
that it is all fully consistent. This closes some issues where the uploader could have
uploaded inconsistent redundant data, which would probably have caused the old downloader to
simply reject that download after getting a Python exception, but perhaps could have caused
greater harm to the old downloader.
I removed the notion of selecting an erasure codec from codec.py based on the string that was
passed in the UEB. Currently "crs" is the only such string that works, so
"_assert(codec_name == 'crs')" is simpler and more explicit. This is also in keeping with the
"validate up front" strategy -- now if someone sets a different string than "crs" in their UEB,
the downloader will reject the download in the "validate this UEB" function instead of in a
separate "select the codec instance" function.
I removed the code to check plaintext hashes and plaintext Merkle Trees. Uploaders do not
produce this information any more (since it potentially exposes confidential information about
the file), and the unit tests for it were disabled. The downloader before this patch would
check that plaintext hash or plaintext merkle tree if they were present, but not complain if
they were absent. The new downloader in this patch complains if they are present and doesn't
check them. (We might in the future re-introduce such hashes over the plaintext, but encrypt
the hashes which are stored in the UEB to preserve confidentiality. This would be a double-
check on the correctness of our own source code -- the current Merkle Tree over the ciphertext
is already sufficient to guarantee the integrity of the download unless there is a bug in our
Merkle Tree or AES implementation.)
This patch increases the lines-of-code count by 8 (from 17,770 to 17,778), and reduces the
uncovered-by-tests lines-of-code count by 24 (from 1408 to 1384). Those numbers would be more
meaningful if we omitted src/allmydata/util/ from the test-coverage statistics.
Brian Warner [Thu, 4 Dec 2008 22:00:53 +0000 (15:00 -0700)]
test_system.py: don't ask the stats-gatherer to poll: it tolerates failures, so it isn't really giving us enough test coverage. Removing the call will make it more clear that we need to improve the tests later
setup: move the requirement on simplejson from setup.py to _auto_deps.py, and loosen it from >= 2.0.5 to > 1.8.1
We'll see if this fixes the tests on all of our current buildslaves, and if it does then I'll be happy to leave it at "> 1.8.1" for now, even though I don't know exactly what versions of simplejson changed exactly what behavior that interacts with exactly what environment. See http://allmydata.org/trac/tahoe/ticket/534 for uncertainties.
Brian Warner [Tue, 25 Nov 2008 21:27:35 +0000 (14:27 -0700)]
mutable publish: reinstate the foolscap-reference-token-bug workaround, both for the original reasons and because of an apparent new foolscap bug that's triggered by reference tokens. See #541 for details.
Brian Warner [Mon, 24 Nov 2008 21:40:46 +0000 (14:40 -0700)]
dirnode manifest: add verifycaps, both to internal API and to webapi. This will give the manual-GC tools more to work with, so they can estimate how much space will be freed.
Brian Warner [Sat, 22 Nov 2008 03:28:12 +0000 (20:28 -0700)]
upload: don't use servers which can't support the share size we need. This ought to avoid #439 problems. Some day we'll have a storage server which advertises support for a larger share size. No tests yet.
setup: use "setup.py develop --multi-version" so that if there is a too-old version of a dependency installed this doesn't prevent Tahoe's "develop" and run-in-place from working
setup: we require setuptools > 0.6a9 in order to parse requirements that have a dot in them such as "zope.interface"
In the near future we might start actually relying on setuptools's pkg_resources's "require()" function to make modules importable, so we can't just skip zope.interface.
projects / tahoe-lafs / tahoe-lafs.git / log