]> git.rkrishnan.org Git - tahoe-lafs/tahoe-lafs.git/log
tahoe-lafs/tahoe-lafs.git
15 years agoimmutable: refactor download to do only download-and-decode, not decryption
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:53:49 +0000 (11:53 -0700)]
immutable: refactor download to do only download-and-decode, not decryption
FileDownloader takes a verify cap and produces ciphertext, instead of taking a read cap and producing plaintext.
FileDownloader does all integrity checking including the mandatory ciphertext hash tree and the optional ciphertext flat hash, rather than expecting its target to do some of that checking.
Rename immutable.download.Output to immutable.download.DecryptingOutput. An instance of DecryptingOutput can be passed to FileDownloader to use as the latter's target.  Text pushed to the DecryptingOutput is decrypted and then pushed to *its* target.
DecryptingOutput satisfies the IConsumer interface, and if its target also satisfies IConsumer, then it forwards and pause/unpause signals to its producer (which is the FileDownloader).
This patch also changes some logging code to use the new logging mixin class.
Check integrity of a segment and decrypt the segment one block-sized buffer at a time instead of copying the buffers together into one segment-sized buffer (reduces peak memory usage, I think, and is probably a tad faster/less CPU, depending on your encoding parameters).
Refactor FileDownloader so that processing of segments and of tail-segment share as much code is possible.
FileDownloader and FileNode take caps as instances of URI (Python objects), not as strings.

15 years agotrivial: tiny changes to test code
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:20:48 +0000 (11:20 -0700)]
trivial: tiny changes to test code

15 years agoimmutable: Make more parts of download use logging mixins and know what their "parent...
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:25:30 +0000 (11:25 -0700)]
immutable: Make more parts of download use logging mixins and know what their "parent msg id" is.

15 years agotrivial: M-x whitespace-cleanup on src/immutable/download.py
Zooko O'Whielacronx [Thu, 8 Jan 2009 17:49:01 +0000 (10:49 -0700)]
trivial: M-x whitespace-cleanup on src/immutable/download.py

15 years agoimmutable: ValidatedExtendedURIProxy computes and stores the tail data size as a...
Zooko O'Whielacronx [Thu, 8 Jan 2009 17:41:39 +0000 (10:41 -0700)]
immutable: ValidatedExtendedURIProxy computes and stores the tail data size as a convenience to its caller.
The "tail data size" is how many of the bytes of the tail segment are data (as opposed to padding).

15 years agoimmutable: define a new interface IImmutableFileURI and declare that CHKFileURI and...
Zooko O'Whielacronx [Wed, 7 Jan 2009 19:24:51 +0000 (12:24 -0700)]
immutable: define a new interface IImmutableFileURI and declare that CHKFileURI and LiteralFileURI provide it

15 years agoutil: log: allow empty msgs (because downloader is using the "format" alternative...
Zooko O'Whielacronx [Wed, 7 Jan 2009 18:54:11 +0000 (11:54 -0700)]
util: log: allow empty msgs (because downloader is using the "format" alternative with no "msg" argument)

15 years ago'tahoe cp -r', upon encountering a dangling symlink, would assert out.
Larry Hosken [Thu, 8 Jan 2009 06:51:14 +0000 (23:51 -0700)]
'tahoe cp -r', upon encountering a dangling symlink, would assert out.
This was somewhat sad; the assertion didn't say what path caused the
error, what went wrong.  So... silently skip over things that are
neither dirs nor files.

15 years agoimmutable: fix error in validation of ciphertext hash tree and add test for that...
Zooko O'Whielacronx [Thu, 8 Jan 2009 06:40:12 +0000 (23:40 -0700)]
immutable: fix error in validation of ciphertext hash tree and add test for that code
pyflakes pointed out to me that I had committed some code that is untested, since it uses an undefined name.  This patch exercises that code -- the validation of the ciphertext hash tree -- by corrupting some of the share files in a very specific way, and also fixes the bug.

15 years agoimmutable: do not catch arbitrary exceptions/failures from the attempt to get a crypt...
Zooko O'Whielacronx [Thu, 8 Jan 2009 05:25:51 +0000 (22:25 -0700)]
immutable: do not catch arbitrary exceptions/failures from the attempt to get a crypttext hash tree -- catch only ServerFailure, IntegrityCheckReject, LayoutInvalid, ShareVersionIncompatible, and DeadReferenceError
Once again I inserted a bug into the code, and once again it was hidden by something catching arbitrary exception/failure and assuming that it means the server failed to provide valid data.

15 years agodownload: make sure you really get all the crypttext hashes
Zooko O'Whielacronx [Thu, 8 Jan 2009 03:26:38 +0000 (20:26 -0700)]
download: make sure you really get all the crypttext hashes
We were not making sure that we really got all the crypttext hashes during download.  If a server were to return less than the complete set of crypttext hashes, then our subsequent attempt to verify the correctness of the ciphertext would fail.  (And it wouldn't be obvious without very careful debugging why it had failed.)
This patch makes it so that you keep trying to get ciphertext hashes until you have a full set or you run out of servers to ask.

15 years agoutil: deferredutil: undo my recent patch to use our own implementation of gatherResults
Zooko O'Whielacronx [Wed, 7 Jan 2009 18:00:05 +0000 (11:00 -0700)]
util: deferredutil: undo my recent patch to use our own implementation of gatherResults
It seems to cause lots of failures on some builders.

15 years agoutil: deferredutil: implement our own gatherResults instead of using Twisted's
Zooko O'Whielacronx [Wed, 7 Jan 2009 17:32:07 +0000 (10:32 -0700)]
util: deferredutil: implement our own gatherResults instead of using Twisted's
Because we want to maintain backwards compatibility to Twisted 2.4.0.

15 years agotrivial: M-x whitespace-cleanup
Zooko O'Whielacronx [Wed, 7 Jan 2009 17:25:28 +0000 (10:25 -0700)]
trivial: M-x whitespace-cleanup

15 years agoutil: deferredutil: add basic test for deferredutil.gatherResults
Zooko O'Whielacronx [Wed, 7 Jan 2009 15:13:42 +0000 (08:13 -0700)]
util: deferredutil: add basic test for deferredutil.gatherResults
Also I checked and Twisted 2.4.0 supports .subFailure and the other parts of the API that we require.

15 years agotrivial: fix redefinition of name "log" in imports (pyflakes)
Zooko O'Whielacronx [Wed, 7 Jan 2009 05:08:29 +0000 (22:08 -0700)]
trivial: fix redefinition of name "log" in imports (pyflakes)

15 years agoimmutable: refactor uploader to do just encoding-and-uploading, not encryption
Zooko O'Whielacronx [Wed, 7 Jan 2009 04:48:22 +0000 (21:48 -0700)]
immutable: refactor uploader to do just encoding-and-uploading, not encryption
This makes Uploader take an EncryptedUploadable object instead of an Uploadable object.  I also changed it to return a verify cap instead of a tuple of the bits of data that one finds in a verify cap.
This will facilitate hooking together an Uploader and a Downloader to make a Repairer.
Also move offloaded.py into src/allmydata/immutable/.

15 years agotrivial: whitespace and docstring tidyups
Zooko O'Whielacronx [Wed, 7 Jan 2009 04:41:04 +0000 (21:41 -0700)]
trivial: whitespace and docstring tidyups

15 years agostorage.py: explain what this large and hard-to-recognize 4294967295 number is
Brian Warner [Tue, 6 Jan 2009 20:57:21 +0000 (13:57 -0700)]
storage.py: explain what this large and hard-to-recognize 4294967295 number is

15 years agorename "checker results" to "check results", because it is more parallel to "check...
Zooko O'Whielacronx [Tue, 6 Jan 2009 20:37:03 +0000 (13:37 -0700)]
rename "checker results" to "check results", because it is more parallel to "check-and-repair results"

15 years agoimmutable: tests: verifier doesn't always catch corrupted share hashes
Zooko O'Whielacronx [Tue, 6 Jan 2009 20:04:49 +0000 (13:04 -0700)]
immutable: tests: verifier doesn't always catch corrupted share hashes
Maybe it already got one of the corrupted hashes from a different server and it doesn't double-check that the hash from every server is correct.  Or another problem.  But in any case I'm marking this as TODO because an even better (more picky) verifier is less urgent than repairer.

15 years agoimmutable: fix the writing of share data size into share file in case the share file...
Zooko O'Whielacronx [Tue, 6 Jan 2009 19:24:04 +0000 (12:24 -0700)]
immutable: fix the writing of share data size into share file in case the share file is used by a < v1.3.0 storage server
Brian noticed that the constant was wrong, and in fixing that I noticed that we should be saturating instead of modding.
This code would never matter unless a server downgraded or a share migrated from Tahoe >= v1.3.0 to Tahoe < v1.3.0.  Even in that case, this bug would never matter unless the share size were exactly 4,294,967,296 bytes long.
Brian, for good reason, wanted this to be spelled "2**32" instead of "4294967296", but I couldn't stand to see a couple of more Python bytecodes interpreted in the middle of a core, frequent operation on the server like immutable share creation.

15 years agotrivial: whitespace cleanup
Zooko O'Whielacronx [Tue, 6 Jan 2009 18:20:58 +0000 (11:20 -0700)]
trivial: whitespace cleanup

15 years agoutil: base32: require str-not-unicode inputs -- effectively rolls back [3306] and...
Zooko O'Whielacronx [Tue, 6 Jan 2009 17:41:22 +0000 (10:41 -0700)]
util: base32: require str-not-unicode inputs -- effectively rolls back [3306] and [3307]

15 years agotrivial: fix a bunch of pyflakes complaints
Zooko O'Whielacronx [Tue, 6 Jan 2009 15:00:54 +0000 (08:00 -0700)]
trivial: fix a bunch of pyflakes complaints

15 years agocli: make startstop_node wait 40 seconds instead of 20 for a process to go away after...
Zooko O'Whielacronx [Tue, 6 Jan 2009 14:51:06 +0000 (07:51 -0700)]
cli: make startstop_node wait 40 seconds instead of 20 for a process to go away after we signalled it to go away, before emitting a warning
Because the unit tests on the VirtualZooko? buildslave failed when it took 31 seconds for a process to go away.
Perhaps getting warning message after only 5 seconds instead of 40 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests. Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning.

15 years agoimmutable: new checker and verifier
Zooko O'Whielacronx [Tue, 6 Jan 2009 01:28:18 +0000 (18:28 -0700)]
immutable: new checker and verifier
New checker and verifier use the new download class.  They are robust against various sorts of failures or corruption.  They return detailed results explaining what they learned about your immutable files.  Some grotesque sorts of corruption are not properly handled yet, and those ones are marked as TODO or commented-out in the unit tests.
There is also a repairer module in this patch with the beginnings of a repairer in it.  That repairer is mostly just the interface to the outside world -- the core operation of actually reconstructing the missing data blocks and uploading them is not in there yet.
This patch also refactors the unit tests in test_immutable so that the handling of each kind of corruption is reported as passing or failing separately, can be separately TODO'ified, etc.  The unit tests are also improved in various ways to require more of the code under test or to stop requiring unreasonable things of it.  :-)

15 years agotrivial: fix inline comment in test code
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:53:42 +0000 (17:53 -0700)]
trivial: fix inline comment in test code

15 years agoimmutable: handle another form of share corruption with LayoutInvalid exception inste...
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:46:45 +0000 (17:46 -0700)]
immutable: handle another form of share corruption with LayoutInvalid exception instead of AssertionError

15 years agotrivial: remove unused import (pyflakes)
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:31:20 +0000 (17:31 -0700)]
trivial: remove unused import (pyflakes)

15 years agoimmutable: skip the test of large files, because that is too hard on the host if...
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:07:27 +0000 (17:07 -0700)]
immutable: skip the test of large files, because that is too hard on the host if it doesn't efficiently handle sparse files

15 years agoimmutable: raise a LayoutInvalid exception instead of an AssertionError if the share...
Zooko O'Whielacronx [Mon, 5 Jan 2009 21:01:14 +0000 (14:01 -0700)]
immutable: raise a LayoutInvalid exception instead of an AssertionError if the share is corrupted so that the sharehashtree is the wrong size

15 years agoimmutable: stop reading past the end of the sharefile in the process of optimizing...
Zooko O'Whielacronx [Mon, 5 Jan 2009 20:40:57 +0000 (13:40 -0700)]
immutable: stop reading past the end of the sharefile in the process of optimizing download -- Tahoe storage servers < 1.3.0 return an error if you read past the end of the share file

15 years agoimmutable: tidy up the notification of waiters for ReadBucketProxy
Zooko O'Whielacronx [Mon, 5 Jan 2009 20:35:22 +0000 (13:35 -0700)]
immutable: tidy up the notification of waiters for ReadBucketProxy

15 years agoimmutable: refactor downloader to be more reusable for checker/verifier/repairer...
Zooko O'Whielacronx [Mon, 5 Jan 2009 16:51:45 +0000 (09:51 -0700)]
immutable: refactor downloader to be more reusable for checker/verifier/repairer (and better)

The code for validating the share hash tree and the block hash tree has been rewritten to make sure it handles all cases, to share metadata about the file (such as the share hash tree, block hash trees, and UEB) among different share downloads, and not to require hashes to be stored on the server unnecessarily, such as the roots of the block hash trees (not needed since they are also the leaves of the share hash tree), and the root of the share hash tree (not needed since it is also included in the UEB).  It also passes the latest tests including handling corrupted shares well.

ValidatedReadBucketProxy takes a share_hash_tree argument to its constructor, which is a reference to a share hash tree shared by all ValidatedReadBucketProxies for that immutable file download.

ValidatedReadBucketProxy requires the block_size and share_size to be provided in its constructor, and it then uses those to compute the offsets and lengths of blocks when it needs them, instead of reading those values out of the share.  The user of ValidatedReadBucketProxy therefore has to have first used a ValidatedExtendedURIProxy to compute those two values from the validated contents of the URI.  This is pleasingly simplifies safety analysis: the client knows which span of bytes corresponds to a given block from the validated URI data, rather than from the unvalidated data stored on the storage server.  It also simplifies unit testing of verifier/repairer, because now it doesn't care about the contents of the "share size" and "block size" fields in the share.  It does not relieve the need for share data v2 layout, because we still need to store and retrieve the offsets of the fields which come after the share data, therefore we still need to use share data v2 with its 8-byte fields if we want to store share data larger than about 2^32.

Specify which subset of the block hashes and share hashes you need while downloading a particular share.  In the future this will hopefully be used to fetch only a subset, for network efficiency, but currently all of them are fetched, regardless of which subset you specify.

ReadBucketProxy hides the question of whether it has "started" or not (sent a request to the server to get metadata) from its user.

Download is optimized to do as few roundtrips and as few requests as possible, hopefully speeding up download a bit.

15 years agoutil: add gatherResults which is a deferred-list-like thing that doesn't wrap failure...
Zooko O'Whielacronx [Sun, 4 Jan 2009 17:52:02 +0000 (10:52 -0700)]
util: add gatherResults which is a deferred-list-like thing that doesn't wrap failures in a FirstError

15 years agoimmutable: fix think-o in previous patch which caused all reads to return "", and...
Zooko O'Whielacronx [Sat, 3 Jan 2009 21:02:45 +0000 (14:02 -0700)]
immutable: fix think-o in previous patch which caused all reads to return "", and also optimize by not opening the file when the answer is going to be ""

15 years agoimmutable: when storage server reads from immutable share, don't try to read past...
Zooko O'Whielacronx [Sat, 3 Jan 2009 20:22:22 +0000 (13:22 -0700)]
immutable: when storage server reads from immutable share, don't try to read past the end of the file (Python allocates space according to the amount of data requested, so if there is corruption and that number is huge it will do a huge memory allocation)

15 years agoimmutable: mark a failing download test as "todo", because I think it is revealing...
Zooko O'Whielacronx [Sat, 3 Jan 2009 20:00:03 +0000 (13:00 -0700)]
immutable: mark a failing download test as "todo", because I think it is revealing a limitation of the current downloader's handling of corrupted shares

15 years agodocs: update install.html to recommend Python v2 instead of Python v2.5.2
Zooko O'Whielacronx [Sat, 3 Jan 2009 19:31:00 +0000 (12:31 -0700)]
docs: update install.html to recommend Python v2 instead of Python v2.5.2

15 years agotrivial: remove unused import (pyflakes)
Zooko O'Whielacronx [Sat, 3 Jan 2009 19:22:15 +0000 (12:22 -0700)]
trivial: remove unused import (pyflakes)

15 years agomerge_install.patch
cgalvan [Fri, 2 Jan 2009 17:44:34 +0000 (10:44 -0700)]
merge_install.patch

15 years agosetup: new install doc -- doesn't require GNU make or a C++ compiler any more!
Zooko O'Whielacronx [Mon, 1 Dec 2008 19:09:33 +0000 (12:09 -0700)]
setup: new install doc -- doesn't require GNU make or a C++ compiler any more!

15 years agoimmutable: fix test for truncated reads of URI extension block size
Zooko O'Whielacronx [Sat, 3 Jan 2009 18:44:27 +0000 (11:44 -0700)]
immutable: fix test for truncated reads of URI extension block size

15 years agoimmutable: further loosen the performance-regression test to allow up to 45 reads
Zooko O'Whielacronx [Sat, 3 Jan 2009 18:41:09 +0000 (11:41 -0700)]
immutable: further loosen the performance-regression test to allow up to 45 reads
This does raise the question of if there is any point to this test, since I apparently don't know what the answer *should* be, and whenever one of the buildbots fails then I redefine success.

But, I'm about to commit a bunch of patches to implement checker, verifier, and repairer as well as to refactor downloader, and I would really like to know if these patches *increase* the number of reads required even higher than it currently is.

15 years agotrivial: another place where I accidentally committed a note-to-self about the lease...
Zooko O'Whielacronx [Sat, 3 Jan 2009 18:29:41 +0000 (11:29 -0700)]
trivial: another place where I accidentally committed a note-to-self about the lease fields in the server-side share file

15 years agoimmutable: fix detection of truncated shares to take into account the fieldsize ...
Zooko O'Whielacronx [Sat, 3 Jan 2009 01:57:45 +0000 (18:57 -0700)]
immutable: fix detection of truncated shares to take into account the fieldsize -- either 4 or 8

15 years agoimmutable: raise LayoutInvalid instead of struct.error when a share is truncated
Zooko O'Whielacronx [Sat, 3 Jan 2009 01:48:06 +0000 (18:48 -0700)]
immutable: raise LayoutInvalid instead of struct.error when a share is truncated
To fix this error from the Windows buildslave:

[ERROR]: allmydata.test.test_immutable.Test.test_download_from_only_3_remaining_shares

Traceback (most recent call last):
  File "C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\download.py", line 135, in _bad
    raise NotEnoughSharesError("ran out of peers, last error was %s" % (f,))
allmydata.interfaces.NotEnoughSharesError: ran out of peers, last error was [Failure instance: Traceback: <class 'struct.error'>: unpack requires a string argument of length 4
c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:667:_done
c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:53:complete
c:\Python25\lib\site-packages\twisted\internet\defer.py:239:callback
c:\Python25\lib\site-packages\twisted\internet\defer.py:304:_startRunCallbacks
--- <exception caught here> ---
c:\Python25\lib\site-packages\twisted\internet\defer.py:317:_runCallbacks
C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\layout.py:374:_got_length
C:\Python25\lib\struct.py:87:unpack
]
===============================================================================

15 years agoimmutable: whoops, it actually takes up to 39 reads sometimes to download a corrupted...
Zooko O'Whielacronx [Sat, 3 Jan 2009 00:43:02 +0000 (17:43 -0700)]
immutable: whoops, it actually takes up to 39 reads sometimes to download a corrupted file

15 years agoimmutable: add more detailed tests of download, including testing the count of how...
Zooko O'Whielacronx [Fri, 2 Jan 2009 23:54:59 +0000 (16:54 -0700)]
immutable: add more detailed tests of download, including testing the count of how many reads different sorts of downloads take

15 years agotrivial: a few improvements to in-line doc and code, and renaming of test/test_immuta...
Zooko O'Whielacronx [Fri, 2 Jan 2009 23:49:41 +0000 (16:49 -0700)]
trivial: a few improvements to in-line doc and code, and renaming of test/test_immutable_checker.py to test/test_immutable.py
That file currently tests checker and verifier and repairer, and will soon also test downloader.

15 years agoimmutable: fix name change from BadOrMissingShareHash to BadOrMissingHash
Zooko O'Whielacronx [Fri, 2 Jan 2009 20:27:09 +0000 (13:27 -0700)]
immutable: fix name change from BadOrMissingShareHash to BadOrMissingHash
One of the instances of the name accidentally didn't get changed, and pyflakes noticed.  The new downloader/checker/verifier/repairer unit tests would also have noticed, but those tests haven't been rolled into a patch and applied to this repo yet...

15 years agotrivial: remove unused import -- thanks, pyflakes
Zooko O'Whielacronx [Fri, 2 Jan 2009 20:21:28 +0000 (13:21 -0700)]
trivial: remove unused import -- thanks, pyflakes

15 years agoimmutable: download.py: Raise the appropriate type of exception to indicate the cause...
Zooko O'Whielacronx [Fri, 2 Jan 2009 19:58:58 +0000 (12:58 -0700)]
immutable: download.py: Raise the appropriate type of exception to indicate the cause of failure, e.g. BadOrMissingHash, ServerFailure, IntegrityCheckReject (which is a supertype of BadOrMissingHash).  This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work".  The tests of verifier/repairer test this code and rely on this code.

15 years agoimmutable: ReadBucketProxy defines classes of exception: LayoutInvalid and its two...
Zooko O'Whielacronx [Fri, 2 Jan 2009 19:15:54 +0000 (12:15 -0700)]
immutable: ReadBucketProxy defines classes of exception: LayoutInvalid and its two subtypes RidiculouslyLargeURIExtensionBlock and ShareVersionIncompatible.  This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work".  This code gets exercised by the verifier/repairer unit tests, which corrupt the shares on disk in order to trigger problems like these.

15 years agoimmutable: ValidatedExtendedURIProxy computes and stores block_size and share_size...
Zooko O'Whielacronx [Fri, 2 Jan 2009 18:43:17 +0000 (11:43 -0700)]
immutable: ValidatedExtendedURIProxy computes and stores block_size and share_size for the convenience of its users

15 years agoremove_sumo_install.patch
cgalvan [Fri, 2 Jan 2009 17:23:47 +0000 (10:23 -0700)]
remove_sumo_install.patch

15 years agodoc: remove notes to self that I accidentally included in a recent patch
Zooko O'Whielacronx [Fri, 2 Jan 2009 05:14:57 +0000 (22:14 -0700)]
doc: remove notes to self that I accidentally included in a recent patch

15 years agodocs: remove caveat about Nevow incompatibility with Python 2.6 since the latest...
Zooko O'Whielacronx [Fri, 2 Jan 2009 04:41:35 +0000 (21:41 -0700)]
docs: remove caveat about Nevow incompatibility with Python 2.6 since the latest version of Nevow has fixed it

15 years agoimmutable: make the test of large files more likely to work by requesting to allocate...
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:59:42 +0000 (15:59 -0700)]
immutable: make the test of large files more likely to work by requesting to allocate space for only one huge share, not three

15 years agotrivial: "M-x whitespace-cleanup", and also remove an unused variable
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:42:33 +0000 (15:42 -0700)]
trivial: "M-x whitespace-cleanup", and also remove an unused variable

15 years agoimmutable: storage servers accept any size shares now
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:42:26 +0000 (15:42 -0700)]
immutable: storage servers accept any size shares now
Nathan Wilcox observed that the storage server can rely on the size of the share file combined with the count of leases to unambiguously identify the location of the leases.  This means that it can hold any size share data, even though the field nominally used to hold the size of the share data is only 32 bits wide.

With this patch, the storage server still writes the "size of the share data" field (just in case the server gets downgraded to an earlier version which requires that field, or the share file gets moved to another server which is of an earlier vintage), but it doesn't use it.  Also, with this patch, the server no longer rejects requests to write shares which are >= 2^32 bytes in size, and it no longer rejects attempts to read such shares.

This fixes http://allmydata.org/trac/tahoe/ticket/346 (increase share-size field to 8 bytes, remove 12GiB filesize limit), although there remains open a question of how clients know that a given server can handle large shares (by using the new versioning scheme, probably).

Note that share size is also limited by another factor -- how big of a file we can store on the local filesystem on the server.  Currently allmydata.com typically uses ext3 and I think we typically have block size = 4 KiB, which means that the largest file is about 2 TiB.  Also, the hard drives themselves are only 1 TB, so the largest share is definitely slightly less than 1 TB, which means (when K == 3), the largest file is less than 3 TB.

This patch also refactors the creation of new sharefiles so that only a single fopen() is used.

This patch also helps with the unit-testing of repairer, since formerly it was unclear what repairer should expect to find if the "share data size" field was corrupted (some corruptions would have no effect, others would cause failure to download).  Now it is clear that repairer is not required to notice if this field is corrupted since it has no effect on download.  :-)

15 years agotrivial: "M-x whitespace-cleanup" on immutable/layout.py
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:07:02 +0000 (15:07 -0700)]
trivial: "M-x whitespace-cleanup" on immutable/layout.py

15 years agotrivial: remove unused import -- thanks, pyflakes
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:25:56 +0000 (15:25 -0700)]
trivial: remove unused import -- thanks, pyflakes

15 years agorrefutil: generically wrap any errback from callRemote() in a ServerFailure instance
Zooko O'Whielacronx [Wed, 31 Dec 2008 21:28:30 +0000 (14:28 -0700)]
rrefutil: generically wrap any errback from callRemote() in a ServerFailure instance
This facilitates client code to easily catch ServerFailures without also catching exceptions arising from client-side code.
See also:
http://foolscap.lothar.com/trac/ticket/105 # make it easy to distinguish server-side failures/exceptions from client-side

15 years agoimmutable: more detailed tests for checker/verifier/repairer
Zooko O'Whielacronx [Wed, 31 Dec 2008 21:18:38 +0000 (14:18 -0700)]
immutable: more detailed tests for checker/verifier/repairer
There are a lot of different ways that a share could be corrupted, or that attempting to download it might fail.  These tests attempt to exercise many of those ways and require the checker/verifier/repairer to handle each kind of failure well.

15 years agodocs: add note about non-ascii chars in cli to NEWS
Zooko O'Whielacronx [Tue, 30 Dec 2008 09:17:28 +0000 (02:17 -0700)]
docs: add note about non-ascii chars in cli to NEWS

15 years agocli: make startstop_node wait 20 seconds instead of 5 for a process to go away after...
Zooko O'Whielacronx [Tue, 30 Dec 2008 08:20:22 +0000 (01:20 -0700)]
cli: make startstop_node wait 20 seconds instead of 5 for a process to go away after we signalled it to go away
Because the unit tests on the VirtualZooko buildslave failed when it took 16 seconds for a process to go away.
Perhaps getting notification after only 5 seconds instead of 20 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests.  Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning.

15 years agodocs: editing changes and updated news in known_issues.txt
Zooko O'Whielacronx [Tue, 30 Dec 2008 08:01:16 +0000 (01:01 -0700)]
docs: editing changes and updated news in known_issues.txt

15 years agodocs: split historical/historical_known_issues.txt out of known_issues.txt
Zooko O'Whielacronx [Tue, 30 Dec 2008 07:52:26 +0000 (00:52 -0700)]
docs: split historical/historical_known_issues.txt out of known_issues.txt
All issues which are relevant to users of v1.1, v1.2, or v1.3 go in known_issues.txt.  All issues which are relevant to users of v1.0 go in historical/historical_known_issues.txt.

15 years agodoc: sundry amendments to docs and in-line code comments
Zooko O'Whielacronx [Sun, 28 Dec 2008 23:59:54 +0000 (16:59 -0700)]
doc: sundry amendments to docs and in-line code comments

15 years agodoc: add mention of "tahoe create-alias" in the security-warning section of CLI.txt
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:46 +0000 (15:16 -0700)]
doc: add mention of "tahoe create-alias" in the security-warning section of CLI.txt

15 years agodoc: trivial: remove trailing whitespace
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:34 +0000 (15:16 -0700)]
doc: trivial: remove trailing whitespace

15 years agodoc: warn that unicode might not work, in CLI.txt
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:18 +0000 (15:16 -0700)]
doc: warn that unicode might not work, in CLI.txt

15 years agodoc: use the term "filesystem" rather than "virtual drive" in CLI.txt
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:14 +0000 (15:16 -0700)]
doc: use the term "filesystem" rather than "virtual drive" in CLI.txt

15 years agocli: mark unicode filenames as unsupported -- see #534 for details
Zooko O'Whielacronx [Wed, 24 Dec 2008 20:28:02 +0000 (13:28 -0700)]
cli: mark unicode filenames as unsupported -- see #534 for details

15 years agocli: undo the effects of [http://allmydata.org/trac/tahoe/changeset/20081222235453...
Zooko O'Whielacronx [Wed, 24 Dec 2008 16:53:17 +0000 (09:53 -0700)]
cli: undo the effects of [http://allmydata.org/trac/tahoe/changeset/20081222235453-92b7f-f841e18afb94e1fd95e6dafb799a3d876dd85c69]
We're just going to mark unicode in the cli as unsupported for tahoe-lafs-1.3.0.  Unicode filenames on the command-line do actually work for some platforms and probably only if the platform encoding is utf-8, but I'm not sure, and in any case for it to be marked as "supported" it would have to work on all platforms, be thoroughly tested, and also we would have to understand why it worked.  :-)

15 years agotest: extend timeout on the hotline file that prevents the client from stopping itself
Zooko O'Whielacronx [Mon, 22 Dec 2008 04:06:29 +0000 (21:06 -0700)]
test: extend timeout on the hotline file that prevents the client from stopping itself
The 20-second timeout was apparently tripped on my Powerbook G4 "draco".

15 years agocli: decode all cli arguments, assuming that they are utf-8 encoded
Zooko O'Whielacronx [Tue, 23 Dec 2008 00:54:53 +0000 (17:54 -0700)]
cli: decode all cli arguments, assuming that they are utf-8 encoded
Also encode all args to urllib as utf-8 because urllib doesn't handle unicode objects.
I'm not sure if it is appropriate to *assume* utf-8 encoding of cli args.  Perhaps the Right thing to do is to detect the platform encoding.  Any ideas?
This patch is mostly due to François Deppierraz.

15 years agoutil/base32: the identity trans table needn't have any contents -- we are using strin...
Zooko O'Whielacronx [Tue, 23 Dec 2008 00:48:08 +0000 (17:48 -0700)]
util/base32: the identity trans table needn't have any contents -- we are using string.translate solely to delete known chars

15 years agoutil/base32: allow unicode inputs to a2b() or could_be_base32_encoded(), and encode...
Zooko O'Whielacronx [Tue, 23 Dec 2008 00:47:13 +0000 (17:47 -0700)]
util/base32: allow unicode inputs to a2b() or could_be_base32_encoded(), and encode them with utf-8 before processing them

15 years agoutil/base32: loosen the precondition forbidding unicode and requiring str -- now...
Zooko O'Whielacronx [Mon, 22 Dec 2008 23:22:37 +0000 (16:22 -0700)]
util/base32: loosen the precondition forbidding unicode and requiring str -- now it requires either unicode or str
Hopefully this will make it so that tests pass with François Deppierraz's patch to fix the tahoe cli's handling of unicode argument.

15 years agoimmutable: don't catch all exception when downloading, catch only DeadReferenceError...
Zooko O'Whielacronx [Mon, 22 Dec 2008 00:41:35 +0000 (17:41 -0700)]
immutable: don't catch all exception when downloading, catch only DeadReferenceError and IntegrityCheckReject

15 years agoimmutable: invent download.BadOrMissingHashError which is raised if either hashtree...
Zooko O'Whielacronx [Mon, 22 Dec 2008 00:41:30 +0000 (17:41 -0700)]
immutable: invent download.BadOrMissingHashError which is raised if either hashtree.BadHashError, hashtree.NotEnoughHashesError, and which is a subclass of IntegrityCheckReject

15 years agodirnode: don't check MAC on entries in dirnodes
Zooko O'Whielacronx [Mon, 22 Dec 2008 00:35:18 +0000 (17:35 -0700)]
dirnode: don't check MAC on entries in dirnodes
In an ancient version of directories, we needed a MAC on each entry.  In modern times, the entire dirnode comes with a digital signature, so the MAC on each entry is redundant.
With this patch, we no longer check those MACs when reading directories, but we still produce them so that older readers will accept directories that we write.

15 years agoimmutable, checker, and tests: improve docstrings, assertions, tests
Zooko O'Whielacronx [Sun, 21 Dec 2008 22:07:52 +0000 (15:07 -0700)]
immutable, checker, and tests: improve docstrings, assertions, tests
No functional changes, but remove unused code, improve or fix docstrings, etc.

15 years agocli: if response code from wapi server is not 200 then stop instead of proceeding
Zooko O'Whielacronx [Sat, 20 Dec 2008 14:49:18 +0000 (07:49 -0700)]
cli: if response code from wapi server is not 200 then stop instead of proceeding
Also, include the data that failed to json parse in an exception raised by the json parser.

15 years agoimmutable: when downloading an immutable file, use primary shares if they are available
Zooko O'Whielacronx [Sat, 20 Dec 2008 14:14:56 +0000 (07:14 -0700)]
immutable: when downloading an immutable file, use primary shares if they are available
Primary shares require no erasure decoding so the more primary shares you have, the less CPU is used.

15 years agotrivial: remove unused import (thanks, pyflakes)
Zooko O'Whielacronx [Fri, 19 Dec 2008 20:46:29 +0000 (13:46 -0700)]
trivial: remove unused import (thanks, pyflakes)

15 years agotry to tidy up uri-as-string vs. uri-as-object
Zooko O'Whielacronx [Fri, 19 Dec 2008 15:39:24 +0000 (08:39 -0700)]
try to tidy up uri-as-string vs. uri-as-object
I get confused about whether a given argument or return value is a uri-as-string or uri-as-object.  This patch adds a lot of assertions that it is one or the other, and also changes CheckerResults to take objects not strings.
In the future, I hope that we generally use Python objects except when importing into or exporting from the Python interpreter e.g. over the wire, the UI, or a stored file.

15 years agoimmutable: remove the last bits of code (only test code or unused code) which did...
Zooko O'Whielacronx [Fri, 19 Dec 2008 15:18:07 +0000 (08:18 -0700)]
immutable: remove the last bits of code (only test code or unused code) which did something with plaintext hashes or plaintext hash trees

15 years agoimmutable: use new logging mixins to simplify logging
Zooko O'Whielacronx [Wed, 17 Dec 2008 01:04:50 +0000 (18:04 -0700)]
immutable: use new logging mixins to simplify logging

15 years agoimmutable: refactor ReadBucketProxy a little
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:53:25 +0000 (17:53 -0700)]
immutable: refactor ReadBucketProxy a little

15 years agodebug: pass empty optional arguments to ReadBucketProxy
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:51:45 +0000 (17:51 -0700)]
debug: pass empty optional arguments to ReadBucketProxy
because those arguments are about to become non-optional (for other code than test/debug code)

15 years agouri: generalize regexp that recognizes tahoe URLs to work for any host and port
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:49:30 +0000 (17:49 -0700)]
uri: generalize regexp that recognizes tahoe URLs to work for any host and port

15 years agoutil: logging: refactor some common logging behavior into mixins
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:38:07 +0000 (17:38 -0700)]
util: logging: refactor some common logging behavior into mixins

15 years agopyutil: assertutil: copy in simplified assertutil from pyutil
Zooko O'Whielacronx [Wed, 17 Dec 2008 00:37:45 +0000 (17:37 -0700)]
pyutil: assertutil: copy in simplified assertutil from pyutil

15 years agopyutil: assertutil: simplify handling of exception during formatting of precondition...
Zooko O'Whielacronx [Wed, 10 Dec 2008 14:10:57 +0000 (07:10 -0700)]
pyutil: assertutil: simplify handling of exception during formatting of precondition message, and reduce dependency to just the Python Standard Library's logging module

15 years agoclient: add get_servers()
Zooko O'Whielacronx [Tue, 9 Dec 2008 00:04:00 +0000 (17:04 -0700)]
client: add get_servers()

15 years agomutable publish: if we are surprised by shares that match what we would have written...
Brian Warner [Wed, 10 Dec 2008 05:44:49 +0000 (22:44 -0700)]
mutable publish: if we are surprised by shares that match what we would have written anyways, don't be surprised. This should fix one of the two #546 problems, in which we re-use a server and forget that we already sent them a share.