Zooko O'Whielacronx [Sat, 10 Jan 2009 18:46:23 +0000 (11:46 -0700)]
immutable: redefine the "sharemap" member of the upload results to be a map from shnum to set of serverids
It used to be a map from shnum to a string saying "placed this share on XYZ server". The new definition is more in keeping with the "sharemap" object that results from immutable file checking and repair, and it is more useful to the repairer, which is a consumer of immutable upload results.
Zooko O'Whielacronx [Sat, 10 Jan 2009 01:00:52 +0000 (18:00 -0700)]
naming: finish renaming "CheckerResults" to "CheckResults"
Brian Warner [Sat, 10 Jan 2009 02:52:22 +0000 (19:52 -0700)]
storage.py : replace
4294967295 with 2**32-1: python does constant folding, I measured this statement as taking 50ns, versus the 400ns for the call to min(), or the 9us required for the 'assert not os.path.exists' syscall
Brian Warner [Sat, 10 Jan 2009 02:37:36 +0000 (19:37 -0700)]
storage.py: announce a maximum-immutable-share-size based upon a 'df' of the disk. Fixes #569, and this should be the last requirement for #346 (remove 12GiB filesize limit)
cgalvan [Fri, 9 Jan 2009 22:06:40 +0000 (15:06 -0700)]
set bin/tahoe executable permissions and leave build_tahoe in sys.argv
Zooko O'Whielacronx [Fri, 9 Jan 2009 20:09:49 +0000 (13:09 -0700)]
setup: merge relaxation of the version of setuptools that we require at runtime with an indentation change
Zooko O'Whielacronx [Sat, 6 Dec 2008 00:22:07 +0000 (17:22 -0700)]
setup: remove custom Trial class inside our setup.py and use the setuptools_trial plugin
Zooko O'Whielacronx [Sat, 6 Dec 2008 00:19:11 +0000 (17:19 -0700)]
setup: we require pywin32 if building on Windows (plus some formatting and comment fixes)
cgalvan [Fri, 9 Jan 2009 19:42:22 +0000 (12:42 -0700)]
fix bin/tahoe executable for Windows
cgalvan [Fri, 9 Jan 2009 19:03:00 +0000 (12:03 -0700)]
use subprocess.call instead of os.execve in bin/tahoe
Zooko O'Whielacronx [Sat, 6 Dec 2008 00:30:54 +0000 (17:30 -0700)]
setup: attempt to remove the custom setuptools-ish logic in setup.py -- the result works on my Windows box but doesn't yield a working ./bin/tahoe on Windows, and hasn't been tested yet on other platforms
Zooko O'Whielacronx [Mon, 1 Dec 2008 18:48:04 +0000 (11:48 -0700)]
setup: integrate the bundled setuptools_trial plugin with Chris Galvan's patch to use that plugin
cgalvan [Fri, 21 Nov 2008 21:57:59 +0000 (14:57 -0700)]
use_setuptools_trial.patch
Zooko O'Whielacronx [Mon, 1 Dec 2008 18:44:38 +0000 (11:44 -0700)]
setup: bundle setuptools_trial in misc/dependencies/
Brian Warner [Fri, 9 Jan 2009 03:59:41 +0000 (20:59 -0700)]
test_helper: hush pyflakes by avoiding use of 'uri' as a variable, since it shadows an import of the same name
Brian Warner [Fri, 9 Jan 2009 03:01:45 +0000 (20:01 -0700)]
immutable/checker: include a summary (with 'Healthy' or 'Not Healthy' and a count of shares) in the checker results
Brian Warner [Fri, 9 Jan 2009 02:59:32 +0000 (19:59 -0700)]
webapi/deep-manifest t=JSON: don't return the (large) manifest/SI/verifycap lists unless the operation has completed, to avoid the considerable CPU+memory cost of creating the JSON (for 330k dirnodes, it could take two minutes to generate 275MB of JSON). They must be paid eventually, but not on every poll
Brian Warner [Fri, 9 Jan 2009 02:41:16 +0000 (19:41 -0700)]
dirnode deep-traversal: remove use of Limiter, stick with strict depth-first-traversal, to reduce memory usage during very large (300k+ dirnode) traversals
Zooko O'Whielacronx [Thu, 8 Jan 2009 21:42:15 +0000 (14:42 -0700)]
immutable: add a monitor API to CiphertextDownloader with which to tell it to stop its work
Zooko O'Whielacronx [Thu, 8 Jan 2009 19:13:07 +0000 (12:13 -0700)]
naming: Rename a few things which I touched or changed in the recent patch to download-without-decrypting.
Rename "downloadable" to "target".
Rename "u" to "v" in FileDownloader.__init__().
Rename "_uri" to "_verifycap" in FileDownloader.
Rename "_downloadable" to "_target" in FileDownloader.
Rename "FileDownloader" to "CiphertextDownloader".
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:53:49 +0000 (11:53 -0700)]
immutable: refactor download to do only download-and-decode, not decryption
FileDownloader takes a verify cap and produces ciphertext, instead of taking a read cap and producing plaintext.
FileDownloader does all integrity checking including the mandatory ciphertext hash tree and the optional ciphertext flat hash, rather than expecting its target to do some of that checking.
Rename immutable.download.Output to immutable.download.DecryptingOutput. An instance of DecryptingOutput can be passed to FileDownloader to use as the latter's target. Text pushed to the DecryptingOutput is decrypted and then pushed to *its* target.
DecryptingOutput satisfies the IConsumer interface, and if its target also satisfies IConsumer, then it forwards and pause/unpause signals to its producer (which is the FileDownloader).
This patch also changes some logging code to use the new logging mixin class.
Check integrity of a segment and decrypt the segment one block-sized buffer at a time instead of copying the buffers together into one segment-sized buffer (reduces peak memory usage, I think, and is probably a tad faster/less CPU, depending on your encoding parameters).
Refactor FileDownloader so that processing of segments and of tail-segment share as much code is possible.
FileDownloader and FileNode take caps as instances of URI (Python objects), not as strings.
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:20:48 +0000 (11:20 -0700)]
trivial: tiny changes to test code
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:25:30 +0000 (11:25 -0700)]
immutable: Make more parts of download use logging mixins and know what their "parent msg id" is.
Zooko O'Whielacronx [Thu, 8 Jan 2009 17:49:01 +0000 (10:49 -0700)]
trivial: M-x whitespace-cleanup on src/immutable/download.py
Zooko O'Whielacronx [Thu, 8 Jan 2009 17:41:39 +0000 (10:41 -0700)]
immutable: ValidatedExtendedURIProxy computes and stores the tail data size as a convenience to its caller.
The "tail data size" is how many of the bytes of the tail segment are data (as opposed to padding).
Zooko O'Whielacronx [Wed, 7 Jan 2009 19:24:51 +0000 (12:24 -0700)]
immutable: define a new interface IImmutableFileURI and declare that CHKFileURI and LiteralFileURI provide it
Zooko O'Whielacronx [Wed, 7 Jan 2009 18:54:11 +0000 (11:54 -0700)]
util: log: allow empty msgs (because downloader is using the "format" alternative with no "msg" argument)
Larry Hosken [Thu, 8 Jan 2009 06:51:14 +0000 (23:51 -0700)]
'tahoe cp -r', upon encountering a dangling symlink, would assert out.
This was somewhat sad; the assertion didn't say what path caused the
error, what went wrong. So... silently skip over things that are
neither dirs nor files.
Zooko O'Whielacronx [Thu, 8 Jan 2009 06:40:12 +0000 (23:40 -0700)]
immutable: fix error in validation of ciphertext hash tree and add test for that code
pyflakes pointed out to me that I had committed some code that is untested, since it uses an undefined name. This patch exercises that code -- the validation of the ciphertext hash tree -- by corrupting some of the share files in a very specific way, and also fixes the bug.
Zooko O'Whielacronx [Thu, 8 Jan 2009 05:25:51 +0000 (22:25 -0700)]
immutable: do not catch arbitrary exceptions/failures from the attempt to get a crypttext hash tree -- catch only ServerFailure, IntegrityCheckReject, LayoutInvalid, ShareVersionIncompatible, and DeadReferenceError
Once again I inserted a bug into the code, and once again it was hidden by something catching arbitrary exception/failure and assuming that it means the server failed to provide valid data.
Zooko O'Whielacronx [Thu, 8 Jan 2009 03:26:38 +0000 (20:26 -0700)]
download: make sure you really get all the crypttext hashes
We were not making sure that we really got all the crypttext hashes during download. If a server were to return less than the complete set of crypttext hashes, then our subsequent attempt to verify the correctness of the ciphertext would fail. (And it wouldn't be obvious without very careful debugging why it had failed.)
This patch makes it so that you keep trying to get ciphertext hashes until you have a full set or you run out of servers to ask.
Zooko O'Whielacronx [Wed, 7 Jan 2009 18:00:05 +0000 (11:00 -0700)]
util: deferredutil: undo my recent patch to use our own implementation of gatherResults
It seems to cause lots of failures on some builders.
Zooko O'Whielacronx [Wed, 7 Jan 2009 17:32:07 +0000 (10:32 -0700)]
util: deferredutil: implement our own gatherResults instead of using Twisted's
Because we want to maintain backwards compatibility to Twisted 2.4.0.
Zooko O'Whielacronx [Wed, 7 Jan 2009 17:25:28 +0000 (10:25 -0700)]
trivial: M-x whitespace-cleanup
Zooko O'Whielacronx [Wed, 7 Jan 2009 15:13:42 +0000 (08:13 -0700)]
util: deferredutil: add basic test for deferredutil.gatherResults
Also I checked and Twisted 2.4.0 supports .subFailure and the other parts of the API that we require.
Zooko O'Whielacronx [Wed, 7 Jan 2009 05:08:29 +0000 (22:08 -0700)]
trivial: fix redefinition of name "log" in imports (pyflakes)
Zooko O'Whielacronx [Wed, 7 Jan 2009 04:48:22 +0000 (21:48 -0700)]
immutable: refactor uploader to do just encoding-and-uploading, not encryption
This makes Uploader take an EncryptedUploadable object instead of an Uploadable object. I also changed it to return a verify cap instead of a tuple of the bits of data that one finds in a verify cap.
This will facilitate hooking together an Uploader and a Downloader to make a Repairer.
Also move offloaded.py into src/allmydata/immutable/.
Zooko O'Whielacronx [Wed, 7 Jan 2009 04:41:04 +0000 (21:41 -0700)]
trivial: whitespace and docstring tidyups
Brian Warner [Tue, 6 Jan 2009 20:57:21 +0000 (13:57 -0700)]
storage.py: explain what this large and hard-to-recognize
4294967295 number is
Zooko O'Whielacronx [Tue, 6 Jan 2009 20:37:03 +0000 (13:37 -0700)]
rename "checker results" to "check results", because it is more parallel to "check-and-repair results"
Zooko O'Whielacronx [Tue, 6 Jan 2009 20:04:49 +0000 (13:04 -0700)]
immutable: tests: verifier doesn't always catch corrupted share hashes
Maybe it already got one of the corrupted hashes from a different server and it doesn't double-check that the hash from every server is correct. Or another problem. But in any case I'm marking this as TODO because an even better (more picky) verifier is less urgent than repairer.
Zooko O'Whielacronx [Tue, 6 Jan 2009 19:24:04 +0000 (12:24 -0700)]
immutable: fix the writing of share data size into share file in case the share file is used by a < v1.3.0 storage server
Brian noticed that the constant was wrong, and in fixing that I noticed that we should be saturating instead of modding.
This code would never matter unless a server downgraded or a share migrated from Tahoe >= v1.3.0 to Tahoe < v1.3.0. Even in that case, this bug would never matter unless the share size were exactly 4,294,967,296 bytes long.
Brian, for good reason, wanted this to be spelled "2**32" instead of "
4294967296", but I couldn't stand to see a couple of more Python bytecodes interpreted in the middle of a core, frequent operation on the server like immutable share creation.
Zooko O'Whielacronx [Tue, 6 Jan 2009 18:20:58 +0000 (11:20 -0700)]
trivial: whitespace cleanup
Zooko O'Whielacronx [Tue, 6 Jan 2009 17:41:22 +0000 (10:41 -0700)]
util: base32: require str-not-unicode inputs -- effectively rolls back [3306] and [3307]
Zooko O'Whielacronx [Tue, 6 Jan 2009 15:00:54 +0000 (08:00 -0700)]
trivial: fix a bunch of pyflakes complaints
Zooko O'Whielacronx [Tue, 6 Jan 2009 14:51:06 +0000 (07:51 -0700)]
cli: make startstop_node wait 40 seconds instead of 20 for a process to go away after we signalled it to go away, before emitting a warning
Because the unit tests on the VirtualZooko? buildslave failed when it took 31 seconds for a process to go away.
Perhaps getting warning message after only 5 seconds instead of 40 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests. Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning.
Zooko O'Whielacronx [Tue, 6 Jan 2009 01:28:18 +0000 (18:28 -0700)]
immutable: new checker and verifier
New checker and verifier use the new download class. They are robust against various sorts of failures or corruption. They return detailed results explaining what they learned about your immutable files. Some grotesque sorts of corruption are not properly handled yet, and those ones are marked as TODO or commented-out in the unit tests.
There is also a repairer module in this patch with the beginnings of a repairer in it. That repairer is mostly just the interface to the outside world -- the core operation of actually reconstructing the missing data blocks and uploading them is not in there yet.
This patch also refactors the unit tests in test_immutable so that the handling of each kind of corruption is reported as passing or failing separately, can be separately TODO'ified, etc. The unit tests are also improved in various ways to require more of the code under test or to stop requiring unreasonable things of it. :-)
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:53:42 +0000 (17:53 -0700)]
trivial: fix inline comment in test code
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:46:45 +0000 (17:46 -0700)]
immutable: handle another form of share corruption with LayoutInvalid exception instead of AssertionError
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:31:20 +0000 (17:31 -0700)]
trivial: remove unused import (pyflakes)
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:07:27 +0000 (17:07 -0700)]
immutable: skip the test of large files, because that is too hard on the host if it doesn't efficiently handle sparse files
Zooko O'Whielacronx [Mon, 5 Jan 2009 21:01:14 +0000 (14:01 -0700)]
immutable: raise a LayoutInvalid exception instead of an AssertionError if the share is corrupted so that the sharehashtree is the wrong size
Zooko O'Whielacronx [Mon, 5 Jan 2009 20:40:57 +0000 (13:40 -0700)]
immutable: stop reading past the end of the sharefile in the process of optimizing download -- Tahoe storage servers < 1.3.0 return an error if you read past the end of the share file
Zooko O'Whielacronx [Mon, 5 Jan 2009 20:35:22 +0000 (13:35 -0700)]
immutable: tidy up the notification of waiters for ReadBucketProxy
Zooko O'Whielacronx [Mon, 5 Jan 2009 16:51:45 +0000 (09:51 -0700)]
immutable: refactor downloader to be more reusable for checker/verifier/repairer (and better)
The code for validating the share hash tree and the block hash tree has been rewritten to make sure it handles all cases, to share metadata about the file (such as the share hash tree, block hash trees, and UEB) among different share downloads, and not to require hashes to be stored on the server unnecessarily, such as the roots of the block hash trees (not needed since they are also the leaves of the share hash tree), and the root of the share hash tree (not needed since it is also included in the UEB). It also passes the latest tests including handling corrupted shares well.
ValidatedReadBucketProxy takes a share_hash_tree argument to its constructor, which is a reference to a share hash tree shared by all ValidatedReadBucketProxies for that immutable file download.
ValidatedReadBucketProxy requires the block_size and share_size to be provided in its constructor, and it then uses those to compute the offsets and lengths of blocks when it needs them, instead of reading those values out of the share. The user of ValidatedReadBucketProxy therefore has to have first used a ValidatedExtendedURIProxy to compute those two values from the validated contents of the URI. This is pleasingly simplifies safety analysis: the client knows which span of bytes corresponds to a given block from the validated URI data, rather than from the unvalidated data stored on the storage server. It also simplifies unit testing of verifier/repairer, because now it doesn't care about the contents of the "share size" and "block size" fields in the share. It does not relieve the need for share data v2 layout, because we still need to store and retrieve the offsets of the fields which come after the share data, therefore we still need to use share data v2 with its 8-byte fields if we want to store share data larger than about 2^32.
Specify which subset of the block hashes and share hashes you need while downloading a particular share. In the future this will hopefully be used to fetch only a subset, for network efficiency, but currently all of them are fetched, regardless of which subset you specify.
ReadBucketProxy hides the question of whether it has "started" or not (sent a request to the server to get metadata) from its user.
Download is optimized to do as few roundtrips and as few requests as possible, hopefully speeding up download a bit.
Zooko O'Whielacronx [Sun, 4 Jan 2009 17:52:02 +0000 (10:52 -0700)]
util: add gatherResults which is a deferred-list-like thing that doesn't wrap failures in a FirstError
Zooko O'Whielacronx [Sat, 3 Jan 2009 21:02:45 +0000 (14:02 -0700)]
immutable: fix think-o in previous patch which caused all reads to return "", and also optimize by not opening the file when the answer is going to be ""
Zooko O'Whielacronx [Sat, 3 Jan 2009 20:22:22 +0000 (13:22 -0700)]
immutable: when storage server reads from immutable share, don't try to read past the end of the file (Python allocates space according to the amount of data requested, so if there is corruption and that number is huge it will do a huge memory allocation)
Zooko O'Whielacronx [Sat, 3 Jan 2009 20:00:03 +0000 (13:00 -0700)]
immutable: mark a failing download test as "todo", because I think it is revealing a limitation of the current downloader's handling of corrupted shares
Zooko O'Whielacronx [Sat, 3 Jan 2009 19:31:00 +0000 (12:31 -0700)]
docs: update install.html to recommend Python v2 instead of Python v2.5.2
Zooko O'Whielacronx [Sat, 3 Jan 2009 19:22:15 +0000 (12:22 -0700)]
trivial: remove unused import (pyflakes)
cgalvan [Fri, 2 Jan 2009 17:44:34 +0000 (10:44 -0700)]
merge_install.patch
Zooko O'Whielacronx [Mon, 1 Dec 2008 19:09:33 +0000 (12:09 -0700)]
setup: new install doc -- doesn't require GNU make or a C++ compiler any more!
Zooko O'Whielacronx [Sat, 3 Jan 2009 18:44:27 +0000 (11:44 -0700)]
immutable: fix test for truncated reads of URI extension block size
Zooko O'Whielacronx [Sat, 3 Jan 2009 18:41:09 +0000 (11:41 -0700)]
immutable: further loosen the performance-regression test to allow up to 45 reads
This does raise the question of if there is any point to this test, since I apparently don't know what the answer *should* be, and whenever one of the buildbots fails then I redefine success.
But, I'm about to commit a bunch of patches to implement checker, verifier, and repairer as well as to refactor downloader, and I would really like to know if these patches *increase* the number of reads required even higher than it currently is.
Zooko O'Whielacronx [Sat, 3 Jan 2009 18:29:41 +0000 (11:29 -0700)]
trivial: another place where I accidentally committed a note-to-self about the lease fields in the server-side share file
Zooko O'Whielacronx [Sat, 3 Jan 2009 01:57:45 +0000 (18:57 -0700)]
immutable: fix detection of truncated shares to take into account the fieldsize -- either 4 or 8
Zooko O'Whielacronx [Sat, 3 Jan 2009 01:48:06 +0000 (18:48 -0700)]
immutable: raise LayoutInvalid instead of struct.error when a share is truncated
To fix this error from the Windows buildslave:
[ERROR]: allmydata.test.test_immutable.Test.test_download_from_only_3_remaining_shares
Traceback (most recent call last):
File "C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\download.py", line 135, in _bad
raise NotEnoughSharesError("ran out of peers, last error was %s" % (f,))
allmydata.interfaces.NotEnoughSharesError: ran out of peers, last error was [Failure instance: Traceback: <class 'struct.error'>: unpack requires a string argument of length 4
c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:667:_done
c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:53:complete
c:\Python25\lib\site-packages\twisted\internet\defer.py:239:callback
c:\Python25\lib\site-packages\twisted\internet\defer.py:304:_startRunCallbacks
--- <exception caught here> ---
c:\Python25\lib\site-packages\twisted\internet\defer.py:317:_runCallbacks
C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\layout.py:374:_got_length
C:\Python25\lib\struct.py:87:unpack
]
===============================================================================
Zooko O'Whielacronx [Sat, 3 Jan 2009 00:43:02 +0000 (17:43 -0700)]
immutable: whoops, it actually takes up to 39 reads sometimes to download a corrupted file
Zooko O'Whielacronx [Fri, 2 Jan 2009 23:54:59 +0000 (16:54 -0700)]
immutable: add more detailed tests of download, including testing the count of how many reads different sorts of downloads take
Zooko O'Whielacronx [Fri, 2 Jan 2009 23:49:41 +0000 (16:49 -0700)]
trivial: a few improvements to in-line doc and code, and renaming of test/test_immutable_checker.py to test/test_immutable.py
That file currently tests checker and verifier and repairer, and will soon also test downloader.
Zooko O'Whielacronx [Fri, 2 Jan 2009 20:27:09 +0000 (13:27 -0700)]
immutable: fix name change from BadOrMissingShareHash to BadOrMissingHash
One of the instances of the name accidentally didn't get changed, and pyflakes noticed. The new downloader/checker/verifier/repairer unit tests would also have noticed, but those tests haven't been rolled into a patch and applied to this repo yet...
Zooko O'Whielacronx [Fri, 2 Jan 2009 20:21:28 +0000 (13:21 -0700)]
trivial: remove unused import -- thanks, pyflakes
Zooko O'Whielacronx [Fri, 2 Jan 2009 19:58:58 +0000 (12:58 -0700)]
immutable: download.py: Raise the appropriate type of exception to indicate the cause of failure, e.g. BadOrMissingHash, ServerFailure, IntegrityCheckReject (which is a supertype of BadOrMissingHash). This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work". The tests of verifier/repairer test this code and rely on this code.
Zooko O'Whielacronx [Fri, 2 Jan 2009 19:15:54 +0000 (12:15 -0700)]
immutable: ReadBucketProxy defines classes of exception: LayoutInvalid and its two subtypes RidiculouslyLargeURIExtensionBlock and ShareVersionIncompatible. This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work". This code gets exercised by the verifier/repairer unit tests, which corrupt the shares on disk in order to trigger problems like these.
Zooko O'Whielacronx [Fri, 2 Jan 2009 18:43:17 +0000 (11:43 -0700)]
immutable: ValidatedExtendedURIProxy computes and stores block_size and share_size for the convenience of its users
cgalvan [Fri, 2 Jan 2009 17:23:47 +0000 (10:23 -0700)]
remove_sumo_install.patch
Zooko O'Whielacronx [Fri, 2 Jan 2009 05:14:57 +0000 (22:14 -0700)]
doc: remove notes to self that I accidentally included in a recent patch
Zooko O'Whielacronx [Fri, 2 Jan 2009 04:41:35 +0000 (21:41 -0700)]
docs: remove caveat about Nevow incompatibility with Python 2.6 since the latest version of Nevow has fixed it
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:59:42 +0000 (15:59 -0700)]
immutable: make the test of large files more likely to work by requesting to allocate space for only one huge share, not three
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:42:33 +0000 (15:42 -0700)]
trivial: "M-x whitespace-cleanup", and also remove an unused variable
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:42:26 +0000 (15:42 -0700)]
immutable: storage servers accept any size shares now
Nathan Wilcox observed that the storage server can rely on the size of the share file combined with the count of leases to unambiguously identify the location of the leases. This means that it can hold any size share data, even though the field nominally used to hold the size of the share data is only 32 bits wide.
With this patch, the storage server still writes the "size of the share data" field (just in case the server gets downgraded to an earlier version which requires that field, or the share file gets moved to another server which is of an earlier vintage), but it doesn't use it. Also, with this patch, the server no longer rejects requests to write shares which are >= 2^32 bytes in size, and it no longer rejects attempts to read such shares.
This fixes http://allmydata.org/trac/tahoe/ticket/346 (increase share-size field to 8 bytes, remove 12GiB filesize limit), although there remains open a question of how clients know that a given server can handle large shares (by using the new versioning scheme, probably).
Note that share size is also limited by another factor -- how big of a file we can store on the local filesystem on the server. Currently allmydata.com typically uses ext3 and I think we typically have block size = 4 KiB, which means that the largest file is about 2 TiB. Also, the hard drives themselves are only 1 TB, so the largest share is definitely slightly less than 1 TB, which means (when K == 3), the largest file is less than 3 TB.
This patch also refactors the creation of new sharefiles so that only a single fopen() is used.
This patch also helps with the unit-testing of repairer, since formerly it was unclear what repairer should expect to find if the "share data size" field was corrupted (some corruptions would have no effect, others would cause failure to download). Now it is clear that repairer is not required to notice if this field is corrupted since it has no effect on download. :-)
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:07:02 +0000 (15:07 -0700)]
trivial: "M-x whitespace-cleanup" on immutable/layout.py
Zooko O'Whielacronx [Wed, 31 Dec 2008 22:25:56 +0000 (15:25 -0700)]
trivial: remove unused import -- thanks, pyflakes
Zooko O'Whielacronx [Wed, 31 Dec 2008 21:28:30 +0000 (14:28 -0700)]
rrefutil: generically wrap any errback from callRemote() in a ServerFailure instance
This facilitates client code to easily catch ServerFailures without also catching exceptions arising from client-side code.
See also:
http://foolscap.lothar.com/trac/ticket/105 # make it easy to distinguish server-side failures/exceptions from client-side
Zooko O'Whielacronx [Wed, 31 Dec 2008 21:18:38 +0000 (14:18 -0700)]
immutable: more detailed tests for checker/verifier/repairer
There are a lot of different ways that a share could be corrupted, or that attempting to download it might fail. These tests attempt to exercise many of those ways and require the checker/verifier/repairer to handle each kind of failure well.
Zooko O'Whielacronx [Tue, 30 Dec 2008 09:17:28 +0000 (02:17 -0700)]
docs: add note about non-ascii chars in cli to NEWS
Zooko O'Whielacronx [Tue, 30 Dec 2008 08:20:22 +0000 (01:20 -0700)]
cli: make startstop_node wait 20 seconds instead of 5 for a process to go away after we signalled it to go away
Because the unit tests on the VirtualZooko buildslave failed when it took 16 seconds for a process to go away.
Perhaps getting notification after only 5 seconds instead of 20 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests. Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning.
Zooko O'Whielacronx [Tue, 30 Dec 2008 08:01:16 +0000 (01:01 -0700)]
docs: editing changes and updated news in known_issues.txt
Zooko O'Whielacronx [Tue, 30 Dec 2008 07:52:26 +0000 (00:52 -0700)]
docs: split historical/historical_known_issues.txt out of known_issues.txt
All issues which are relevant to users of v1.1, v1.2, or v1.3 go in known_issues.txt. All issues which are relevant to users of v1.0 go in historical/historical_known_issues.txt.
Zooko O'Whielacronx [Sun, 28 Dec 2008 23:59:54 +0000 (16:59 -0700)]
doc: sundry amendments to docs and in-line code comments
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:46 +0000 (15:16 -0700)]
doc: add mention of "tahoe create-alias" in the security-warning section of CLI.txt
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:34 +0000 (15:16 -0700)]
doc: trivial: remove trailing whitespace
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:18 +0000 (15:16 -0700)]
doc: warn that unicode might not work, in CLI.txt
Zooko O'Whielacronx [Wed, 24 Dec 2008 22:16:14 +0000 (15:16 -0700)]
doc: use the term "filesystem" rather than "virtual drive" in CLI.txt
Zooko O'Whielacronx [Wed, 24 Dec 2008 20:28:02 +0000 (13:28 -0700)]
cli: mark unicode filenames as unsupported -- see #534 for details
Zooko O'Whielacronx [Wed, 24 Dec 2008 16:53:17 +0000 (09:53 -0700)]
cli: undo the effects of [http://allmydata.org/trac/tahoe/changeset/
20081222235453-92b7f-
f841e18afb94e1fd95e6dafb799a3d876dd85c69]
We're just going to mark unicode in the cli as unsupported for tahoe-lafs-1.3.0. Unicode filenames on the command-line do actually work for some platforms and probably only if the platform encoding is utf-8, but I'm not sure, and in any case for it to be marked as "supported" it would have to work on all platforms, be thoroughly tested, and also we would have to understand why it worked. :-)
Zooko O'Whielacronx [Mon, 22 Dec 2008 04:06:29 +0000 (21:06 -0700)]
test: extend timeout on the hotline file that prevents the client from stopping itself
The 20-second timeout was apparently tripped on my Powerbook G4 "draco".
Zooko O'Whielacronx [Tue, 23 Dec 2008 00:54:53 +0000 (17:54 -0700)]
cli: decode all cli arguments, assuming that they are utf-8 encoded
Also encode all args to urllib as utf-8 because urllib doesn't handle unicode objects.
I'm not sure if it is appropriate to *assume* utf-8 encoding of cli args. Perhaps the Right thing to do is to detect the platform encoding. Any ideas?
This patch is mostly due to François Deppierraz.
Zooko O'Whielacronx [Tue, 23 Dec 2008 00:48:08 +0000 (17:48 -0700)]
util/base32: the identity trans table needn't have any contents -- we are using string.translate solely to delete known chars