Zooko O'Whielacronx [Wed, 21 Jan 2009 03:50:42 +0000 (20:50 -0700)]
docs: trivial naming change
Zooko O'Whielacronx [Wed, 21 Jan 2009 03:47:35 +0000 (20:47 -0700)]
rollback the feature of making "ambient upload authority" configurable
This reverses some, but not all, of the changes that were committed in the following set of patches.
rolling back:
Sun Jan 18 09:54:30 MST 2009 toby.murray
* add 'web.ambient_upload_authority' as a paramater to tahoe.cfg
M ./src/allmydata/client.py -1 +3
M ./src/allmydata/test/common.py -7 +9
A ./src/allmydata/test/test_ambient_upload_authority.py
M ./src/allmydata/web/root.py +12
M ./src/allmydata/webish.py -1 +4
Sun Jan 18 09:56:08 MST 2009 zooko@zooko.com
* trivial: whitespace
I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines.
M ./src/allmydata/test/test_ambient_upload_authority.py -9 +8
M ./src/allmydata/web/root.py -2 +1
M ./src/allmydata/webish.py -2 +1
Mon Jan 19 14:16:19 MST 2009 zooko@zooko.com
* trivial: remove unused import noticed by pyflakes
M ./src/allmydata/test/test_ambient_upload_authority.py -1
Mon Jan 19 21:38:35 MST 2009 toby.murray
* doc: describe web.ambient_upload_authority
M ./docs/configuration.txt +14
M ./docs/frontends/webapi.txt +11
Mon Jan 19 21:38:57 MST 2009 zooko@zooko.com
* doc: add Toby Murray to the CREDITS
M ./CREDITS +4
Zooko O'Whielacronx [Wed, 21 Jan 2009 00:40:12 +0000 (17:40 -0700)]
setup: add new bundled setuptools_trial-0.4
Zooko O'Whielacronx [Wed, 21 Jan 2009 00:39:18 +0000 (17:39 -0700)]
setup: remove old bundled setuptools_trial-0.2
Zooko O'Whielacronx [Tue, 20 Jan 2009 19:42:29 +0000 (12:42 -0700)]
setup: require darcsver always, and not just when we see the string "darcsver" in sys.argv, because the new aliases hack means that the string might not appear in sys.argv
Zooko O'Whielacronx [Tue, 20 Jan 2009 19:38:09 +0000 (12:38 -0700)]
setup: fix test_system to require tahoe under its package==distribution name "allmydata-tahoe" instead of its module name "allmydata"
Zooko O'Whielacronx [Tue, 20 Jan 2009 19:37:23 +0000 (12:37 -0700)]
setup: use setup.cfg aliases to map "setup.py test" to "setup.py trial" and "setup.py build" to "setup.py darcsver --count-all-patches build_tahoe"
Thanks to dpeterson for the suggestion.
Zooko O'Whielacronx [Tue, 20 Jan 2009 05:38:57 +0000 (22:38 -0700)]
doc: add Toby Murray to the CREDITS
toby.murray [Tue, 20 Jan 2009 05:38:35 +0000 (22:38 -0700)]
doc: describe web.ambient_upload_authority
Zooko O'Whielacronx [Mon, 19 Jan 2009 22:16:19 +0000 (15:16 -0700)]
trivial: remove unused import noticed by pyflakes
Zooko O'Whielacronx [Mon, 19 Jan 2009 22:04:47 +0000 (15:04 -0700)]
setup: simplify install.html a tad
Zooko O'Whielacronx [Mon, 19 Jan 2009 22:04:35 +0000 (15:04 -0700)]
setup: refactor versions-and-paths and use pkg_resources to find them
Using pkg_resources is probably better if it works -- zope.interface doesn't have a __version__ attribute that we can query, but pkg_resources knows zope.interface's version number, for one thing.
This code falls back to the old way -- looking at the __version__ attributes and __file__ attributes -- if the pkg_resources way doesn't answer.
Note that this patch also changes the capitalization of "Nevow", "Twisted", and "pyOpenSSL", and the spelling of "allmydata-tahoe". These changes are not frivolous: they are reflecting the fact that we are naming Python packages (technically called Python "distributions") instead of Python modules (technically and confusingly called Python "packages") here. The package ("distribution") is named "allmydata-tahoe". The module ("package") is named "allmydata".
Zooko O'Whielacronx [Mon, 19 Jan 2009 21:53:52 +0000 (14:53 -0700)]
setup: undo (for the second time) the use of the --multi-version feature
When this feature is turned on, then setuptools doesn't create easy-install.pth, setuptools.pth, or site.py in the target site-packages dir. I don't know why not and we should probably open a ticket on the setuptools tracker and/or hack setuptools to create those files anyway. But for now (for the Tahoe-1.3.0 release), we're going to leave --multi-version mode off and require users to manually uninstall any packages which are too old and thus conflict with our newer dependencies.
Zooko O'Whielacronx [Sun, 18 Jan 2009 17:58:15 +0000 (10:58 -0700)]
trivial: whitespace
Ran "M-x whitespace-cleanup" on files that Toby's recent patch touched, even though they didn't have trailing whitespace.
Zooko O'Whielacronx [Sun, 18 Jan 2009 17:56:08 +0000 (10:56 -0700)]
trivial: whitespace
I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines.
Zooko O'Whielacronx [Sun, 18 Jan 2009 17:54:58 +0000 (10:54 -0700)]
trivial: whitespace
toby.murray [Sun, 18 Jan 2009 17:54:30 +0000 (10:54 -0700)]
add 'web.ambient_upload_authority' as a paramater to tahoe.cfg
Zooko O'Whielacronx [Sat, 17 Jan 2009 17:49:43 +0000 (10:49 -0700)]
doc: add Larry Hosken to CREDITS
cgalvan [Sat, 17 Jan 2009 01:00:47 +0000 (18:00 -0700)]
run build_tahoe command with trial commmand
cgalvan [Sat, 17 Jan 2009 00:03:26 +0000 (17:03 -0700)]
adding multi-version support
Zooko O'Whielacronx [Fri, 16 Jan 2009 21:05:54 +0000 (14:05 -0700)]
setup: addition of setuptools_trial egg to the base dir is boring from the point of view of revision control
cgalvan [Fri, 16 Jan 2009 20:57:32 +0000 (13:57 -0700)]
prevent --site-dirs from being passed to the 'install' command
cgalvan [Fri, 16 Jan 2009 19:47:51 +0000 (12:47 -0700)]
add option to show version and path to the tahoe executable
Zooko O'Whielacronx [Thu, 15 Jan 2009 03:37:51 +0000 (20:37 -0700)]
setup: put back configuration of the PYTHONPATH which is necessary to build the Windows packages
Brian Warner [Thu, 15 Jan 2009 03:00:15 +0000 (20:00 -0700)]
node.py: use NODEDIR/tmp for the 'tempfile' module's temporary directory, so webapi upload tempfiles are put there instead of /tmp . You can set it to something else by setting [node]tempdir in tahoe.cfg
Brian Warner [Thu, 15 Jan 2009 02:14:59 +0000 (19:14 -0700)]
web/operations: undo the disable-ophandle-expiration change that inadvertently got included in zooko's recent 'rename wapi.txt to webapi.txt' patch, causing test failures
Zooko O'Whielacronx [Thu, 15 Jan 2009 01:10:10 +0000 (18:10 -0700)]
cli: tests: skip symlink test if there is no os.symlink
Brian Warner [Thu, 15 Jan 2009 01:08:03 +0000 (18:08 -0700)]
webapi.txt: explain POST /uri/?t=upload, thanks to Toby Murray for the catch
Brian Warner [Thu, 15 Jan 2009 00:36:20 +0000 (17:36 -0700)]
mutable: move recent operation history management code (MutableWatcher) into history.py, have History provide stats
Brian Warner [Wed, 14 Jan 2009 23:41:51 +0000 (16:41 -0700)]
download: tiny cleanup of history code
Brian Warner [Wed, 14 Jan 2009 23:41:06 +0000 (16:41 -0700)]
upload: move upload history into History object
Brian Warner [Wed, 14 Jan 2009 23:14:24 +0000 (16:14 -0700)]
immutable/download.py move recent-downloads history out of Downloader and into a separate class. upload/etc will follow soon.
Zooko O'Whielacronx [Wed, 14 Jan 2009 20:53:48 +0000 (13:53 -0700)]
docs: rename wapi.txt to webapi.txt
Because Brian argues that the file contains a description of the wui as well as of the wapi, and because the name "webapi.txt" might be more obvious to the untrained eye.
Zooko O'Whielacronx [Wed, 14 Jan 2009 19:50:23 +0000 (12:50 -0700)]
setup: configure setup.cfg to point setuptools at the index.html page instead of straight at the tahoe directory listing
This is necessary, because if setuptools looks at the directory listing, then it follows the link named "More Info" in attempt to download the file, and this fails.
Zooko O'Whielacronx [Wed, 14 Jan 2009 17:40:22 +0000 (10:40 -0700)]
setup: fix previous patch to set reactor to poll reactor on linux or cygwin
Zooko O'Whielacronx [Wed, 14 Jan 2009 16:15:46 +0000 (09:15 -0700)]
setup: use poll reactor for trial if on linux2 or cygwin
Zooko O'Whielacronx [Wed, 14 Jan 2009 03:51:43 +0000 (20:51 -0700)]
docs: rename frontends/webapi.txt to frontends/wapi.txt
rename CLI.txt to frontends/CLI.txt
change a few mentions of "webapi" to "wapi"
fixes #582
Brian Warner [Tue, 13 Jan 2009 03:14:42 +0000 (20:14 -0700)]
upload: use WriteBucketProxy_v2 when uploading a large file (with shares larger than 4GiB). This finally closes #346. I think we can now handle immutable files up to 48EiB.
Brian Warner [Tue, 13 Jan 2009 01:56:19 +0000 (18:56 -0700)]
deep-check-and-repair: improve results and their HTML representation
Brian Warner [Mon, 12 Jan 2009 22:45:09 +0000 (15:45 -0700)]
test_repairer.py: hush pyflakes: remove duplicate/shadowed function name, by using the earlier definition (which is identical)
Brian Warner [Mon, 12 Jan 2009 22:41:20 +0000 (15:41 -0700)]
hush pyflakes by removing unused imports
Zooko O'Whielacronx [Mon, 12 Jan 2009 18:00:22 +0000 (11:00 -0700)]
immutable repairer
This implements an immutable repairer by marrying a CiphertextDownloader to a CHKUploader. It extends the IDownloadTarget interface so that the downloader can provide some metadata that the uploader requires.
The processing is incremental -- it uploads the first segments before it finishes downloading the whole file. This is necessary so that you can repair large files without running out of RAM or using a temporary file on the repairer.
It requires only a verifycap, not a readcap. That is: it doesn't need or use the decryption key, only the integrity check codes.
There are several tests marked TODO and several instances of XXX in the source code. I intend to open tickets to document further improvements to functionality and testing, but the current version is probably good enough for Tahoe-1.3.0.
Zooko O'Whielacronx [Mon, 12 Jan 2009 17:55:39 +0000 (10:55 -0700)]
util: dictutil: add DictOfSets.union(key, values) and DictOfSets.update(otherdictofsets)
Zooko O'Whielacronx [Sun, 11 Jan 2009 16:13:19 +0000 (09:13 -0700)]
setup: update doc in setup.cfg
Zooko O'Whielacronx [Sun, 11 Jan 2009 16:11:26 +0000 (09:11 -0700)]
setup: Point setuptools at a directory on the allmydata.org test grid to find dependencies.
Don't include an unrouteable IP address in find_links (fixes #574).
Zooko O'Whielacronx [Sat, 10 Jan 2009 22:07:39 +0000 (15:07 -0700)]
immutable: separate tests of immutable upload/download from tests of immutable checking/repair
Zooko O'Whielacronx [Sat, 10 Jan 2009 21:56:01 +0000 (14:56 -0700)]
trivial: minor changes to in-line comments -- mark plaintext-hash-tree as obsolete
Zooko O'Whielacronx [Sat, 10 Jan 2009 21:02:09 +0000 (14:02 -0700)]
immutable: make the web display of upload results more human-friendly, like they were before my recent change to the meaning of the "sharemap"
Zooko O'Whielacronx [Sat, 10 Jan 2009 19:54:08 +0000 (12:54 -0700)]
immutable: fix edit-o in interfaces.py documentation introduced in recent patch
Zooko O'Whielacronx [Sat, 10 Jan 2009 18:46:23 +0000 (11:46 -0700)]
immutable: redefine the "sharemap" member of the upload results to be a map from shnum to set of serverids
It used to be a map from shnum to a string saying "placed this share on XYZ server". The new definition is more in keeping with the "sharemap" object that results from immutable file checking and repair, and it is more useful to the repairer, which is a consumer of immutable upload results.
Zooko O'Whielacronx [Sat, 10 Jan 2009 01:00:52 +0000 (18:00 -0700)]
naming: finish renaming "CheckerResults" to "CheckResults"
Brian Warner [Sat, 10 Jan 2009 02:52:22 +0000 (19:52 -0700)]
storage.py : replace
4294967295 with 2**32-1: python does constant folding, I measured this statement as taking 50ns, versus the 400ns for the call to min(), or the 9us required for the 'assert not os.path.exists' syscall
Brian Warner [Sat, 10 Jan 2009 02:37:36 +0000 (19:37 -0700)]
storage.py: announce a maximum-immutable-share-size based upon a 'df' of the disk. Fixes #569, and this should be the last requirement for #346 (remove 12GiB filesize limit)
cgalvan [Fri, 9 Jan 2009 22:06:40 +0000 (15:06 -0700)]
set bin/tahoe executable permissions and leave build_tahoe in sys.argv
Zooko O'Whielacronx [Fri, 9 Jan 2009 20:09:49 +0000 (13:09 -0700)]
setup: merge relaxation of the version of setuptools that we require at runtime with an indentation change
Zooko O'Whielacronx [Sat, 6 Dec 2008 00:22:07 +0000 (17:22 -0700)]
setup: remove custom Trial class inside our setup.py and use the setuptools_trial plugin
Zooko O'Whielacronx [Sat, 6 Dec 2008 00:19:11 +0000 (17:19 -0700)]
setup: we require pywin32 if building on Windows (plus some formatting and comment fixes)
cgalvan [Fri, 9 Jan 2009 19:42:22 +0000 (12:42 -0700)]
fix bin/tahoe executable for Windows
cgalvan [Fri, 9 Jan 2009 19:03:00 +0000 (12:03 -0700)]
use subprocess.call instead of os.execve in bin/tahoe
Zooko O'Whielacronx [Sat, 6 Dec 2008 00:30:54 +0000 (17:30 -0700)]
setup: attempt to remove the custom setuptools-ish logic in setup.py -- the result works on my Windows box but doesn't yield a working ./bin/tahoe on Windows, and hasn't been tested yet on other platforms
Zooko O'Whielacronx [Mon, 1 Dec 2008 18:48:04 +0000 (11:48 -0700)]
setup: integrate the bundled setuptools_trial plugin with Chris Galvan's patch to use that plugin
cgalvan [Fri, 21 Nov 2008 21:57:59 +0000 (14:57 -0700)]
use_setuptools_trial.patch
Zooko O'Whielacronx [Mon, 1 Dec 2008 18:44:38 +0000 (11:44 -0700)]
setup: bundle setuptools_trial in misc/dependencies/
Brian Warner [Fri, 9 Jan 2009 03:59:41 +0000 (20:59 -0700)]
test_helper: hush pyflakes by avoiding use of 'uri' as a variable, since it shadows an import of the same name
Brian Warner [Fri, 9 Jan 2009 03:01:45 +0000 (20:01 -0700)]
immutable/checker: include a summary (with 'Healthy' or 'Not Healthy' and a count of shares) in the checker results
Brian Warner [Fri, 9 Jan 2009 02:59:32 +0000 (19:59 -0700)]
webapi/deep-manifest t=JSON: don't return the (large) manifest/SI/verifycap lists unless the operation has completed, to avoid the considerable CPU+memory cost of creating the JSON (for 330k dirnodes, it could take two minutes to generate 275MB of JSON). They must be paid eventually, but not on every poll
Brian Warner [Fri, 9 Jan 2009 02:41:16 +0000 (19:41 -0700)]
dirnode deep-traversal: remove use of Limiter, stick with strict depth-first-traversal, to reduce memory usage during very large (300k+ dirnode) traversals
Zooko O'Whielacronx [Thu, 8 Jan 2009 21:42:15 +0000 (14:42 -0700)]
immutable: add a monitor API to CiphertextDownloader with which to tell it to stop its work
Zooko O'Whielacronx [Thu, 8 Jan 2009 19:13:07 +0000 (12:13 -0700)]
naming: Rename a few things which I touched or changed in the recent patch to download-without-decrypting.
Rename "downloadable" to "target".
Rename "u" to "v" in FileDownloader.__init__().
Rename "_uri" to "_verifycap" in FileDownloader.
Rename "_downloadable" to "_target" in FileDownloader.
Rename "FileDownloader" to "CiphertextDownloader".
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:53:49 +0000 (11:53 -0700)]
immutable: refactor download to do only download-and-decode, not decryption
FileDownloader takes a verify cap and produces ciphertext, instead of taking a read cap and producing plaintext.
FileDownloader does all integrity checking including the mandatory ciphertext hash tree and the optional ciphertext flat hash, rather than expecting its target to do some of that checking.
Rename immutable.download.Output to immutable.download.DecryptingOutput. An instance of DecryptingOutput can be passed to FileDownloader to use as the latter's target. Text pushed to the DecryptingOutput is decrypted and then pushed to *its* target.
DecryptingOutput satisfies the IConsumer interface, and if its target also satisfies IConsumer, then it forwards and pause/unpause signals to its producer (which is the FileDownloader).
This patch also changes some logging code to use the new logging mixin class.
Check integrity of a segment and decrypt the segment one block-sized buffer at a time instead of copying the buffers together into one segment-sized buffer (reduces peak memory usage, I think, and is probably a tad faster/less CPU, depending on your encoding parameters).
Refactor FileDownloader so that processing of segments and of tail-segment share as much code is possible.
FileDownloader and FileNode take caps as instances of URI (Python objects), not as strings.
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:20:48 +0000 (11:20 -0700)]
trivial: tiny changes to test code
Zooko O'Whielacronx [Thu, 8 Jan 2009 18:25:30 +0000 (11:25 -0700)]
immutable: Make more parts of download use logging mixins and know what their "parent msg id" is.
Zooko O'Whielacronx [Thu, 8 Jan 2009 17:49:01 +0000 (10:49 -0700)]
trivial: M-x whitespace-cleanup on src/immutable/download.py
Zooko O'Whielacronx [Thu, 8 Jan 2009 17:41:39 +0000 (10:41 -0700)]
immutable: ValidatedExtendedURIProxy computes and stores the tail data size as a convenience to its caller.
The "tail data size" is how many of the bytes of the tail segment are data (as opposed to padding).
Zooko O'Whielacronx [Wed, 7 Jan 2009 19:24:51 +0000 (12:24 -0700)]
immutable: define a new interface IImmutableFileURI and declare that CHKFileURI and LiteralFileURI provide it
Zooko O'Whielacronx [Wed, 7 Jan 2009 18:54:11 +0000 (11:54 -0700)]
util: log: allow empty msgs (because downloader is using the "format" alternative with no "msg" argument)
Larry Hosken [Thu, 8 Jan 2009 06:51:14 +0000 (23:51 -0700)]
'tahoe cp -r', upon encountering a dangling symlink, would assert out.
This was somewhat sad; the assertion didn't say what path caused the
error, what went wrong. So... silently skip over things that are
neither dirs nor files.
Zooko O'Whielacronx [Thu, 8 Jan 2009 06:40:12 +0000 (23:40 -0700)]
immutable: fix error in validation of ciphertext hash tree and add test for that code
pyflakes pointed out to me that I had committed some code that is untested, since it uses an undefined name. This patch exercises that code -- the validation of the ciphertext hash tree -- by corrupting some of the share files in a very specific way, and also fixes the bug.
Zooko O'Whielacronx [Thu, 8 Jan 2009 05:25:51 +0000 (22:25 -0700)]
immutable: do not catch arbitrary exceptions/failures from the attempt to get a crypttext hash tree -- catch only ServerFailure, IntegrityCheckReject, LayoutInvalid, ShareVersionIncompatible, and DeadReferenceError
Once again I inserted a bug into the code, and once again it was hidden by something catching arbitrary exception/failure and assuming that it means the server failed to provide valid data.
Zooko O'Whielacronx [Thu, 8 Jan 2009 03:26:38 +0000 (20:26 -0700)]
download: make sure you really get all the crypttext hashes
We were not making sure that we really got all the crypttext hashes during download. If a server were to return less than the complete set of crypttext hashes, then our subsequent attempt to verify the correctness of the ciphertext would fail. (And it wouldn't be obvious without very careful debugging why it had failed.)
This patch makes it so that you keep trying to get ciphertext hashes until you have a full set or you run out of servers to ask.
Zooko O'Whielacronx [Wed, 7 Jan 2009 18:00:05 +0000 (11:00 -0700)]
util: deferredutil: undo my recent patch to use our own implementation of gatherResults
It seems to cause lots of failures on some builders.
Zooko O'Whielacronx [Wed, 7 Jan 2009 17:32:07 +0000 (10:32 -0700)]
util: deferredutil: implement our own gatherResults instead of using Twisted's
Because we want to maintain backwards compatibility to Twisted 2.4.0.
Zooko O'Whielacronx [Wed, 7 Jan 2009 17:25:28 +0000 (10:25 -0700)]
trivial: M-x whitespace-cleanup
Zooko O'Whielacronx [Wed, 7 Jan 2009 15:13:42 +0000 (08:13 -0700)]
util: deferredutil: add basic test for deferredutil.gatherResults
Also I checked and Twisted 2.4.0 supports .subFailure and the other parts of the API that we require.
Zooko O'Whielacronx [Wed, 7 Jan 2009 05:08:29 +0000 (22:08 -0700)]
trivial: fix redefinition of name "log" in imports (pyflakes)
Zooko O'Whielacronx [Wed, 7 Jan 2009 04:48:22 +0000 (21:48 -0700)]
immutable: refactor uploader to do just encoding-and-uploading, not encryption
This makes Uploader take an EncryptedUploadable object instead of an Uploadable object. I also changed it to return a verify cap instead of a tuple of the bits of data that one finds in a verify cap.
This will facilitate hooking together an Uploader and a Downloader to make a Repairer.
Also move offloaded.py into src/allmydata/immutable/.
Zooko O'Whielacronx [Wed, 7 Jan 2009 04:41:04 +0000 (21:41 -0700)]
trivial: whitespace and docstring tidyups
Brian Warner [Tue, 6 Jan 2009 20:57:21 +0000 (13:57 -0700)]
storage.py: explain what this large and hard-to-recognize
4294967295 number is
Zooko O'Whielacronx [Tue, 6 Jan 2009 20:37:03 +0000 (13:37 -0700)]
rename "checker results" to "check results", because it is more parallel to "check-and-repair results"
Zooko O'Whielacronx [Tue, 6 Jan 2009 20:04:49 +0000 (13:04 -0700)]
immutable: tests: verifier doesn't always catch corrupted share hashes
Maybe it already got one of the corrupted hashes from a different server and it doesn't double-check that the hash from every server is correct. Or another problem. But in any case I'm marking this as TODO because an even better (more picky) verifier is less urgent than repairer.
Zooko O'Whielacronx [Tue, 6 Jan 2009 19:24:04 +0000 (12:24 -0700)]
immutable: fix the writing of share data size into share file in case the share file is used by a < v1.3.0 storage server
Brian noticed that the constant was wrong, and in fixing that I noticed that we should be saturating instead of modding.
This code would never matter unless a server downgraded or a share migrated from Tahoe >= v1.3.0 to Tahoe < v1.3.0. Even in that case, this bug would never matter unless the share size were exactly 4,294,967,296 bytes long.
Brian, for good reason, wanted this to be spelled "2**32" instead of "
4294967296", but I couldn't stand to see a couple of more Python bytecodes interpreted in the middle of a core, frequent operation on the server like immutable share creation.
Zooko O'Whielacronx [Tue, 6 Jan 2009 18:20:58 +0000 (11:20 -0700)]
trivial: whitespace cleanup
Zooko O'Whielacronx [Tue, 6 Jan 2009 17:41:22 +0000 (10:41 -0700)]
util: base32: require str-not-unicode inputs -- effectively rolls back [3306] and [3307]
Zooko O'Whielacronx [Tue, 6 Jan 2009 15:00:54 +0000 (08:00 -0700)]
trivial: fix a bunch of pyflakes complaints
Zooko O'Whielacronx [Tue, 6 Jan 2009 14:51:06 +0000 (07:51 -0700)]
cli: make startstop_node wait 40 seconds instead of 20 for a process to go away after we signalled it to go away, before emitting a warning
Because the unit tests on the VirtualZooko? buildslave failed when it took 31 seconds for a process to go away.
Perhaps getting warning message after only 5 seconds instead of 40 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests. Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning.
Zooko O'Whielacronx [Tue, 6 Jan 2009 01:28:18 +0000 (18:28 -0700)]
immutable: new checker and verifier
New checker and verifier use the new download class. They are robust against various sorts of failures or corruption. They return detailed results explaining what they learned about your immutable files. Some grotesque sorts of corruption are not properly handled yet, and those ones are marked as TODO or commented-out in the unit tests.
There is also a repairer module in this patch with the beginnings of a repairer in it. That repairer is mostly just the interface to the outside world -- the core operation of actually reconstructing the missing data blocks and uploading them is not in there yet.
This patch also refactors the unit tests in test_immutable so that the handling of each kind of corruption is reported as passing or failing separately, can be separately TODO'ified, etc. The unit tests are also improved in various ways to require more of the code under test or to stop requiring unreasonable things of it. :-)
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:53:42 +0000 (17:53 -0700)]
trivial: fix inline comment in test code
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:46:45 +0000 (17:46 -0700)]
immutable: handle another form of share corruption with LayoutInvalid exception instead of AssertionError
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:31:20 +0000 (17:31 -0700)]
trivial: remove unused import (pyflakes)
Zooko O'Whielacronx [Tue, 6 Jan 2009 00:07:27 +0000 (17:07 -0700)]
immutable: skip the test of large files, because that is too hard on the host if it doesn't efficiently handle sparse files