Brian Warner [Wed, 15 Jul 2009 07:29:29 +0000 (00:29 -0700)]
node.py: record pid in twisted log at startup. Closes #476.
Brian Warner [Wed, 15 Jul 2009 06:45:10 +0000 (23:45 -0700)]
Allow tests to pass with -OO by turning some AssertionErrors (the ones that
we actually exercise during tests) into more specific exceptions, so they
don't get optimized away. The best rule to follow is probably this: if an
exception is worth testing, then it's part of the API, and AssertionError
should never be part of the API. Closes #749.
Zooko O'Whielacronx [Wed, 15 Jul 2009 02:52:46 +0000 (19:52 -0700)]
setup: tell build to search a directory (hosted on allmydata.org) which contains eggs
Zooko O'Whielacronx [Wed, 15 Jul 2009 03:34:29 +0000 (20:34 -0700)]
wui: fix test for welcome page
Zooko O'Whielacronx [Wed, 15 Jul 2009 02:58:14 +0000 (19:58 -0700)]
wui: adjust headers/titles and "Attach something to this directory" text in accordance with #691
Also name it as "TahoeLAFS" in this text.
Zooko O'Whielacronx [Wed, 15 Jul 2009 01:43:38 +0000 (18:43 -0700)]
mac: emit the stdout and stderr from a failed call to Allmydata (this is very useful diagnostic info)
Zooko O'Whielacronx [Tue, 14 Jul 2009 12:26:55 +0000 (05:26 -0700)]
mac: improve shutdown and clean-up after the test of the macintosh packaging
Zooko O'Whielacronx [Sun, 12 Jul 2009 23:09:40 +0000 (16:09 -0700)]
setup: add "test_mac_diskimage" command which attempts to mount and use a .dmg to make sure it has a good Tahoe distribution in it
Zooko O'Whielacronx [Mon, 13 Jul 2009 00:13:20 +0000 (17:13 -0700)]
dirnode: finish renaming "iv" to "salt" in the code and the hash tag
Brian Warner [Sun, 12 Jul 2009 23:50:25 +0000 (00:50 +0100)]
dirnode.py/_encrypt_rwcap: rename IV to "salt", which is more accurate
Brian Warner [Sun, 12 Jul 2009 23:47:50 +0000 (00:47 +0100)]
dirnode.py: security bug: also use child writecap to derive child enc key,
not just the dirnode writecap. The previous code (which only hashed the
dirnode writecap) would use the same key for all children, which is very bad.
This is the correct implementation of #750.
Brian Warner [Sat, 11 Jul 2009 23:47:03 +0000 (00:47 +0100)]
NEWS: update with all recent changes
Brian Warner [Sat, 11 Jul 2009 13:25:00 +0000 (14:25 +0100)]
proposed/mutable-DSA.txt: add some new design criteria
kevan [Sat, 4 Jul 2009 03:43:01 +0000 (20:43 -0700)]
Use CachingDict instead of dict in dirnode.py
Zooko O'Whielacronx [Fri, 10 Jul 2009 03:20:28 +0000 (20:20 -0700)]
directories: fix semantic conflict between my "keep track of position" optimization patch and Kevan's "cache serialized entries" optimization patch
kevan [Sun, 5 Jul 2009 22:31:42 +0000 (15:31 -0700)]
Modify bench_dirnode.py to use CachingDict.
kevan [Sat, 4 Jul 2009 03:43:28 +0000 (20:43 -0700)]
Add tests for CachingDict, _pack_contents, _unpack_contents
Zooko O'Whielacronx [Thu, 9 Jul 2009 13:05:13 +0000 (06:05 -0700)]
trivial: removed unused import noticed by pyflakes
Zooko O'Whielacronx [Sun, 5 Jul 2009 02:48:15 +0000 (19:48 -0700)]
directories: make the IV for the writecaps in directory entries be computed from the secure hash of the writecap itself
This makes encoding of directory entries deterministic, and it is also a tad faster on Macbook Pro than getting a random IV with os.urandom(16).
Zooko O'Whielacronx [Thu, 9 Jul 2009 04:47:29 +0000 (21:47 -0700)]
tests: raise the timeout on pollmixin from 100s to 1000s -- it looks like it may have triggered too eagerly on Zandr's 266 MHz armel
Zooko O'Whielacronx [Thu, 9 Jul 2009 00:45:33 +0000 (17:45 -0700)]
setup: add sys.maxunicode to the show-tool-versions output in order to investigate http://bugs.python.org/setuptools/issue78
Zooko O'Whielacronx [Wed, 8 Jul 2009 15:26:33 +0000 (08:26 -0700)]
immutable: base32-encode the keys to generate cache filenames that will work on all platforms
Zooko O'Whielacronx [Wed, 8 Jul 2009 00:40:40 +0000 (17:40 -0700)]
directories: make initialization of the download cache lazy
If you open up a directory containing thousands of files, it currently computes the cache filename and checks for the cache file on disk immediately for each immutble file in that directory. With this patch, it delays those steps until you try to do something with an immutable file that could use the cache.
kevan [Sun, 5 Jul 2009 21:23:45 +0000 (14:23 -0700)]
Add CachingDict dict subclass to dirnode.py
Zooko O'Whielacronx [Tue, 7 Jul 2009 14:34:04 +0000 (07:34 -0700)]
directories: rename internal data member download_cache to download_cache_dirman (benchmarks set this internal member and use the new name, so changing this makes the bench_dirnode.py work again)
Zooko O'Whielacronx [Sun, 5 Jul 2009 02:51:09 +0000 (19:51 -0700)]
directories: keep track of your position as you decode netstring after netstring from an input buffer instead of copying the trailing part
This makes decoding linear in the number of netstrings instead of O(N^2).
Zooko O'Whielacronx [Tue, 7 Jul 2009 03:41:19 +0000 (20:41 -0700)]
directories: in bench_dirnode.py, use a real CacheDirectoryManager instead of a fake one (because CacheDirectoryManager is a significant user of CPU and/or time)
Zooko O'Whielacronx [Tue, 7 Jul 2009 03:30:35 +0000 (20:30 -0700)]
directories: make the profiling behavior of bench_dirnode.py accessible by adding '--profile' to the cmdline
Zooko O'Whielacronx [Mon, 6 Jul 2009 14:08:15 +0000 (07:08 -0700)]
setup: increase requirement on pycryptopp to >= 0.5.15
Zooko O'Whielacronx [Sun, 5 Jul 2009 16:29:53 +0000 (09:29 -0700)]
directories: update the directory benchmarks to exercise the unpack-and-repack functionality, and add optional profiling
Zooko O'Whielacronx [Sat, 4 Jul 2009 22:43:00 +0000 (15:43 -0700)]
test/benchmark: benchmark the time to pack and unpack dirnodes
See also the prof_benchmarks() function in this file which will run the benchmarks under profiling.
Zooko O'Whielacronx [Fri, 3 Jul 2009 17:57:02 +0000 (10:57 -0700)]
setup: the mac-exe build (using py2app) requires macholib>=1.2.
Here is someone else's description of this problem:
http://rtmpy.org/ticket/7
Brian Warner [Fri, 3 Jul 2009 07:28:04 +0000 (00:28 -0700)]
clean up debian packaging: we have control files for etch/lenny/sid, and
everything else uses one of those. Add dependency on python-pysqlite2 for
platforms that use py2.4 by default. Update foolscap dependency to 0.4.1.
Brian Warner [Fri, 3 Jul 2009 01:43:03 +0000 (18:43 -0700)]
update NEWS to cover all recent changes, sort by end-user importance
Brian Warner [Fri, 3 Jul 2009 01:07:49 +0000 (18:07 -0700)]
Tolerate unknown URI types in directory structures. Part of #683.
The idea is that future versions of Tahoe will add new URI types that this
version won't recognize, but might store them in directories that we *can*
read. We should handle these "objects from the future" as best we can.
Previous releases of Tahoe would just explode. With this change, we'll
continue to be able to work with everything else in the directory.
The code change is to wrap anything we don't recognize as an UnknownNode
instance (as opposed to a FileNode or DirectoryNode). Then webapi knows how
to render these (mostly by leaving fields blank), deep-check knows to skip
over them, deep-stats counts them in "count-unknown". You can rename and
delete these things, but you can't add new ones (because we wouldn't know how
to generate a readcap to put into the dirnode's rocap slot, and because this
lets us catch typos better).
Brian Warner [Fri, 3 Jul 2009 00:54:50 +0000 (17:54 -0700)]
test_client.py: minor refactoring of BASECONFIG usage
Brian Warner [Thu, 2 Jul 2009 22:25:37 +0000 (15:25 -0700)]
create_node_from_uri: take both writecap+readcap, move logic out of dirnode.py
Brian Warner [Thu, 2 Jul 2009 21:12:54 +0000 (14:12 -0700)]
dirnode.py: prepare to preserve both rwcap+rocap when copying
This will make it easier to tolerate unknown nodes safely.
Brian Warner [Thu, 2 Jul 2009 01:57:28 +0000 (18:57 -0700)]
interfaces.py: wrap some lines to 80cols
Brian Warner [Wed, 1 Jul 2009 20:05:48 +0000 (13:05 -0700)]
cli: webopen: when called with no arguments, open the Welcome page
Brian Warner [Wed, 1 Jul 2009 01:13:43 +0000 (18:13 -0700)]
mutable repairer: skip repair of readcaps instead of throwing an exception.
This should improve the behavior of #625 a bit: at least all the files will
get repaired.
Brian Warner [Wed, 1 Jul 2009 00:19:25 +0000 (17:19 -0700)]
interfaces.py: improve ICheckAndRepairResults docs a bit
Brian Warner [Wed, 1 Jul 2009 00:00:47 +0000 (17:00 -0700)]
repairer.py: wrap to 80cols. No code changes.
Zooko O'Whielacronx [Tue, 30 Jun 2009 18:48:07 +0000 (11:48 -0700)]
setup: require pycryptopp>=0.5.14 if on Windows and with Python>=2.6
Brian Warner [Tue, 30 Jun 2009 17:41:15 +0000 (10:41 -0700)]
edit NEWS
Brian Warner [Tue, 30 Jun 2009 17:07:34 +0000 (10:07 -0700)]
NEWS: list all user-visible changes since 1.4.1 . Needs lots of editing.
Brian Warner [Mon, 29 Jun 2009 22:46:58 +0000 (15:46 -0700)]
* find-trailing-spaces.py: exit rc=1 if whitespace found, to be a pre-commit hook
Brian Warner [Mon, 29 Jun 2009 22:31:24 +0000 (15:31 -0700)]
use 522-bit RSA keys in all unit tests (except one)
This reduces the total test time on my laptop from 400s to 283s.
* src/allmydata/test/test_system.py (SystemTest.test_mutable._test_debug):
Remove assertion about container_size/data_size, this changes with keysize
and was too variable anyways.
* src/allmydata/mutable/filenode.py (MutableFileNode.create): add keysize=
* src/allmydata/dirnode.py (NewDirectoryNode.create): same
* src/allmydata/client.py (Client.DEFAULT_MUTABLE_KEYSIZE): add default,
this overrides the one in MutableFileNode
Brian Warner [Mon, 29 Jun 2009 20:03:58 +0000 (13:03 -0700)]
remove trailing whitespace
Brian Warner [Mon, 29 Jun 2009 20:03:41 +0000 (13:03 -0700)]
remove trailing whitespace from some util classes
Brian Warner [Fri, 26 Jun 2009 06:32:30 +0000 (23:32 -0700)]
repairer: raise a better exception when faced with a readonly filenode. Still
produces an error, though.
Brian Warner [Fri, 26 Jun 2009 06:20:22 +0000 (23:20 -0700)]
mutable/filenode.py: set _writekey to None, rather than leaving it missing
This will at least turn the really really weird error when a repair of a
readonly mutable file is attempted into a merely really weird assertion that
mentions "repair currently requires a writecap".
Brian Warner [Fri, 26 Jun 2009 05:26:27 +0000 (22:26 -0700)]
one last tweak
Brian Warner [Fri, 26 Jun 2009 05:16:52 +0000 (22:16 -0700)]
more 'tahoe cp' docs tweaks
Brian Warner [Fri, 26 Jun 2009 02:45:57 +0000 (19:45 -0700)]
cli.py: minor tweaks to test posthooks
Brian Warner [Fri, 26 Jun 2009 02:38:37 +0000 (19:38 -0700)]
cli.py: update comments on 'tahoe cp' --help a bit
Brian Warner [Thu, 25 Jun 2009 23:57:51 +0000 (16:57 -0700)]
cli: add some --help text to 'tahoe cp'
Brian Warner [Thu, 25 Jun 2009 23:18:24 +0000 (16:18 -0700)]
make it possible to add/renew-lease from the WUI
add add/renew-lease checkbox on the "more info" page check/deep-check forms
Brian Warner [Thu, 25 Jun 2009 02:18:09 +0000 (19:18 -0700)]
hush pyflakes
Brian Warner [Thu, 25 Jun 2009 02:17:07 +0000 (19:17 -0700)]
Split out NoSharesError, stop adding attributes to NotEnoughSharesError, change humanize_failure to include the original exception string, update tests, behave better if humanize_failure fails.
Brian Warner [Wed, 24 Jun 2009 19:40:38 +0000 (12:40 -0700)]
remove introducer/old.py, will create something similar when the RIIntroducer changes
midnightmagic [Wed, 17 Jun 2009 23:36:47 +0000 (16:36 -0700)]
Basically just a trivial platform detection patch for NetBSD.
Brian Warner [Tue, 23 Jun 2009 02:28:33 +0000 (19:28 -0700)]
check_grid.py: print stderr when a subcommand fails
Brian Warner [Tue, 23 Jun 2009 02:10:47 +0000 (19:10 -0700)]
big rework of introducer client: change local API, split division of responsibilites better, remove old-code testing, improve error logging
Brian Warner [Tue, 23 Jun 2009 02:09:09 +0000 (19:09 -0700)]
web/welcome.xhtml: remove trailing whitespace
Brian Warner [Tue, 23 Jun 2009 02:08:26 +0000 (19:08 -0700)]
rrefutil: add trap_deafref utility, to make the callRemote-plus-ignore-DeadReferenceError-plug-log-other-errors pattern easier
Brian Warner [Tue, 23 Jun 2009 02:07:31 +0000 (19:07 -0700)]
PollMixin: snoop trial's error observer to halt the test early if an error is seen. This turns a lot of timeouts into fast failures.
Brian Warner [Sun, 21 Jun 2009 23:51:19 +0000 (16:51 -0700)]
clean up storage_broker interface: should fix #732
Brian Warner [Sun, 21 Jun 2009 23:17:57 +0000 (16:17 -0700)]
hush pyflakes with recent FileTooLarge removal
Brian Warner [Sun, 21 Jun 2009 19:12:04 +0000 (12:12 -0700)]
add docs/proposed/GridID.txt (cleaning out some of my old branches)
Zooko O'Whielacronx [Sun, 21 Jun 2009 17:50:05 +0000 (10:50 -0700)]
docs: remove warning about inability to build modules on py2.6 on Windows with mingw, differentiate between clients and servers, reflow to a consistent column width (79), add hint about firewall/NAT docs.
Zooko O'Whielacronx [Sun, 21 Jun 2009 05:58:46 +0000 (22:58 -0700)]
setup: copy in misc/show-tools-version.py from zfec -- it prints out platform and setuptools versions
Zooko O'Whielacronx [Sun, 21 Jun 2009 05:51:14 +0000 (22:51 -0700)]
docs: start updating the NEWS and relnotes.txt files, add Kevan to CREDITS
kevan [Sat, 20 Jun 2009 21:28:22 +0000 (14:28 -0700)]
change max filesize limit tests
Instead of testing to see that the previous SDMF filesize limit was being
obeyed, we now test to make sure that we can insert files larger than that
limit.
kevan [Sat, 20 Jun 2009 21:31:30 +0000 (14:31 -0700)]
remove upper limit on SDMF filesize
Brian Warner [Sun, 21 Jun 2009 04:12:09 +0000 (21:12 -0700)]
immutable/download: instrument do-you-have-block responses to investigate #732
Brian Warner [Thu, 18 Jun 2009 05:05:02 +0000 (22:05 -0700)]
Makefile: add jaunty support, rearrange debian sections in order of release
Brian Warner [Thu, 18 Jun 2009 04:57:09 +0000 (21:57 -0700)]
test_util: add known-answer tests for hashutil tags
Zooko O'Whielacronx [Sat, 13 Jun 2009 19:57:03 +0000 (12:57 -0700)]
tests: bump up a timeout that expired on Zandr's box
Zooko O'Whielacronx [Sat, 13 Jun 2009 16:01:12 +0000 (09:01 -0700)]
util: hooray! A clean implementation of this simple utility! Black Dew pointed out that the inverse of time.gmtime() is hidden in the "calendar" module.
Zooko O'Whielacronx [Fri, 12 Jun 2009 20:45:56 +0000 (13:45 -0700)]
util: Brian's horrible hack to figure out how much localtime and utctime differ. Now we'll see if it works on Windows.
Zooko O'Whielacronx [Fri, 12 Jun 2009 00:09:20 +0000 (17:09 -0700)]
util: oops, time.tzset() doesn't work on Windows -- hopefully the new "London" unit test passes on Windows when we skip tzset() on platforms that don't have it
Zooko O'Whielacronx [Thu, 11 Jun 2009 22:55:06 +0000 (15:55 -0700)]
setup: edit install.html to warn Windows users away from Python v2.6
Zooko O'Whielacronx [Thu, 11 Jun 2009 22:11:29 +0000 (15:11 -0700)]
util: fix time_format.iso_utc_time_to_seconds() so that it works even in London
Zooko O'Whielacronx [Thu, 11 Jun 2009 19:32:14 +0000 (12:32 -0700)]
setup: run the same "make quicktest" on Windows as on non-Windows
I checked and it behaves about as well on Windows as the previous version did.
Zooko O'Whielacronx [Wed, 10 Jun 2009 16:10:43 +0000 (09:10 -0700)]
tests: significantly increase timeouts that triggered on Zandr's ARM box
Zooko O'Whielacronx [Wed, 10 Jun 2009 12:56:39 +0000 (05:56 -0700)]
test: multiple by 10 or so all timeouts that Zandr's ARM box just overran
Zooko O'Whielacronx [Wed, 10 Jun 2009 04:46:28 +0000 (21:46 -0700)]
tests: bump up timeout on a test that timed out on draco
Zooko O'Whielacronx [Tue, 9 Jun 2009 21:05:09 +0000 (14:05 -0700)]
tests: raise the timeout for test_cli since Zandr's ARM machine totally burst through the old one
Brian Warner [Tue, 9 Jun 2009 05:28:01 +0000 (22:28 -0700)]
test_cli.Backup: increase timeout massively, it takes 1200s on zandr's ARM linkstation
Zooko O'Whielacronx [Tue, 9 Jun 2009 02:17:53 +0000 (19:17 -0700)]
tests: double the timeouts on some tests which time-out on Francois's box
Zooko O'Whielacronx [Mon, 8 Jun 2009 22:55:57 +0000 (15:55 -0700)]
tests: bump up timeouts so that the tests can finish before timeout on Francois's little arm box
Zooko O'Whielacronx [Fri, 5 Jun 2009 14:34:37 +0000 (07:34 -0700)]
tests: increase timeouts on some other tests that timed-out on Francois's arm box
Zooko O'Whielacronx [Fri, 5 Jun 2009 03:14:44 +0000 (20:14 -0700)]
tests: bump up the timeout on a bunch of tests that took longer than the default timeout (120s) on François Lenny-armv5tel
Zooko O'Whielacronx [Thu, 4 Jun 2009 17:31:31 +0000 (10:31 -0700)]
backup: remove the --no-backupdb command, the handling of "can't import sqlite", and the related tests, and change an error message to more correctly indicate failure to load the database from disk rather than failure to import sqlite module
Fixes #728.
Zooko O'Whielacronx [Thu, 4 Jun 2009 15:45:48 +0000 (08:45 -0700)]
setup: require pysqlite >= v2.0.5. if we are running on Python < 2.5
Zooko O'Whielacronx [Thu, 4 Jun 2009 15:37:28 +0000 (08:37 -0700)]
setup: add pysqlite and sqlite to get_package_versions()
Brian Warner [Tue, 2 Jun 2009 03:07:50 +0000 (20:07 -0700)]
more refactoring: move get_all_serverids() and get_nickname_for_serverid() from Client to storage_broker
Brian Warner [Tue, 2 Jun 2009 02:25:11 +0000 (19:25 -0700)]
more storage_broker refactoring: downloader gets a broker instead of a client,
use Client.get_storage_broker() accessor instead of direct attribute access.
Brian Warner [Mon, 1 Jun 2009 22:50:07 +0000 (15:50 -0700)]
test_runner.py: remove test_client_no_noise: the issue in question is
ticketed in http://divmod.org/trac/ticket/2830 and doesn't need a Tahoe-side
change, plus this test fails on win32 for unrelated reasons (and test_client
is the place to think about the win32 issue).