Brian Warner [Sat, 21 Feb 2009 07:17:10 +0000 (00:17 -0700)]
test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path
Brian Warner [Sat, 21 Feb 2009 07:15:33 +0000 (00:15 -0700)]
crawler: tolerate low-resolution system clocks (i.e. windows)
Brian Warner [Sat, 21 Feb 2009 04:58:31 +0000 (21:58 -0700)]
BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise
Brian Warner [Sat, 21 Feb 2009 04:57:55 +0000 (21:57 -0700)]
test_storage: startService the server, as is now the standard practice
Brian Warner [Sat, 21 Feb 2009 04:57:20 +0000 (21:57 -0700)]
crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early
Brian Warner [Sat, 21 Feb 2009 04:46:06 +0000 (21:46 -0700)]
BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate
Brian Warner [Sat, 21 Feb 2009 04:28:56 +0000 (21:28 -0700)]
storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better
Brian Warner [Sat, 21 Feb 2009 04:04:08 +0000 (21:04 -0700)]
storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page
Brian Warner [Sat, 21 Feb 2009 04:03:09 +0000 (21:03 -0700)]
storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py
Brian Warner [Sat, 21 Feb 2009 01:27:43 +0000 (18:27 -0700)]
crawler: add get_progress, clean up get_state
Brian Warner [Fri, 20 Feb 2009 23:03:53 +0000 (16:03 -0700)]
web/storage: make sure we can handle platforms without os.statvfs too
Brian Warner [Fri, 20 Feb 2009 22:19:11 +0000 (15:19 -0700)]
crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters
Brian Warner [Fri, 20 Feb 2009 21:29:26 +0000 (14:29 -0700)]
web: add Storage status page, improve tests
Brian Warner [Fri, 20 Feb 2009 21:29:20 +0000 (14:29 -0700)]
storage: include reserved_space in stats
Brian Warner [Fri, 20 Feb 2009 19:29:22 +0000 (12:29 -0700)]
web/check_results: sort share identifiers in the sharemap display
Brian Warner [Fri, 20 Feb 2009 19:15:54 +0000 (12:15 -0700)]
webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation
Brian Warner [Fri, 20 Feb 2009 03:22:54 +0000 (20:22 -0700)]
test/no_network: do startService on the storage servers, make it easier to customize the storage servers
Brian Warner [Fri, 20 Feb 2009 02:31:42 +0000 (19:31 -0700)]
crawler: modify API to support upcoming bucket-counting crawler
Brian Warner [Fri, 20 Feb 2009 01:04:09 +0000 (18:04 -0700)]
test_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2 on OS-X is easily fooled
Brian Warner [Thu, 19 Feb 2009 23:29:18 +0000 (16:29 -0700)]
web/reliability: add parameter descriptions, adapted from a patch from Terrell Russell.
Brian Warner [Thu, 19 Feb 2009 21:23:40 +0000 (14:23 -0700)]
test_crawler: hush pyflakes
Brian Warner [Thu, 19 Feb 2009 21:16:54 +0000 (14:16 -0700)]
test_crawler: disable the percentage-of-cpu-used test, since it is too unreliable on our slow buildslaves. But leave the code in place for developers to run by hand.
Brian Warner [Thu, 19 Feb 2009 20:55:15 +0000 (13:55 -0700)]
reliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell Russell for the help.
Brian Warner [Thu, 19 Feb 2009 08:44:35 +0000 (01:44 -0700)]
reliability: switch to NumPy, since Numeric is deprecated
Brian Warner [Thu, 19 Feb 2009 08:36:43 +0000 (01:36 -0700)]
setup.py: fix pyflakes complaints
Brian Warner [Thu, 19 Feb 2009 08:35:58 +0000 (01:35 -0700)]
move show-tool-versions out of setup.py and into a separate script in misc/ , since setuptools is trying to build and install a bunch of stuff first
Brian Warner [Thu, 19 Feb 2009 07:58:18 +0000 (00:58 -0700)]
test_crawler: don't require >=1 cycle on cygwin
Brian Warner [Thu, 19 Feb 2009 07:24:36 +0000 (00:24 -0700)]
setup.py: add show_tool_versions command, for the benefit of a new buildbot step
Brian Warner [Thu, 19 Feb 2009 06:57:51 +0000 (23:57 -0700)]
setup.py: wrap to 80 cols, no functional changes
Brian Warner [Thu, 19 Feb 2009 06:13:42 +0000 (23:13 -0700)]
crawler: use fileutil.move_info_place in preference to our own version
Brian Warner [Thu, 19 Feb 2009 06:13:10 +0000 (23:13 -0700)]
fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows
Brian Warner [Thu, 19 Feb 2009 05:24:31 +0000 (22:24 -0700)]
crawler: fix problems on windows and our slow cygwin slave
Brian Warner [Thu, 19 Feb 2009 04:46:33 +0000 (21:46 -0700)]
#633: first version of a rate-limited interruptable share-crawler
Brian Warner [Wed, 18 Feb 2009 23:23:01 +0000 (16:23 -0700)]
change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away
Brian Warner [Wed, 18 Feb 2009 22:42:34 +0000 (15:42 -0700)]
test_system: split off checker tests to test_deepcheck.py, this file is too big
Brian Warner [Wed, 18 Feb 2009 21:46:55 +0000 (14:46 -0700)]
break storage.py into smaller pieces in storage/*.py . No behavioral changes.
Brian Warner [Wed, 18 Feb 2009 21:46:48 +0000 (14:46 -0700)]
immutable/layout: minor change to repr name
Brian Warner [Wed, 18 Feb 2009 21:41:37 +0000 (14:41 -0700)]
docs: add lease-tradeoffs diagram
Brian Warner [Wed, 18 Feb 2009 20:29:03 +0000 (13:29 -0700)]
interfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients (the first to use these calls) can tolerate future storage servers which might return something other than None
Brian Warner [Wed, 18 Feb 2009 04:22:12 +0000 (21:22 -0700)]
docs/debian.txt: minor edit
Brian Warner [Wed, 18 Feb 2009 02:32:43 +0000 (19:32 -0700)]
add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi.
Brian Warner [Wed, 18 Feb 2009 02:30:53 +0000 (19:30 -0700)]
change RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket, instead of raising IndexError, because that makes the upcoming --add-lease feature faster and less noisy
Brian Warner [Wed, 18 Feb 2009 00:15:11 +0000 (17:15 -0700)]
CLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests
Brian Warner [Tue, 17 Feb 2009 20:48:09 +0000 (13:48 -0700)]
interfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I change it
Brian Warner [Tue, 17 Feb 2009 18:08:38 +0000 (11:08 -0700)]
test_backupdb: improve error messages if the test fails
Brian Warner [Tue, 17 Feb 2009 06:35:53 +0000 (23:35 -0700)]
webapi #590: add streaming deep-check. Still need a CLI tool to use it.
Brian Warner [Tue, 17 Feb 2009 06:00:34 +0000 (23:00 -0700)]
test_web.Grid: change the CHECK() function to make it easier to test t= values with hyphens in them
Brian Warner [Tue, 17 Feb 2009 05:12:42 +0000 (22:12 -0700)]
test_web: improve checker-results coverage with a no-network -based test, enhance no-network harness to assist, fix some bugs in web/check_results.py that were exposed
Brian Warner [Tue, 17 Feb 2009 02:56:58 +0000 (19:56 -0700)]
web: fix handling of reliability page when Numeric is not available
Brian Warner [Tue, 17 Feb 2009 01:06:43 +0000 (18:06 -0700)]
test/no_network: update comments with setup timing: no_network takes 50ms, SystemTestMixin takes 2s (on my laptop)
Brian Warner [Tue, 17 Feb 2009 00:44:57 +0000 (17:44 -0700)]
test_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary
Brian Warner [Tue, 17 Feb 2009 00:36:58 +0000 (17:36 -0700)]
test_download: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary
Brian Warner [Tue, 17 Feb 2009 00:23:48 +0000 (17:23 -0700)]
test_dirnode.py: convert Deleter to new no-network gridtest
Brian Warner [Tue, 17 Feb 2009 00:20:05 +0000 (17:20 -0700)]
test_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin, which speeds it up from 73s to 43s on my system
Brian Warner [Tue, 17 Feb 2009 00:19:47 +0000 (17:19 -0700)]
tests: fix no_network framework to work with upload/download and checker
Brian Warner [Tue, 17 Feb 2009 00:15:32 +0000 (17:15 -0700)]
client.py: improve docstring
Brian Warner [Mon, 16 Feb 2009 22:08:33 +0000 (15:08 -0700)]
test_cli: add test coverage for help strings
Brian Warner [Mon, 16 Feb 2009 21:58:44 +0000 (14:58 -0700)]
test/no_network: new test harness, like system-test but doesn't use the network so it's faster
Brian Warner [Mon, 16 Feb 2009 21:58:16 +0000 (14:58 -0700)]
interfaces.py: minor docstring edit
Zooko O'Whielacronx [Mon, 16 Feb 2009 21:15:58 +0000 (14:15 -0700)]
setup: if you sdist_dsc (to produce the input files for dpkg-buildpackage) then run darcsver first
Zooko O'Whielacronx [Mon, 16 Feb 2009 21:15:39 +0000 (14:15 -0700)]
doc: a few edits to docs made after the 1.3.0 release
Brian Warner [Mon, 16 Feb 2009 04:04:51 +0000 (21:04 -0700)]
test_cli: improve test coverage slightly
Brian Warner [Mon, 16 Feb 2009 03:32:10 +0000 (20:32 -0700)]
test_util: get almost full test coverage of dictutil, starting with the original pyutil tests as a base. The remaining three uncovered lines involve funny cases of ValueOrderedDict that I can't figure out how to get at
Brian Warner [Sun, 15 Feb 2009 23:24:51 +0000 (16:24 -0700)]
provisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links
Brian Warner [Sun, 15 Feb 2009 23:23:26 +0000 (16:23 -0700)]
util/statistics: add tests, fix mean_repair_cost
Brian Warner [Sun, 15 Feb 2009 23:23:10 +0000 (16:23 -0700)]
test_repairer: hush pyflakes
Brian Warner [Sun, 15 Feb 2009 23:19:05 +0000 (16:19 -0700)]
lossmodel.lyx: move draft paper into docs/proposed/, since it's unfinished
Brian Warner [Sat, 14 Feb 2009 00:42:34 +0000 (17:42 -0700)]
build a 'reliability' web page, with a simulation of file decay and repair over time
Shawn Willden [Fri, 16 Jan 2009 03:56:48 +0000 (20:56 -0700)]
More lossmodel work, on repair.
Shawn Willden [Thu, 15 Jan 2009 04:00:58 +0000 (21:00 -0700)]
Loss model work (temp1)
Shawn Willden [Wed, 14 Jan 2009 03:12:35 +0000 (20:12 -0700)]
Statistics module
Added a statistics module for calculating various facets of
share survival statistics.
Zooko O'Whielacronx [Sun, 15 Feb 2009 17:35:10 +0000 (10:35 -0700)]
docs: relnotes-short.txt
linkedin.com imposed a strict limit on the number of characters I could post. This forced me to prune and prune and edit and edit until relnotes.txt was a quarter of its former size. Here's the short version.
Zooko O'Whielacronx [Sat, 14 Feb 2009 01:05:56 +0000 (18:05 -0700)]
TAG allmydata-tahoe-1.3.0
Zooko O'Whielacronx [Sat, 14 Feb 2009 01:05:00 +0000 (18:05 -0700)]
docs: a few last-minute edits to the docs for 1.3.0 (also this patch will accompany the tag and conveniently trigger the buildbots to build a 1.3.0 version)
Zooko O'Whielacronx [Sat, 14 Feb 2009 00:49:39 +0000 (17:49 -0700)]
setup: remove attempt to automatically satisfy dependency on pywin32
Brian Warner [Fri, 13 Feb 2009 23:55:34 +0000 (16:55 -0700)]
NEWS: mention #625 (deep-repair breaks when it tries to repair a read-only directory)
Brian Warner [Fri, 13 Feb 2009 21:53:37 +0000 (14:53 -0700)]
test_dirnode: #625 run deep-check on a readonly dirnode too
Zooko O'Whielacronx [Fri, 13 Feb 2009 21:12:45 +0000 (14:12 -0700)]
docs: CREDITS
Brian Warner [Fri, 13 Feb 2009 16:57:53 +0000 (09:57 -0700)]
figleaf_htmlizer: fix order of summary counters
Brian Warner [Fri, 13 Feb 2009 08:23:41 +0000 (01:23 -0700)]
Makefile: minor, add .PHONY declarations
Brian Warner [Fri, 13 Feb 2009 07:00:22 +0000 (00:00 -0700)]
figleaf_htmlizer: oops, re-ignore files that aren't under root, like code in auto-built eggs
Zooko O'Whielacronx [Fri, 13 Feb 2009 05:41:21 +0000 (22:41 -0700)]
setup: relnotes.txt mention the iPhone app and CIFS/SMB (tahoe-w32-client)
Zooko O'Whielacronx [Fri, 13 Feb 2009 05:37:38 +0000 (22:37 -0700)]
setup: make the "full version string" be "allmydata-tahoe/1.3.0" instead of "allmydata-tahoe-1.3.0" and the UserAgent string of the cli be "allmydata-tahoe/1.3.0 (tahoe-client)"
This is webbish. Thanks to kpreid for suggesting it.
Zooko O'Whielacronx [Fri, 13 Feb 2009 05:28:14 +0000 (22:28 -0700)]
docs: relnotes.txt final (!?) update for 1.3.0!
Zooko O'Whielacronx [Fri, 13 Feb 2009 05:16:21 +0000 (22:16 -0700)]
docs: known_issues.txt: my version of #615, remove "issue numbers", edits, move tahoe-1.1.0 issues to historical
Zooko O'Whielacronx [Fri, 13 Feb 2009 04:42:28 +0000 (21:42 -0700)]
docs: CREDITS
Brian Warner [Thu, 12 Feb 2009 22:18:29 +0000 (15:18 -0700)]
Makefile: add figleaf-delta-output, to render figleaf coverage differences with a previous run
Brian Warner [Thu, 12 Feb 2009 22:10:20 +0000 (15:10 -0700)]
figleaf_htmlizer: emit stats to stdout, so buildbot can see it
Brian Warner [Thu, 12 Feb 2009 22:04:12 +0000 (15:04 -0700)]
figleaf_htmlizer: render changes in coverage relative to a previous test run using --old-coverage
Zooko O'Whielacronx [Fri, 13 Feb 2009 00:04:47 +0000 (17:04 -0700)]
immutable repairer: fix DownUpConnector so that it satisfies short reads the were requested after the last write and before the close
This is probably the cause of the very rare "loss of progress" bug. This is tested by unit tests. A recent patch changed this to errback instead of losing progress, and now this patch is changing it again to return a short read instead of errbacking. Returning a short read is what the uploader (in encode.py) is expecting, when it is reading the last block of the ciphertext, which might be shorter than the other blocks.
Zooko O'Whielacronx [Thu, 12 Feb 2009 23:31:32 +0000 (16:31 -0700)]
immutable repairer: add an assertion that a certain value in this tricky function is always what I think it is
Zooko O'Whielacronx [Thu, 12 Feb 2009 23:48:53 +0000 (16:48 -0700)]
immutable repairer: add tests of how it handles short reads
Zooko O'Whielacronx [Thu, 12 Feb 2009 22:21:31 +0000 (15:21 -0700)]
nodeadmin: node stops itself if a hotline file hasn't been touched in 60 seconds now, instead of in 40 seconds
A test failed on draco (MacPPC) because it took 49 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 40 seconds old.
Zooko O'Whielacronx [Thu, 12 Feb 2009 03:11:29 +0000 (20:11 -0700)]
immutable repairer: errback any pending readers of DownUpConnectorwhen it runs out of bytes, and test that fact
Brian Warner [Thu, 12 Feb 2009 03:05:15 +0000 (20:05 -0700)]
figleaf_htmlizer: more rearranging, behavior should still be unchanged
Brian Warner [Thu, 12 Feb 2009 02:56:07 +0000 (19:56 -0700)]
figleaf_htmlizer: break it up into subfunctions, behavior should still be unchanged
Brian Warner [Thu, 12 Feb 2009 02:40:50 +0000 (19:40 -0700)]
figleaf_htmlizer: rewrite in class form, behavior should be the same as before
Brian Warner [Thu, 12 Feb 2009 02:16:43 +0000 (19:16 -0700)]
figleaf_htmlizer: rewrite with twisted.python.usage, remove logging: should behave the same as before
Brian Warner [Thu, 12 Feb 2009 02:05:42 +0000 (19:05 -0700)]
figleaf_htmlizer: expand tabs, fix to 4-space indents. No functional changes.
Brian Warner [Thu, 12 Feb 2009 01:09:13 +0000 (18:09 -0700)]
Makefile: include the figleaf pickle in the uploaded coverage data, for later differential analysis