Brian Warner [Thu, 19 Mar 2009 00:21:38 +0000 (17:21 -0700)]
expirer: change setup, config options, in preparation for adding tahoe.cfg controls
Brian Warner [Wed, 18 Mar 2009 20:25:04 +0000 (13:25 -0700)]
expirer: track mutable-vs-immutable sharecounts and sizes, report them on the web status page for comparison
Brian Warner [Tue, 17 Mar 2009 07:01:17 +0000 (00:01 -0700)]
add utility function to parse durations, for lease-expiration config
Brian Warner [Tue, 17 Mar 2009 06:51:18 +0000 (23:51 -0700)]
expirer: add mode to expire only-mutable or only-immutable shares
Brian Warner [Tue, 17 Mar 2009 05:12:06 +0000 (22:12 -0700)]
tahoe_ls.py: add comment about error cases to improve
Brian Warner [Tue, 17 Mar 2009 05:10:41 +0000 (22:10 -0700)]
GC: add date-cutoff -based expiration, add proposed docs
Brian Warner [Mon, 16 Mar 2009 00:51:34 +0000 (17:51 -0700)]
test_deepcheck: remove the 10s timeout: our dapper buildslave requires 30s, and the reduced timeout was only there because this tests fails by timeout rather than explicitly
Brian Warner [Sun, 15 Mar 2009 23:19:58 +0000 (16:19 -0700)]
tahoe cp -r: add --caps-only flag, to write filecaps into local files instead of actual file contents. Used only for debugging and as a quick tree-comparison tool.
Brian Warner [Fri, 13 Mar 2009 23:31:35 +0000 (16:31 -0700)]
dirnode deep_traverse: insert a turn break (fireEventually) at least once every 100 files, otherwise a CHK followed by more than 158 LITs can overflow the stack, sort of like #237.
Brian Warner [Fri, 13 Mar 2009 08:27:51 +0000 (01:27 -0700)]
consolidate: remove pointless 'else' after for loop
Brian Warner [Fri, 13 Mar 2009 08:24:51 +0000 (01:24 -0700)]
consolidate: add eta, flush stdout
Brian Warner [Fri, 13 Mar 2009 06:54:02 +0000 (23:54 -0700)]
consolidate: tolerate unicode dirnames
Brian Warner [Fri, 13 Mar 2009 06:50:46 +0000 (23:50 -0700)]
dirnode.py: when doing deep-traverse, walk each directory in alphabetical order, to make things like 'manifest' more predictable
Brian Warner [Fri, 13 Mar 2009 05:47:28 +0000 (22:47 -0700)]
consolidator: add progress to scan-old-directory passes
Brian Warner [Fri, 13 Mar 2009 04:22:29 +0000 (21:22 -0700)]
consolidator: fix cycle detection to not trigger on merely shared directories, add snapshot counter to progress
Brian Warner [Fri, 13 Mar 2009 03:48:01 +0000 (20:48 -0700)]
consolidator: re-use more directories, add total directories seen-vs-used counts
Brian Warner [Fri, 13 Mar 2009 01:16:00 +0000 (18:16 -0700)]
tahoe_backup.py: tolerate more time formats
Brian Warner [Thu, 12 Mar 2009 23:29:00 +0000 (16:29 -0700)]
consolidator: add more verbose traversal of directories
Brian Warner [Thu, 12 Mar 2009 23:04:27 +0000 (16:04 -0700)]
consolidate: create multiple numbered backups of the original Archives directory, not just the first time
Brian Warner [Thu, 12 Mar 2009 20:56:06 +0000 (13:56 -0700)]
add 'tahoe debug consolidate' command, to merge directories created by repeated 'tahoe cp -r' or the allmydata win32 backup tool, into the form that would have been created by 'tahoe backup'.
Brian Warner [Thu, 12 Mar 2009 20:53:45 +0000 (13:53 -0700)]
tahoe_cp.py: return 0 for success, instead of None
Zooko O'Whielacronx [Wed, 11 Mar 2009 22:59:02 +0000 (15:59 -0700)]
setup: specify in the debian/control files that tahoe is compatible with Python 2.6
Zooko O'Whielacronx [Wed, 11 Mar 2009 15:29:52 +0000 (08:29 -0700)]
setup: update the debian/copyright text to reflect the current licences
Brian Warner [Mon, 9 Mar 2009 20:42:17 +0000 (13:42 -0700)]
expirer: fix prediction math, thanks to Zandr for the catch
Brian Warner [Mon, 9 Mar 2009 03:55:16 +0000 (20:55 -0700)]
storage.expirer: oops, fix upgrade-handler code
Brian Warner [Mon, 9 Mar 2009 03:42:20 +0000 (20:42 -0700)]
storage.expirer: handle upgrades better
Brian Warner [Mon, 9 Mar 2009 03:38:28 +0000 (20:38 -0700)]
storage.expirer: exercise the last missing line of webstatus code
Brian Warner [Mon, 9 Mar 2009 03:08:40 +0000 (20:08 -0700)]
expirer: tolerate corrupt shares, add them to the state and history for future examination
Brian Warner [Mon, 9 Mar 2009 03:07:32 +0000 (20:07 -0700)]
storage/immutable: raise a specific error upon seeing a bad version number, instead of using assert. Also wrap to 80cols.
Brian Warner [Mon, 9 Mar 2009 03:05:27 +0000 (20:05 -0700)]
storage: add test coverage for BucketReader.__repr__
Brian Warner [Mon, 9 Mar 2009 02:02:01 +0000 (19:02 -0700)]
storage/mutable: raise a specific error upon seeing bad magic, instead of using assert
Zooko O'Whielacronx [Sun, 8 Mar 2009 03:50:39 +0000 (20:50 -0700)]
nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds
A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old.
Brian Warner [Sat, 7 Mar 2009 23:14:42 +0000 (16:14 -0700)]
expirer: make web display a bit more consistent
Brian Warner [Sat, 7 Mar 2009 23:02:43 +0000 (16:02 -0700)]
web/storage.py: tolerate unknown-future displays, I'm not sure why LeaseCrawler.test_unpredictable_future didn't catch this
Brian Warner [Sat, 7 Mar 2009 12:40:51 +0000 (05:40 -0700)]
tahoe_cp.py: improve error reporting slightly: don't json-interpret HTTP errors, pass through tahoe webapi error messages
Brian Warner [Sat, 7 Mar 2009 12:08:15 +0000 (05:08 -0700)]
test_cli.py: assert that 'ls' on an unrecoverable file now gives a better error message
Brian Warner [Sat, 7 Mar 2009 11:57:07 +0000 (04:57 -0700)]
test_web: test behavior of broken-dirnode GET, both html and json
Brian Warner [Sat, 7 Mar 2009 11:56:01 +0000 (04:56 -0700)]
web: when a dirnode can't be read, emit a regular HTML page but with the child-table and upload-forms replaced with an apologetic message. Make sure to include the 'get info' links so the user can do a filecheck
Brian Warner [Sat, 7 Mar 2009 11:54:08 +0000 (04:54 -0700)]
web/common: split out exception-to-explanation+code mapping to a separate humanize_failure() function, so it can be used by other code. Add explanation for mutable UnrecoverableFileError.
Brian Warner [Sat, 7 Mar 2009 10:04:28 +0000 (03:04 -0700)]
test_cli: use explicit (and stable) testdirs, instead of using self.mktemp
Brian Warner [Sat, 7 Mar 2009 09:45:18 +0000 (02:45 -0700)]
test_storage: solaris doesn't appear to give the same block count as other platforms, so don't assert as much about 'diskbytes' recovered
Brian Warner [Sat, 7 Mar 2009 09:44:21 +0000 (02:44 -0700)]
tahoe_cp.py: don't be verbose unless --verbose is passed: default is one line of success
Brian Warner [Sat, 7 Mar 2009 05:57:53 +0000 (22:57 -0700)]
setup.cfg: use allmydata.org as a mirror for some of our necessary dependencies
Brian Warner [Sat, 7 Mar 2009 05:45:17 +0000 (22:45 -0700)]
storage: add a lease-checker-and-expirer crawler, plus web status page.
This walks slowly through all shares, examining their leases, deciding which
are still valid and which have expired. Once enabled, it will then remove the
expired leases, and delete shares which no longer have any valid leases. Note
that there is not yet a tahoe.cfg option to enable lease-deletion: the
current code is read-only. A subsequent patch will add a tahoe.cfg knob to
control this, as well as docs. Some other minor items included in this patch:
tahoe debug dump-share has a new --leases-only flag
storage sharefile/leaseinfo code is cleaned up
storage web status page (/storage) has more info, more tests coverage
space-left measurement on OS-X should be more accurate (it was off by 2048x)
(use stat .f_frsize instead of f_bsize)
Zooko O'Whielacronx [Sat, 7 Mar 2009 05:43:32 +0000 (22:43 -0700)]
setup: comment-out the use of the directory full of dependency packages on the test grid -- the test grid is too unreliable! :-(
Zooko O'Whielacronx [Fri, 6 Mar 2009 20:10:57 +0000 (13:10 -0700)]
setup: create a "make deb-lenny-head" target
I made this patch by copying [
20090305220021-92b7f-
89d987c7d05306b5cb03a64f2956a652c10a7296] and changing the name from "intrepid" to "lenny". I haven't tested it.
Zooko O'Whielacronx [Thu, 5 Mar 2009 23:32:04 +0000 (16:32 -0700)]
trivial: remove unused import detected by pyflakes, and remove trailing whitespace
Zooko O'Whielacronx [Thu, 5 Mar 2009 23:00:21 +0000 (16:00 -0700)]
setup: enable build of .debs for Ubuntu Intrepid, thanks to DarKNesS_WolF
Zooko O'Whielacronx [Sun, 22 Feb 2009 18:57:51 +0000 (11:57 -0700)]
trivial: use more specific function for ascii-encoding storage index
Zooko O'Whielacronx [Sun, 22 Feb 2009 18:57:39 +0000 (11:57 -0700)]
doc: update how_to_make_a_tahoe_release.txt
Brian Warner [Wed, 4 Mar 2009 05:11:46 +0000 (22:11 -0700)]
test_cli: validate non-HTML error response of 'tahoe get' on an unrecoverable file
Brian Warner [Wed, 4 Mar 2009 04:56:30 +0000 (21:56 -0700)]
web: full patch for HTML-vs-plaintext traceback renderings, improve test coverage of exception rendering
Brian Warner [Wed, 4 Mar 2009 04:54:57 +0000 (21:54 -0700)]
web/common.py: use 'Accept:' header to control HTML-vs-text/plain traceback renderings
Brian Warner [Wed, 4 Mar 2009 02:40:59 +0000 (19:40 -0700)]
test/common: rearrange shouldHTTPError
Brian Warner [Wed, 4 Mar 2009 02:40:19 +0000 (19:40 -0700)]
web: move plural() to common.py
Brian Warner [Wed, 4 Mar 2009 02:37:15 +0000 (19:37 -0700)]
add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash
Brian Warner [Tue, 3 Mar 2009 23:56:20 +0000 (16:56 -0700)]
test_web: move shouldHTTPError into common.WebErrorMixin
Brian Warner [Tue, 3 Mar 2009 23:46:04 +0000 (16:46 -0700)]
test_web: clean up shouldHTTPError methods
Brian Warner [Fri, 27 Feb 2009 08:17:24 +0000 (01:17 -0700)]
immutable checker add-lease: catch remote IndexError here too
Brian Warner [Fri, 27 Feb 2009 08:04:26 +0000 (01:04 -0700)]
servermap add-lease: fix the code that's supposed to catch remote IndexErrors, I forgot that they present as ServerFailures instead. This should stop the deluge of Incidents that occur when you do add-lease against 1.3.0 servers
Brian Warner [Fri, 27 Feb 2009 07:59:57 +0000 (00:59 -0700)]
rrefutil: add check_remote utility function
Brian Warner [Fri, 27 Feb 2009 07:55:24 +0000 (00:55 -0700)]
rrefutil: add trap_remote utility and friends
Brian Warner [Fri, 27 Feb 2009 02:58:38 +0000 (19:58 -0700)]
web: improve layout of storage status with a table
Brian Warner [Fri, 27 Feb 2009 02:42:48 +0000 (19:42 -0700)]
crawler: add ETA to get_progress()
Alberto Berti [Thu, 26 Feb 2009 16:02:37 +0000 (09:02 -0700)]
Fix for bug #645, correct path handling logic so that it works from sshfs
Zooko O'Whielacronx [Thu, 26 Feb 2009 04:09:04 +0000 (21:09 -0700)]
wui: s/Provisioning/Reliability/ ; suggested by Terrell
Brian Warner [Thu, 26 Feb 2009 02:52:45 +0000 (19:52 -0700)]
startstop_node.py: when restarting, tolerate a stale pid too
Brian Warner [Wed, 25 Feb 2009 08:46:21 +0000 (01:46 -0700)]
web: fix the ERROR: line to work the same in python2.4 and 2.5
Brian Warner [Wed, 25 Feb 2009 07:01:28 +0000 (00:01 -0700)]
test_cli/test_web: fix spurious test failure on solaris (maybe python2.4?) due to variations in the way that exceptions are stringified
Brian Warner [Wed, 25 Feb 2009 06:44:15 +0000 (23:44 -0700)]
CLI: modify 'tahoe manifest' and 'tahoe deep-check' to report ERROR: properly. For #590.
Brian Warner [Wed, 25 Feb 2009 06:13:35 +0000 (23:13 -0700)]
webapi: modify streaming deep-manifest/deep-checker to emit an ERROR: line if they encounter an unrecoverable+untraversable directory. For #590.
Brian Warner [Wed, 25 Feb 2009 05:21:36 +0000 (22:21 -0700)]
scripts/common: fix alias handling on windows again, emit slightly nicer error message in response to an unknown alias
Brian Warner [Wed, 25 Feb 2009 04:42:13 +0000 (21:42 -0700)]
#165: make 'tahoe restart --force' the default behavior: warn but do not stop if restart is used on something that wasn't a running node, and always try to start it afterwards. This is particularly important for #315 (restart -m), because otherwise a single not-already-running node will prevent all nodes from being restarted, resulting in longer downtime than necessary
Brian Warner [Wed, 25 Feb 2009 04:04:57 +0000 (21:04 -0700)]
test_deepcheck: switch deep-check tests to use no-network too. This cuts the runtime down by about 50%
Brian Warner [Wed, 25 Feb 2009 03:00:10 +0000 (20:00 -0700)]
test_deepcheck: convert MutableChecker to no-network GridTest
Brian Warner [Wed, 25 Feb 2009 01:33:00 +0000 (18:33 -0700)]
tests/no_network: move GET into the GridTestMixin class
Brian Warner [Wed, 25 Feb 2009 01:04:59 +0000 (18:04 -0700)]
common_web.py: oops, add .fields to the other FakeRequest
Brian Warner [Wed, 25 Feb 2009 00:56:20 +0000 (17:56 -0700)]
test_cli: exercise the recent tolerate-'c:\dir\file.txt' fix in scripts/common, recorded in a separate match to make it easier to merge the fix to prod
Brian Warner [Wed, 25 Feb 2009 00:55:22 +0000 (17:55 -0700)]
scripts/common: on windows, tolerate paths like 'c:\dir\file.txt', by treating single-letter aliases on windows/cygwin as non-aliases
Brian Warner [Wed, 25 Feb 2009 00:20:50 +0000 (17:20 -0700)]
test/common_web.py: add a .fields attribute to our FakeRequest, since we support versions of Nevow that are old enough to not do it themselves
Alberto Berti [Tue, 24 Feb 2009 23:36:34 +0000 (16:36 -0700)]
Two small fixes on documentation for cli backup command.
Brian Warner [Tue, 24 Feb 2009 22:40:17 +0000 (15:40 -0700)]
test_web: add (disabled) test to see what happens when deep-check encounters an unrecoverable directory. We still need code changes to improve this behavior.
Alberto Berti [Tue, 24 Feb 2009 18:14:25 +0000 (11:14 -0700)]
Add elapsed timestamp to cli backup command final summary.
Alberto Berti [Tue, 24 Feb 2009 16:30:49 +0000 (09:30 -0700)]
Added documentation for '--exclude' and friends cli backup command.
Brian Warner [Tue, 24 Feb 2009 05:15:06 +0000 (22:15 -0700)]
test_repairer: change to use faster no_network.GridTestMixin, split Verifier tests into separate cases, refactor judgement funcs into shared methods
Brian Warner [Tue, 24 Feb 2009 05:14:05 +0000 (22:14 -0700)]
immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check
instead of examining the value returned by f.trap, because the latter appears
to squash exception types down into their base classes (i.e. since
ShareVersionIncompatible is a subclass of LayoutInvalid,
f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid).
All this resulted in 'incompatible' shares being misclassified as 'corrupt'.
Brian Warner [Tue, 24 Feb 2009 01:58:37 +0000 (18:58 -0700)]
immutable/layout.py: wrap to 80 cols, no functional changes
Brian Warner [Tue, 24 Feb 2009 00:42:27 +0000 (17:42 -0700)]
test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts
Brian Warner [Tue, 24 Feb 2009 00:39:37 +0000 (17:39 -0700)]
test/no_network.py: add a basic stats provider
Brian Warner [Mon, 23 Feb 2009 21:43:12 +0000 (14:43 -0700)]
tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0
Brian Warner [Mon, 23 Feb 2009 21:19:43 +0000 (14:19 -0700)]
test_checker: improve test coverage for checker results
Alberto Berti [Mon, 23 Feb 2009 01:35:02 +0000 (18:35 -0700)]
Fixed tests again so they will pass on windows.
Brian Warner [Mon, 23 Feb 2009 00:34:01 +0000 (17:34 -0700)]
misc/*: remove RuntimeError too
Brian Warner [Mon, 23 Feb 2009 00:31:06 +0000 (17:31 -0700)]
scripts: stop using RuntimeError, for #639
Brian Warner [Mon, 23 Feb 2009 00:30:56 +0000 (17:30 -0700)]
mutable/publish: stop using RuntimeError, for #639
Brian Warner [Mon, 23 Feb 2009 00:28:55 +0000 (17:28 -0700)]
remove more RuntimeError from unit tests, for #639
Brian Warner [Mon, 23 Feb 2009 00:27:22 +0000 (17:27 -0700)]
stop using RuntimeError in unit tests, for #639
Brian Warner [Mon, 23 Feb 2009 00:24:26 +0000 (17:24 -0700)]
ftpd/sftpd: stop using RuntimeError, for #639
Alberto Berti [Sun, 22 Feb 2009 23:43:56 +0000 (16:43 -0700)]
Added tests for the cse when listdir is an iterator
Alberto Berti [Sun, 22 Feb 2009 23:43:11 +0000 (16:43 -0700)]
Fixed tests so that they pass also on buildbots.