Brian Warner [Mon, 9 Mar 2009 03:38:28 +0000 (20:38 -0700)]
storage.expirer: exercise the last missing line of webstatus code
Brian Warner [Mon, 9 Mar 2009 03:08:40 +0000 (20:08 -0700)]
expirer: tolerate corrupt shares, add them to the state and history for future examination
Brian Warner [Mon, 9 Mar 2009 03:07:32 +0000 (20:07 -0700)]
storage/immutable: raise a specific error upon seeing a bad version number, instead of using assert. Also wrap to 80cols.
Brian Warner [Mon, 9 Mar 2009 03:05:27 +0000 (20:05 -0700)]
storage: add test coverage for BucketReader.__repr__
Brian Warner [Mon, 9 Mar 2009 02:02:01 +0000 (19:02 -0700)]
storage/mutable: raise a specific error upon seeing bad magic, instead of using assert
Zooko O'Whielacronx [Sun, 8 Mar 2009 03:50:39 +0000 (20:50 -0700)]
nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds
A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old.
Brian Warner [Sat, 7 Mar 2009 23:14:42 +0000 (16:14 -0700)]
expirer: make web display a bit more consistent
Brian Warner [Sat, 7 Mar 2009 23:02:43 +0000 (16:02 -0700)]
web/storage.py: tolerate unknown-future displays, I'm not sure why LeaseCrawler.test_unpredictable_future didn't catch this
Brian Warner [Sat, 7 Mar 2009 12:40:51 +0000 (05:40 -0700)]
tahoe_cp.py: improve error reporting slightly: don't json-interpret HTTP errors, pass through tahoe webapi error messages
Brian Warner [Sat, 7 Mar 2009 12:08:15 +0000 (05:08 -0700)]
test_cli.py: assert that 'ls' on an unrecoverable file now gives a better error message
Brian Warner [Sat, 7 Mar 2009 11:57:07 +0000 (04:57 -0700)]
test_web: test behavior of broken-dirnode GET, both html and json
Brian Warner [Sat, 7 Mar 2009 11:56:01 +0000 (04:56 -0700)]
web: when a dirnode can't be read, emit a regular HTML page but with the child-table and upload-forms replaced with an apologetic message. Make sure to include the 'get info' links so the user can do a filecheck
Brian Warner [Sat, 7 Mar 2009 11:54:08 +0000 (04:54 -0700)]
web/common: split out exception-to-explanation+code mapping to a separate humanize_failure() function, so it can be used by other code. Add explanation for mutable UnrecoverableFileError.
Brian Warner [Sat, 7 Mar 2009 10:04:28 +0000 (03:04 -0700)]
test_cli: use explicit (and stable) testdirs, instead of using self.mktemp
Brian Warner [Sat, 7 Mar 2009 09:45:18 +0000 (02:45 -0700)]
test_storage: solaris doesn't appear to give the same block count as other platforms, so don't assert as much about 'diskbytes' recovered
Brian Warner [Sat, 7 Mar 2009 09:44:21 +0000 (02:44 -0700)]
tahoe_cp.py: don't be verbose unless --verbose is passed: default is one line of success
Brian Warner [Sat, 7 Mar 2009 05:57:53 +0000 (22:57 -0700)]
setup.cfg: use allmydata.org as a mirror for some of our necessary dependencies
Brian Warner [Sat, 7 Mar 2009 05:45:17 +0000 (22:45 -0700)]
storage: add a lease-checker-and-expirer crawler, plus web status page.
This walks slowly through all shares, examining their leases, deciding which
are still valid and which have expired. Once enabled, it will then remove the
expired leases, and delete shares which no longer have any valid leases. Note
that there is not yet a tahoe.cfg option to enable lease-deletion: the
current code is read-only. A subsequent patch will add a tahoe.cfg knob to
control this, as well as docs. Some other minor items included in this patch:
tahoe debug dump-share has a new --leases-only flag
storage sharefile/leaseinfo code is cleaned up
storage web status page (/storage) has more info, more tests coverage
space-left measurement on OS-X should be more accurate (it was off by 2048x)
(use stat .f_frsize instead of f_bsize)
Zooko O'Whielacronx [Sat, 7 Mar 2009 05:43:32 +0000 (22:43 -0700)]
setup: comment-out the use of the directory full of dependency packages on the test grid -- the test grid is too unreliable! :-(
Zooko O'Whielacronx [Fri, 6 Mar 2009 20:10:57 +0000 (13:10 -0700)]
setup: create a "make deb-lenny-head" target
I made this patch by copying [
20090305220021-92b7f-
89d987c7d05306b5cb03a64f2956a652c10a7296] and changing the name from "intrepid" to "lenny". I haven't tested it.
Zooko O'Whielacronx [Thu, 5 Mar 2009 23:32:04 +0000 (16:32 -0700)]
trivial: remove unused import detected by pyflakes, and remove trailing whitespace
Zooko O'Whielacronx [Thu, 5 Mar 2009 23:00:21 +0000 (16:00 -0700)]
setup: enable build of .debs for Ubuntu Intrepid, thanks to DarKNesS_WolF
Zooko O'Whielacronx [Sun, 22 Feb 2009 18:57:51 +0000 (11:57 -0700)]
trivial: use more specific function for ascii-encoding storage index
Zooko O'Whielacronx [Sun, 22 Feb 2009 18:57:39 +0000 (11:57 -0700)]
doc: update how_to_make_a_tahoe_release.txt
Brian Warner [Wed, 4 Mar 2009 05:11:46 +0000 (22:11 -0700)]
test_cli: validate non-HTML error response of 'tahoe get' on an unrecoverable file
Brian Warner [Wed, 4 Mar 2009 04:56:30 +0000 (21:56 -0700)]
web: full patch for HTML-vs-plaintext traceback renderings, improve test coverage of exception rendering
Brian Warner [Wed, 4 Mar 2009 04:54:57 +0000 (21:54 -0700)]
web/common.py: use 'Accept:' header to control HTML-vs-text/plain traceback renderings
Brian Warner [Wed, 4 Mar 2009 02:40:59 +0000 (19:40 -0700)]
test/common: rearrange shouldHTTPError
Brian Warner [Wed, 4 Mar 2009 02:40:19 +0000 (19:40 -0700)]
web: move plural() to common.py
Brian Warner [Wed, 4 Mar 2009 02:37:15 +0000 (19:37 -0700)]
add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash
Brian Warner [Tue, 3 Mar 2009 23:56:20 +0000 (16:56 -0700)]
test_web: move shouldHTTPError into common.WebErrorMixin
Brian Warner [Tue, 3 Mar 2009 23:46:04 +0000 (16:46 -0700)]
test_web: clean up shouldHTTPError methods
Brian Warner [Fri, 27 Feb 2009 08:17:24 +0000 (01:17 -0700)]
immutable checker add-lease: catch remote IndexError here too
Brian Warner [Fri, 27 Feb 2009 08:04:26 +0000 (01:04 -0700)]
servermap add-lease: fix the code that's supposed to catch remote IndexErrors, I forgot that they present as ServerFailures instead. This should stop the deluge of Incidents that occur when you do add-lease against 1.3.0 servers
Brian Warner [Fri, 27 Feb 2009 07:59:57 +0000 (00:59 -0700)]
rrefutil: add check_remote utility function
Brian Warner [Fri, 27 Feb 2009 07:55:24 +0000 (00:55 -0700)]
rrefutil: add trap_remote utility and friends
Brian Warner [Fri, 27 Feb 2009 02:58:38 +0000 (19:58 -0700)]
web: improve layout of storage status with a table
Brian Warner [Fri, 27 Feb 2009 02:42:48 +0000 (19:42 -0700)]
crawler: add ETA to get_progress()
Alberto Berti [Thu, 26 Feb 2009 16:02:37 +0000 (09:02 -0700)]
Fix for bug #645, correct path handling logic so that it works from sshfs
Zooko O'Whielacronx [Thu, 26 Feb 2009 04:09:04 +0000 (21:09 -0700)]
wui: s/Provisioning/Reliability/ ; suggested by Terrell
Brian Warner [Thu, 26 Feb 2009 02:52:45 +0000 (19:52 -0700)]
startstop_node.py: when restarting, tolerate a stale pid too
Brian Warner [Wed, 25 Feb 2009 08:46:21 +0000 (01:46 -0700)]
web: fix the ERROR: line to work the same in python2.4 and 2.5
Brian Warner [Wed, 25 Feb 2009 07:01:28 +0000 (00:01 -0700)]
test_cli/test_web: fix spurious test failure on solaris (maybe python2.4?) due to variations in the way that exceptions are stringified
Brian Warner [Wed, 25 Feb 2009 06:44:15 +0000 (23:44 -0700)]
CLI: modify 'tahoe manifest' and 'tahoe deep-check' to report ERROR: properly. For #590.
Brian Warner [Wed, 25 Feb 2009 06:13:35 +0000 (23:13 -0700)]
webapi: modify streaming deep-manifest/deep-checker to emit an ERROR: line if they encounter an unrecoverable+untraversable directory. For #590.
Brian Warner [Wed, 25 Feb 2009 05:21:36 +0000 (22:21 -0700)]
scripts/common: fix alias handling on windows again, emit slightly nicer error message in response to an unknown alias
Brian Warner [Wed, 25 Feb 2009 04:42:13 +0000 (21:42 -0700)]
#165: make 'tahoe restart --force' the default behavior: warn but do not stop if restart is used on something that wasn't a running node, and always try to start it afterwards. This is particularly important for #315 (restart -m), because otherwise a single not-already-running node will prevent all nodes from being restarted, resulting in longer downtime than necessary
Brian Warner [Wed, 25 Feb 2009 04:04:57 +0000 (21:04 -0700)]
test_deepcheck: switch deep-check tests to use no-network too. This cuts the runtime down by about 50%
Brian Warner [Wed, 25 Feb 2009 03:00:10 +0000 (20:00 -0700)]
test_deepcheck: convert MutableChecker to no-network GridTest
Brian Warner [Wed, 25 Feb 2009 01:33:00 +0000 (18:33 -0700)]
tests/no_network: move GET into the GridTestMixin class
Brian Warner [Wed, 25 Feb 2009 01:04:59 +0000 (18:04 -0700)]
common_web.py: oops, add .fields to the other FakeRequest
Brian Warner [Wed, 25 Feb 2009 00:56:20 +0000 (17:56 -0700)]
test_cli: exercise the recent tolerate-'c:\dir\file.txt' fix in scripts/common, recorded in a separate match to make it easier to merge the fix to prod
Brian Warner [Wed, 25 Feb 2009 00:55:22 +0000 (17:55 -0700)]
scripts/common: on windows, tolerate paths like 'c:\dir\file.txt', by treating single-letter aliases on windows/cygwin as non-aliases
Brian Warner [Wed, 25 Feb 2009 00:20:50 +0000 (17:20 -0700)]
test/common_web.py: add a .fields attribute to our FakeRequest, since we support versions of Nevow that are old enough to not do it themselves
Alberto Berti [Tue, 24 Feb 2009 23:36:34 +0000 (16:36 -0700)]
Two small fixes on documentation for cli backup command.
Brian Warner [Tue, 24 Feb 2009 22:40:17 +0000 (15:40 -0700)]
test_web: add (disabled) test to see what happens when deep-check encounters an unrecoverable directory. We still need code changes to improve this behavior.
Alberto Berti [Tue, 24 Feb 2009 18:14:25 +0000 (11:14 -0700)]
Add elapsed timestamp to cli backup command final summary.
Alberto Berti [Tue, 24 Feb 2009 16:30:49 +0000 (09:30 -0700)]
Added documentation for '--exclude' and friends cli backup command.
Brian Warner [Tue, 24 Feb 2009 05:15:06 +0000 (22:15 -0700)]
test_repairer: change to use faster no_network.GridTestMixin, split Verifier tests into separate cases, refactor judgement funcs into shared methods
Brian Warner [Tue, 24 Feb 2009 05:14:05 +0000 (22:14 -0700)]
immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check
instead of examining the value returned by f.trap, because the latter appears
to squash exception types down into their base classes (i.e. since
ShareVersionIncompatible is a subclass of LayoutInvalid,
f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid).
All this resulted in 'incompatible' shares being misclassified as 'corrupt'.
Brian Warner [Tue, 24 Feb 2009 01:58:37 +0000 (18:58 -0700)]
immutable/layout.py: wrap to 80 cols, no functional changes
Brian Warner [Tue, 24 Feb 2009 00:42:27 +0000 (17:42 -0700)]
test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts
Brian Warner [Tue, 24 Feb 2009 00:39:37 +0000 (17:39 -0700)]
test/no_network.py: add a basic stats provider
Brian Warner [Mon, 23 Feb 2009 21:43:12 +0000 (14:43 -0700)]
tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0
Brian Warner [Mon, 23 Feb 2009 21:19:43 +0000 (14:19 -0700)]
test_checker: improve test coverage for checker results
Alberto Berti [Mon, 23 Feb 2009 01:35:02 +0000 (18:35 -0700)]
Fixed tests again so they will pass on windows.
Brian Warner [Mon, 23 Feb 2009 00:34:01 +0000 (17:34 -0700)]
misc/*: remove RuntimeError too
Brian Warner [Mon, 23 Feb 2009 00:31:06 +0000 (17:31 -0700)]
scripts: stop using RuntimeError, for #639
Brian Warner [Mon, 23 Feb 2009 00:30:56 +0000 (17:30 -0700)]
mutable/publish: stop using RuntimeError, for #639
Brian Warner [Mon, 23 Feb 2009 00:28:55 +0000 (17:28 -0700)]
remove more RuntimeError from unit tests, for #639
Brian Warner [Mon, 23 Feb 2009 00:27:22 +0000 (17:27 -0700)]
stop using RuntimeError in unit tests, for #639
Brian Warner [Mon, 23 Feb 2009 00:24:26 +0000 (17:24 -0700)]
ftpd/sftpd: stop using RuntimeError, for #639
Alberto Berti [Sun, 22 Feb 2009 23:43:56 +0000 (16:43 -0700)]
Added tests for the cse when listdir is an iterator
Alberto Berti [Sun, 22 Feb 2009 23:43:11 +0000 (16:43 -0700)]
Fixed tests so that they pass also on buildbots.
Alberto Berti [Sun, 22 Feb 2009 23:42:14 +0000 (16:42 -0700)]
Use failUnlessEqual instead of failUnless(a == b)
Alberto Berti [Sun, 22 Feb 2009 23:40:49 +0000 (16:40 -0700)]
Better implementation of filtering algorithm.
Alberto Berti [Sun, 22 Feb 2009 23:39:46 +0000 (16:39 -0700)]
Removed '.hgrags' from vcs excludes
Zooko O'Whielacronx [Sun, 22 Feb 2009 20:33:14 +0000 (13:33 -0700)]
docs: CREDITS to Alberto Berti
Alberto Berti [Sun, 22 Feb 2009 17:51:06 +0000 (10:51 -0700)]
Added tests for the --exclude* options of backup command.
Alberto Berti [Sun, 22 Feb 2009 18:08:29 +0000 (11:08 -0700)]
Added --exclude, --exclude-from and --exclude-vcs options to backup command.
It is still impossible to specify absolute exclusion path, only
relative. I must check with tar or rsync how they allow them to be
specified.
Alberto Berti [Sun, 22 Feb 2009 18:02:52 +0000 (11:02 -0700)]
Raise a more explanatory exception for errors encountered during backup processing.
Alberto Berti [Sun, 22 Feb 2009 17:37:32 +0000 (10:37 -0700)]
Added tests for the fixed alias related command's synopsis
Alberto Berti [Sat, 21 Feb 2009 01:31:06 +0000 (18:31 -0700)]
Add missing synopsis and descriptions for alias commands.
Brian Warner [Sun, 22 Feb 2009 06:40:54 +0000 (23:40 -0700)]
docs: move many specification-like documents into specifications/
Brian Warner [Sun, 22 Feb 2009 01:01:16 +0000 (18:01 -0700)]
test_web.py: increase test coverage of web.status.plural()
Brian Warner [Sat, 21 Feb 2009 21:56:49 +0000 (14:56 -0700)]
crawler: fix performance problems: only save state once per timeslice (not after every bucket), don't start the crawler until 5 minutes after node startup
Brian Warner [Sat, 21 Feb 2009 07:17:10 +0000 (00:17 -0700)]
test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path
Brian Warner [Sat, 21 Feb 2009 07:15:33 +0000 (00:15 -0700)]
crawler: tolerate low-resolution system clocks (i.e. windows)
Brian Warner [Sat, 21 Feb 2009 04:58:31 +0000 (21:58 -0700)]
BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise
Brian Warner [Sat, 21 Feb 2009 04:57:55 +0000 (21:57 -0700)]
test_storage: startService the server, as is now the standard practice
Brian Warner [Sat, 21 Feb 2009 04:57:20 +0000 (21:57 -0700)]
crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early
Brian Warner [Sat, 21 Feb 2009 04:46:06 +0000 (21:46 -0700)]
BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate
Brian Warner [Sat, 21 Feb 2009 04:28:56 +0000 (21:28 -0700)]
storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better
Brian Warner [Sat, 21 Feb 2009 04:04:08 +0000 (21:04 -0700)]
storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page
Brian Warner [Sat, 21 Feb 2009 04:03:09 +0000 (21:03 -0700)]
storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py
Brian Warner [Sat, 21 Feb 2009 01:27:43 +0000 (18:27 -0700)]
crawler: add get_progress, clean up get_state
Brian Warner [Fri, 20 Feb 2009 23:03:53 +0000 (16:03 -0700)]
web/storage: make sure we can handle platforms without os.statvfs too
Brian Warner [Fri, 20 Feb 2009 22:19:11 +0000 (15:19 -0700)]
crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters
Brian Warner [Fri, 20 Feb 2009 21:29:26 +0000 (14:29 -0700)]
web: add Storage status page, improve tests
Brian Warner [Fri, 20 Feb 2009 21:29:20 +0000 (14:29 -0700)]
storage: include reserved_space in stats