Zooko O'Whielacronx [Tue, 7 Apr 2009 18:54:59 +0000 (11:54 -0700)]
wui: edit some of the human-readable parts of the wui such as button labels
(The word "parent" suggests that you can go up a directory hierarchy -- perhaps that word is vestigial.)
Zooko O'Whielacronx [Tue, 7 Apr 2009 18:28:34 +0000 (11:28 -0700)]
wui: fix bug in which empty directory is marked as "unreadable", add test, remove exclamation point
Brian Warner [Tue, 7 Apr 2009 02:11:55 +0000 (19:11 -0700)]
NEWS: add unformatted list of code changes since previous release
Brian Warner [Tue, 7 Apr 2009 02:11:35 +0000 (19:11 -0700)]
docs: modify how-to-relase notes a tiny bit
Brian Warner [Sat, 4 Apr 2009 04:35:27 +0000 (21:35 -0700)]
hashtree.py: another micro-optimization, expand on a comment
Zooko O'Whielacronx [Fri, 3 Apr 2009 23:56:14 +0000 (16:56 -0700)]
hashtree: fix tests of depth_of
Zooko O'Whielacronx [Fri, 3 Apr 2009 23:33:15 +0000 (16:33 -0700)]
trivial: source code metadata
Zooko O'Whielacronx [Fri, 3 Apr 2009 23:33:02 +0000 (16:33 -0700)]
trivial: whitespace
Zooko O'Whielacronx [Fri, 3 Apr 2009 23:30:37 +0000 (16:30 -0700)]
hashtree: a couple of tiny improvements
* calculate depth-first with math instead of traversing the actual tree
* don't mark a node with a red dot if you instead compare it with an extant hash value (tiny optimization)
* edit a comment about checking the root node
Zooko O'Whielacronx [Fri, 3 Apr 2009 22:59:04 +0000 (15:59 -0700)]
leases, time_format: modify time stamping in lease description
* emit lease expiry date in ISO-8601'ish format as well as Brian's format
* rename iso_utc_time_to_localseconds() to iso_utc_time_to_seconds()
* add iso_utc_date()
* simplify the body of iso_utc_time_to_seconds()
Brian Warner [Tue, 31 Mar 2009 20:21:27 +0000 (13:21 -0700)]
hashtree: fix O(N**2) behavior, to improve fatal alacrity problems in a 10GB file (#670). Also improve docstring.
Zooko O'Whielacronx [Wed, 25 Mar 2009 03:54:59 +0000 (20:54 -0700)]
docs: setup: Norm Hardy suggested that it would be easier if users realized that they already had Python (especially true for Mac users)
Brian Warner [Tue, 24 Mar 2009 20:51:37 +0000 (13:51 -0700)]
expirer: include crawler progress in the JSON status output
Brian Warner [Tue, 24 Mar 2009 01:57:52 +0000 (18:57 -0700)]
docs/proposed: new Accounting overview, discuss in #666
Brian Warner [Tue, 24 Mar 2009 01:54:57 +0000 (18:54 -0700)]
docs/proposed: move old accounting docs out of the way
Brian Warner [Mon, 23 Mar 2009 23:08:20 +0000 (16:08 -0700)]
move GC docs out of proposed/, since it's all implemented now. Add reference to configuration.txt . Add expire.*= suggestions to tahoe.cfg .
Brian Warner [Mon, 23 Mar 2009 03:21:28 +0000 (20:21 -0700)]
storage: use constant-time comparison for write-enablers and lease-secrets
Brian Warner [Mon, 23 Mar 2009 03:20:55 +0000 (20:20 -0700)]
hashutil: add constant-time comparison function, to avoid timing attacks when python's short-circuiting data-dependent == operator is used to, say, check a write-enabler
Brian Warner [Fri, 20 Mar 2009 23:58:09 +0000 (16:58 -0700)]
WUI: fix display of empty directories, it threw an exception before
Brian Warner [Fri, 20 Mar 2009 22:44:50 +0000 (15:44 -0700)]
storage webstatus: insert spaces when we're configured to expire multiple sharetypes
Brian Warner [Fri, 20 Mar 2009 19:18:16 +0000 (12:18 -0700)]
expirer: tolerate empty buckets, refactor bucketsize-counting code a bit, don't increment -mutable/-immutable counters unless we actually know the sharetype
Brian Warner [Thu, 19 Mar 2009 18:48:37 +0000 (11:48 -0700)]
storage: improve wording of status message
Brian Warner [Thu, 19 Mar 2009 18:44:43 +0000 (11:44 -0700)]
parse_date: insist that it returns an int
Brian Warner [Thu, 19 Mar 2009 18:07:56 +0000 (11:07 -0700)]
storage status: report expiration-cutoff-date like 19-Mar-2009 (as opposed to the tahoe.cfg input format of 2009-03-19), for redundancy: someone who gets the month and day switched will have a better chance to spot the problem in the storage-status output if it's in a different format
Brian Warner [Thu, 19 Mar 2009 01:00:09 +0000 (18:00 -0700)]
expirer: clean up constructor args, add tahoe.cfg controls, use cutoff_date instead of date_cutoff
Brian Warner [Thu, 19 Mar 2009 00:58:14 +0000 (17:58 -0700)]
util/time_format: new routine to parse dates like 2009-03-18, switch expirer to use it. I'd prefer to use 18-Mar-2009, but it is surprisingly non-trivial to build a parser that will take UTC dates instead of local dates
Brian Warner [Thu, 19 Mar 2009 00:21:38 +0000 (17:21 -0700)]
expirer: change setup, config options, in preparation for adding tahoe.cfg controls
Brian Warner [Wed, 18 Mar 2009 20:25:04 +0000 (13:25 -0700)]
expirer: track mutable-vs-immutable sharecounts and sizes, report them on the web status page for comparison
Brian Warner [Tue, 17 Mar 2009 07:01:17 +0000 (00:01 -0700)]
add utility function to parse durations, for lease-expiration config
Brian Warner [Tue, 17 Mar 2009 06:51:18 +0000 (23:51 -0700)]
expirer: add mode to expire only-mutable or only-immutable shares
Brian Warner [Tue, 17 Mar 2009 05:12:06 +0000 (22:12 -0700)]
tahoe_ls.py: add comment about error cases to improve
Brian Warner [Tue, 17 Mar 2009 05:10:41 +0000 (22:10 -0700)]
GC: add date-cutoff -based expiration, add proposed docs
Brian Warner [Mon, 16 Mar 2009 00:51:34 +0000 (17:51 -0700)]
test_deepcheck: remove the 10s timeout: our dapper buildslave requires 30s, and the reduced timeout was only there because this tests fails by timeout rather than explicitly
Brian Warner [Sun, 15 Mar 2009 23:19:58 +0000 (16:19 -0700)]
tahoe cp -r: add --caps-only flag, to write filecaps into local files instead of actual file contents. Used only for debugging and as a quick tree-comparison tool.
Brian Warner [Fri, 13 Mar 2009 23:31:35 +0000 (16:31 -0700)]
dirnode deep_traverse: insert a turn break (fireEventually) at least once every 100 files, otherwise a CHK followed by more than 158 LITs can overflow the stack, sort of like #237.
Brian Warner [Fri, 13 Mar 2009 08:27:51 +0000 (01:27 -0700)]
consolidate: remove pointless 'else' after for loop
Brian Warner [Fri, 13 Mar 2009 08:24:51 +0000 (01:24 -0700)]
consolidate: add eta, flush stdout
Brian Warner [Fri, 13 Mar 2009 06:54:02 +0000 (23:54 -0700)]
consolidate: tolerate unicode dirnames
Brian Warner [Fri, 13 Mar 2009 06:50:46 +0000 (23:50 -0700)]
dirnode.py: when doing deep-traverse, walk each directory in alphabetical order, to make things like 'manifest' more predictable
Brian Warner [Fri, 13 Mar 2009 05:47:28 +0000 (22:47 -0700)]
consolidator: add progress to scan-old-directory passes
Brian Warner [Fri, 13 Mar 2009 04:22:29 +0000 (21:22 -0700)]
consolidator: fix cycle detection to not trigger on merely shared directories, add snapshot counter to progress
Brian Warner [Fri, 13 Mar 2009 03:48:01 +0000 (20:48 -0700)]
consolidator: re-use more directories, add total directories seen-vs-used counts
Brian Warner [Fri, 13 Mar 2009 01:16:00 +0000 (18:16 -0700)]
tahoe_backup.py: tolerate more time formats
Brian Warner [Thu, 12 Mar 2009 23:29:00 +0000 (16:29 -0700)]
consolidator: add more verbose traversal of directories
Brian Warner [Thu, 12 Mar 2009 23:04:27 +0000 (16:04 -0700)]
consolidate: create multiple numbered backups of the original Archives directory, not just the first time
Brian Warner [Thu, 12 Mar 2009 20:56:06 +0000 (13:56 -0700)]
add 'tahoe debug consolidate' command, to merge directories created by repeated 'tahoe cp -r' or the allmydata win32 backup tool, into the form that would have been created by 'tahoe backup'.
Brian Warner [Thu, 12 Mar 2009 20:53:45 +0000 (13:53 -0700)]
tahoe_cp.py: return 0 for success, instead of None
Zooko O'Whielacronx [Wed, 11 Mar 2009 22:59:02 +0000 (15:59 -0700)]
setup: specify in the debian/control files that tahoe is compatible with Python 2.6
Zooko O'Whielacronx [Wed, 11 Mar 2009 15:29:52 +0000 (08:29 -0700)]
setup: update the debian/copyright text to reflect the current licences
Brian Warner [Mon, 9 Mar 2009 20:42:17 +0000 (13:42 -0700)]
expirer: fix prediction math, thanks to Zandr for the catch
Brian Warner [Mon, 9 Mar 2009 03:55:16 +0000 (20:55 -0700)]
storage.expirer: oops, fix upgrade-handler code
Brian Warner [Mon, 9 Mar 2009 03:42:20 +0000 (20:42 -0700)]
storage.expirer: handle upgrades better
Brian Warner [Mon, 9 Mar 2009 03:38:28 +0000 (20:38 -0700)]
storage.expirer: exercise the last missing line of webstatus code
Brian Warner [Mon, 9 Mar 2009 03:08:40 +0000 (20:08 -0700)]
expirer: tolerate corrupt shares, add them to the state and history for future examination
Brian Warner [Mon, 9 Mar 2009 03:07:32 +0000 (20:07 -0700)]
storage/immutable: raise a specific error upon seeing a bad version number, instead of using assert. Also wrap to 80cols.
Brian Warner [Mon, 9 Mar 2009 03:05:27 +0000 (20:05 -0700)]
storage: add test coverage for BucketReader.__repr__
Brian Warner [Mon, 9 Mar 2009 02:02:01 +0000 (19:02 -0700)]
storage/mutable: raise a specific error upon seeing bad magic, instead of using assert
Zooko O'Whielacronx [Sun, 8 Mar 2009 03:50:39 +0000 (20:50 -0700)]
nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds
A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old.
Brian Warner [Sat, 7 Mar 2009 23:14:42 +0000 (16:14 -0700)]
expirer: make web display a bit more consistent
Brian Warner [Sat, 7 Mar 2009 23:02:43 +0000 (16:02 -0700)]
web/storage.py: tolerate unknown-future displays, I'm not sure why LeaseCrawler.test_unpredictable_future didn't catch this
Brian Warner [Sat, 7 Mar 2009 12:40:51 +0000 (05:40 -0700)]
tahoe_cp.py: improve error reporting slightly: don't json-interpret HTTP errors, pass through tahoe webapi error messages
Brian Warner [Sat, 7 Mar 2009 12:08:15 +0000 (05:08 -0700)]
test_cli.py: assert that 'ls' on an unrecoverable file now gives a better error message
Brian Warner [Sat, 7 Mar 2009 11:57:07 +0000 (04:57 -0700)]
test_web: test behavior of broken-dirnode GET, both html and json
Brian Warner [Sat, 7 Mar 2009 11:56:01 +0000 (04:56 -0700)]
web: when a dirnode can't be read, emit a regular HTML page but with the child-table and upload-forms replaced with an apologetic message. Make sure to include the 'get info' links so the user can do a filecheck
Brian Warner [Sat, 7 Mar 2009 11:54:08 +0000 (04:54 -0700)]
web/common: split out exception-to-explanation+code mapping to a separate humanize_failure() function, so it can be used by other code. Add explanation for mutable UnrecoverableFileError.
Brian Warner [Sat, 7 Mar 2009 10:04:28 +0000 (03:04 -0700)]
test_cli: use explicit (and stable) testdirs, instead of using self.mktemp
Brian Warner [Sat, 7 Mar 2009 09:45:18 +0000 (02:45 -0700)]
test_storage: solaris doesn't appear to give the same block count as other platforms, so don't assert as much about 'diskbytes' recovered
Brian Warner [Sat, 7 Mar 2009 09:44:21 +0000 (02:44 -0700)]
tahoe_cp.py: don't be verbose unless --verbose is passed: default is one line of success
Brian Warner [Sat, 7 Mar 2009 05:57:53 +0000 (22:57 -0700)]
setup.cfg: use allmydata.org as a mirror for some of our necessary dependencies
Brian Warner [Sat, 7 Mar 2009 05:45:17 +0000 (22:45 -0700)]
storage: add a lease-checker-and-expirer crawler, plus web status page.
This walks slowly through all shares, examining their leases, deciding which
are still valid and which have expired. Once enabled, it will then remove the
expired leases, and delete shares which no longer have any valid leases. Note
that there is not yet a tahoe.cfg option to enable lease-deletion: the
current code is read-only. A subsequent patch will add a tahoe.cfg knob to
control this, as well as docs. Some other minor items included in this patch:
tahoe debug dump-share has a new --leases-only flag
storage sharefile/leaseinfo code is cleaned up
storage web status page (/storage) has more info, more tests coverage
space-left measurement on OS-X should be more accurate (it was off by 2048x)
(use stat .f_frsize instead of f_bsize)
Zooko O'Whielacronx [Sat, 7 Mar 2009 05:43:32 +0000 (22:43 -0700)]
setup: comment-out the use of the directory full of dependency packages on the test grid -- the test grid is too unreliable! :-(
Zooko O'Whielacronx [Fri, 6 Mar 2009 20:10:57 +0000 (13:10 -0700)]
setup: create a "make deb-lenny-head" target
I made this patch by copying [
20090305220021-92b7f-
89d987c7d05306b5cb03a64f2956a652c10a7296] and changing the name from "intrepid" to "lenny". I haven't tested it.
Zooko O'Whielacronx [Thu, 5 Mar 2009 23:32:04 +0000 (16:32 -0700)]
trivial: remove unused import detected by pyflakes, and remove trailing whitespace
Zooko O'Whielacronx [Thu, 5 Mar 2009 23:00:21 +0000 (16:00 -0700)]
setup: enable build of .debs for Ubuntu Intrepid, thanks to DarKNesS_WolF
Zooko O'Whielacronx [Sun, 22 Feb 2009 18:57:51 +0000 (11:57 -0700)]
trivial: use more specific function for ascii-encoding storage index
Zooko O'Whielacronx [Sun, 22 Feb 2009 18:57:39 +0000 (11:57 -0700)]
doc: update how_to_make_a_tahoe_release.txt
Brian Warner [Wed, 4 Mar 2009 05:11:46 +0000 (22:11 -0700)]
test_cli: validate non-HTML error response of 'tahoe get' on an unrecoverable file
Brian Warner [Wed, 4 Mar 2009 04:56:30 +0000 (21:56 -0700)]
web: full patch for HTML-vs-plaintext traceback renderings, improve test coverage of exception rendering
Brian Warner [Wed, 4 Mar 2009 04:54:57 +0000 (21:54 -0700)]
web/common.py: use 'Accept:' header to control HTML-vs-text/plain traceback renderings
Brian Warner [Wed, 4 Mar 2009 02:40:59 +0000 (19:40 -0700)]
test/common: rearrange shouldHTTPError
Brian Warner [Wed, 4 Mar 2009 02:40:19 +0000 (19:40 -0700)]
web: move plural() to common.py
Brian Warner [Wed, 4 Mar 2009 02:37:15 +0000 (19:37 -0700)]
add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash
Brian Warner [Tue, 3 Mar 2009 23:56:20 +0000 (16:56 -0700)]
test_web: move shouldHTTPError into common.WebErrorMixin
Brian Warner [Tue, 3 Mar 2009 23:46:04 +0000 (16:46 -0700)]
test_web: clean up shouldHTTPError methods
Brian Warner [Fri, 27 Feb 2009 08:17:24 +0000 (01:17 -0700)]
immutable checker add-lease: catch remote IndexError here too
Brian Warner [Fri, 27 Feb 2009 08:04:26 +0000 (01:04 -0700)]
servermap add-lease: fix the code that's supposed to catch remote IndexErrors, I forgot that they present as ServerFailures instead. This should stop the deluge of Incidents that occur when you do add-lease against 1.3.0 servers
Brian Warner [Fri, 27 Feb 2009 07:59:57 +0000 (00:59 -0700)]
rrefutil: add check_remote utility function
Brian Warner [Fri, 27 Feb 2009 07:55:24 +0000 (00:55 -0700)]
rrefutil: add trap_remote utility and friends
Brian Warner [Fri, 27 Feb 2009 02:58:38 +0000 (19:58 -0700)]
web: improve layout of storage status with a table
Brian Warner [Fri, 27 Feb 2009 02:42:48 +0000 (19:42 -0700)]
crawler: add ETA to get_progress()
Alberto Berti [Thu, 26 Feb 2009 16:02:37 +0000 (09:02 -0700)]
Fix for bug #645, correct path handling logic so that it works from sshfs
Zooko O'Whielacronx [Thu, 26 Feb 2009 04:09:04 +0000 (21:09 -0700)]
wui: s/Provisioning/Reliability/ ; suggested by Terrell
Brian Warner [Thu, 26 Feb 2009 02:52:45 +0000 (19:52 -0700)]
startstop_node.py: when restarting, tolerate a stale pid too
Brian Warner [Wed, 25 Feb 2009 08:46:21 +0000 (01:46 -0700)]
web: fix the ERROR: line to work the same in python2.4 and 2.5
Brian Warner [Wed, 25 Feb 2009 07:01:28 +0000 (00:01 -0700)]
test_cli/test_web: fix spurious test failure on solaris (maybe python2.4?) due to variations in the way that exceptions are stringified
Brian Warner [Wed, 25 Feb 2009 06:44:15 +0000 (23:44 -0700)]
CLI: modify 'tahoe manifest' and 'tahoe deep-check' to report ERROR: properly. For #590.
Brian Warner [Wed, 25 Feb 2009 06:13:35 +0000 (23:13 -0700)]
webapi: modify streaming deep-manifest/deep-checker to emit an ERROR: line if they encounter an unrecoverable+untraversable directory. For #590.
Brian Warner [Wed, 25 Feb 2009 05:21:36 +0000 (22:21 -0700)]
scripts/common: fix alias handling on windows again, emit slightly nicer error message in response to an unknown alias
Brian Warner [Wed, 25 Feb 2009 04:42:13 +0000 (21:42 -0700)]
#165: make 'tahoe restart --force' the default behavior: warn but do not stop if restart is used on something that wasn't a running node, and always try to start it afterwards. This is particularly important for #315 (restart -m), because otherwise a single not-already-running node will prevent all nodes from being restarted, resulting in longer downtime than necessary
Brian Warner [Wed, 25 Feb 2009 04:04:57 +0000 (21:04 -0700)]
test_deepcheck: switch deep-check tests to use no-network too. This cuts the runtime down by about 50%