Brian Warner [Wed, 13 Feb 2008 02:31:23 +0000 (19:31 -0700)]
webish: censor all caps before logging the HTTP request, to preserve user privacy
Brian Warner [Wed, 13 Feb 2008 02:01:03 +0000 (19:01 -0700)]
download status: refactor into a separate object, so we don't need to keep the Download itself around for a long time
Brian Warner [Wed, 13 Feb 2008 01:59:54 +0000 (18:59 -0700)]
introducer: remove encoding-parameter config, for now
Zooko O'Whielacronx [Wed, 13 Feb 2008 03:48:37 +0000 (20:48 -0700)]
use base62 encoding for storage indexes, on disk and in verifier caps, and in logging and diagnostic tools
base62 encoding fits more information into alphanumeric chars while avoiding the troublesome non-alphanumeric chars of base64 encoding. In particular, this allows us to work around the ext3 "32,000 entries in a directory" limit while retaining the convenient property that the intermediate directory names are leading prefixes of the storage index file names.
Brian Warner [Tue, 12 Feb 2008 22:40:05 +0000 (15:40 -0700)]
webish: add /status page to display current uploads/downloads
Brian Warner [Tue, 12 Feb 2008 22:39:45 +0000 (15:39 -0700)]
make current upload/download status objects available from the client
Brian Warner [Tue, 12 Feb 2008 22:38:39 +0000 (15:38 -0700)]
add download-status objects, to track download progress
Brian Warner [Tue, 12 Feb 2008 22:36:05 +0000 (15:36 -0700)]
add upload-status objects, to track upload progress
Brian Warner [Tue, 12 Feb 2008 02:53:36 +0000 (19:53 -0700)]
misc/find-share-anomalies.py: tool to analyze 'tahoe catalog-shares' output and look for problems
Brian Warner [Tue, 12 Feb 2008 02:14:10 +0000 (19:14 -0700)]
webish: add edge metadata to t=json output, including timestamps
Brian Warner [Tue, 12 Feb 2008 01:44:54 +0000 (18:44 -0700)]
catalog-shares: add expiration time to output
Brian Warner [Tue, 12 Feb 2008 01:17:01 +0000 (18:17 -0700)]
add 'tahoe catalog-shares' tool, to make a one-line summary of each share file. This can help do cross-server correlation of sharefiles, looking for anomalies
Brian Warner [Tue, 12 Feb 2008 00:37:51 +0000 (17:37 -0700)]
test_dirnode.py: improve error messages in case timestamp tests fail
Brian Warner [Tue, 12 Feb 2008 00:35:17 +0000 (17:35 -0700)]
test_dirnode.py: simplejson-1.7.1 incorrectly rounds floats to two decimal places. Don't let this bug flunk the timestamp test.
Brian Warner [Mon, 11 Feb 2008 22:26:58 +0000 (15:26 -0700)]
test_system.py: refactor bounce_client, probably make it stop failing on cygwin
Brian Warner [Mon, 11 Feb 2008 22:13:18 +0000 (15:13 -0700)]
webish: display timestamps
Brian Warner [Mon, 11 Feb 2008 22:12:55 +0000 (15:12 -0700)]
test_dirnode.py: assert that we update mtime and preserve ctime
Brian Warner [Mon, 11 Feb 2008 21:53:28 +0000 (14:53 -0700)]
dirnode.py: add metadata= to add_file(), add tests
Brian Warner [Mon, 11 Feb 2008 21:13:07 +0000 (14:13 -0700)]
test_dirnode.py: add diag output to test-ctime/mtime tests
Brian Warner [Sat, 9 Feb 2008 01:43:47 +0000 (18:43 -0700)]
dirnode: add ctime/mtime to metadata, update metadata-modifying APIs. Needs more testing and sanity checking.
Brian Warner [Fri, 8 Feb 2008 04:50:04 +0000 (21:50 -0700)]
docs/install-details.html: wrap to 80 cols, no content changes
Brian Warner [Fri, 8 Feb 2008 03:15:37 +0000 (20:15 -0700)]
test_system: remove the hackish debug_interrupt= attribute magic used to exercise interrupted-upload resumption, instead just make the Uploadable bounce the helper halfway through the upload
Brian Warner [Fri, 8 Feb 2008 03:14:05 +0000 (20:14 -0700)]
dirnodes.txt: update docs a bit, we don't yet do choose-latest-version, just choose-any-version
Brian Warner [Fri, 8 Feb 2008 03:10:28 +0000 (20:10 -0700)]
webapi.txt: document POST /uri?t=mkdir
Brian Warner [Fri, 8 Feb 2008 00:27:30 +0000 (17:27 -0700)]
test_system.py: remove that ugly debug_stash_RemoteencryptedUploadable hack, now that UploadResults give us a better approach
Brian Warner [Fri, 8 Feb 2008 00:26:59 +0000 (17:26 -0700)]
upload: if we lose the helper, go back to doing direct-to-server uploads instead of causing all subsequent uploads to fail
Brian Warner [Fri, 8 Feb 2008 00:10:13 +0000 (17:10 -0700)]
upload.EncryptAnUploadable: use 50KiB read chunks (instead of 50KB), to match the chunksize requested by the upload helper
Brian Warner [Thu, 7 Feb 2008 03:04:31 +0000 (20:04 -0700)]
test_upload.py: hush pyflakes
Brian Warner [Thu, 7 Feb 2008 03:03:35 +0000 (20:03 -0700)]
test_upload.py: add test to exercise CHK hashing variations
Brian Warner [Thu, 7 Feb 2008 02:58:53 +0000 (19:58 -0700)]
test_cli: oops, need to update this when the CHK hash changes
Brian Warner [Thu, 7 Feb 2008 02:50:47 +0000 (19:50 -0700)]
change encryption-key hash to include encoding parameters. This is a minor compatibility break: CHK files encoded (with convergence) before and after this will have different keys and ciphertexts. Also switched to SHA-256d for both the data-to-key hash and the key-to-storageindex hash
Brian Warner [Thu, 7 Feb 2008 02:36:43 +0000 (19:36 -0700)]
hashutil: add tagged_hash_256d and tagged_hasher_256d
Brian Warner [Thu, 7 Feb 2008 01:39:03 +0000 (18:39 -0700)]
upload: rework passing of default encoding parameters: move more responsibility into BaseUploadable
Brian Warner [Thu, 7 Feb 2008 00:51:11 +0000 (17:51 -0700)]
offloaded: oops, need more tricks to make the unit tests pass
Brian Warner [Thu, 7 Feb 2008 00:30:58 +0000 (17:30 -0700)]
helper: return full uri-extension data to the client, so it can compare encoding parameters
Brian Warner [Wed, 6 Feb 2008 23:41:26 +0000 (16:41 -0700)]
interfaces: remove spurious line that counted against the figleaf coverage
Brian Warner [Wed, 6 Feb 2008 23:41:04 +0000 (16:41 -0700)]
test_util: add full coverage for allmydata.util.deferredutil
robk-tahoe [Wed, 6 Feb 2008 21:12:49 +0000 (14:12 -0700)]
windows: track uninstalls
the confwiz and the native_client backend both gained hooks to track
uninstall events. however that somehow didn't make it to the uninstaller :-)
Brian Warner [Wed, 6 Feb 2008 21:15:33 +0000 (14:15 -0700)]
tahoe dump-share: remove --filename option, just take it from argv
Brian Warner [Wed, 6 Feb 2008 21:05:11 +0000 (14:05 -0700)]
test_system: deferred-handling logic in test_upload_and_download was broken
Brian Warner [Wed, 6 Feb 2008 20:37:43 +0000 (13:37 -0700)]
dump-share: tweak formatting a little bit, to make dumping multiple shares in a row easier to read
Brian Warner [Wed, 6 Feb 2008 20:19:51 +0000 (13:19 -0700)]
add 'tahoe find-shares' command, to locate share files on a local node's disk
Brian Warner [Wed, 6 Feb 2008 19:48:19 +0000 (12:48 -0700)]
dump-cap: include UEB_hash in output
Brian Warner [Wed, 6 Feb 2008 09:50:34 +0000 (02:50 -0700)]
download.py: make logging safe in ValidatedBucket
Brian Warner [Wed, 6 Feb 2008 09:39:01 +0000 (02:39 -0700)]
webish: more upload stats: total encode-and-push rate, already-in-grid existence check time
Brian Warner [Wed, 6 Feb 2008 09:38:16 +0000 (02:38 -0700)]
webish: add when_done= to POST /uri?t=upload . I did not add a 'recent uploads' section to the welcome page, but I think the new upload-results page provides the desired data
Brian Warner [Wed, 6 Feb 2008 09:12:35 +0000 (02:12 -0700)]
UploadResults: add more helper timing stats (ciphertext fetch times)
Brian Warner [Wed, 6 Feb 2008 08:52:25 +0000 (01:52 -0700)]
add upload-results timing info for helper uploads. This changes the Helper protocol, and introduces a compatibility break
Brian Warner [Wed, 6 Feb 2008 07:41:51 +0000 (00:41 -0700)]
add upload timings and rates to the POST /uri?t=upload results page
Brian Warner [Wed, 6 Feb 2008 06:01:37 +0000 (23:01 -0700)]
webish: make POST /uri?t=upload deposit you on an 'Upload Results' page
Brian Warner [Wed, 6 Feb 2008 05:38:20 +0000 (22:38 -0700)]
webish.py: refactor /uri handlers, one rend.Page class per operation
Brian Warner [Wed, 6 Feb 2008 05:37:51 +0000 (22:37 -0700)]
test_web.py: hush pyflakes
Brian Warner [Wed, 6 Feb 2008 05:18:02 +0000 (22:18 -0700)]
webish: add PUT /uri?mutable=true
Brian Warner [Wed, 6 Feb 2008 05:10:22 +0000 (22:10 -0700)]
webish: add POST /uri?t=upload&mutable=true
Brian Warner [Wed, 6 Feb 2008 04:44:40 +0000 (21:44 -0700)]
webish: add upload/view-uri forms (not associated with any particular directory) to the welcome page. Document POST /uri?t=upload .
Brian Warner [Wed, 6 Feb 2008 04:01:38 +0000 (21:01 -0700)]
upload: return an UploadResults instance (with .uri) instead of just a URI
Brian Warner [Wed, 6 Feb 2008 03:29:39 +0000 (20:29 -0700)]
webish: add storage-consumed estimate on welcome page
Brian Warner [Wed, 6 Feb 2008 03:28:59 +0000 (20:28 -0700)]
storage: change service name from 'storageserver' to 'storage'
Brian Warner [Wed, 6 Feb 2008 03:08:49 +0000 (20:08 -0700)]
webish: display tahoe import path on the welcome page, to help figure out where the code is coming from
Brian Warner [Wed, 6 Feb 2008 02:58:38 +0000 (19:58 -0700)]
refactor node startup, remove tub_ready()
Zooko O'Whielacronx [Wed, 6 Feb 2008 19:35:29 +0000 (12:35 -0700)]
setup: fix the md5sum of the bundled setuptools egg
Brian Warner [Wed, 6 Feb 2008 01:48:26 +0000 (18:48 -0700)]
docs/testgrid/introducer.furl: update to new introducer
Brian Warner [Wed, 6 Feb 2008 01:29:28 +0000 (18:29 -0700)]
webish: condense display of nickname a little bit
Brian Warner [Wed, 6 Feb 2008 01:26:05 +0000 (18:26 -0700)]
webish: show nickname too
Brian Warner [Wed, 6 Feb 2008 00:32:27 +0000 (17:32 -0700)]
webish: add extra introducer data (version, timestamps) to Welcome page
Brian Warner [Tue, 5 Feb 2008 23:37:58 +0000 (16:37 -0700)]
test_introducer.py: increase timeouts on poll() calls
Brian Warner [Tue, 5 Feb 2008 22:27:41 +0000 (15:27 -0700)]
bundle foolscap-0.2.4 instead of 0.2.3
Brian Warner [Tue, 5 Feb 2008 22:27:14 +0000 (15:27 -0700)]
bump foolscap dependency to 0.2.4, since we now use log.err
Brian Warner [Tue, 5 Feb 2008 21:35:12 +0000 (14:35 -0700)]
docs/configuration.txt: wrap to 80 cols
Brian Warner [Tue, 5 Feb 2008 21:33:29 +0000 (14:33 -0700)]
docs/configuration.txt: document nickname, no_storage, readonly_storage
Brian Warner [Tue, 5 Feb 2008 21:16:01 +0000 (14:16 -0700)]
introducer: remove remaining bits of 'push-to-myself' flags. The uploading/downloading node is no longer special.
Brian Warner [Tue, 5 Feb 2008 21:15:49 +0000 (14:15 -0700)]
introducer: remove PeerCountObserver, tests are managing with purely poll-for-connected approachers
Brian Warner [Tue, 5 Feb 2008 20:05:13 +0000 (13:05 -0700)]
big introducer refactoring: separate publish+subscribe. Addresses #271.
Brian Warner [Tue, 5 Feb 2008 03:35:07 +0000 (20:35 -0700)]
PollMixin: add timeout= argument, rewrite to avoid tail-recursion problems
robk-tahoe [Tue, 5 Feb 2008 01:12:50 +0000 (18:12 -0700)]
windows: include latest windown build, winfuse and tray.exe
robk-tahoe [Sat, 2 Feb 2008 01:57:31 +0000 (18:57 -0700)]
stats: fix service issues
having moved inititalisation into startService to handle tub init cleanly,
I neglected the up-call to startService, which wound up not starting the
load_monitor.
also I changed the 'running' attribute to 'started' since 'running' is
the name used internally by MultiService itself.
robk-tahoe [Sat, 2 Feb 2008 01:44:57 +0000 (18:44 -0700)]
munin stats: don't suppress series with no current data
having changed tahoe-stats to not report data series if there was no recent
data recorded for a node, I wound up making it hide the data series. this
change causes it to report all data series for which stats exist in the
'config' phase, so that they show up, but only report actual data if the
stats are recent, so that they show up as missing if the node is not
reporting stats currently
nejucomo [Wed, 30 Jan 2008 10:14:48 +0000 (03:14 -0700)]
tahoe_fuse: system test: Verify file contents can be properly read.
nejucomo [Wed, 30 Jan 2008 09:59:43 +0000 (02:59 -0700)]
tahoe_fuse: system test: Populate a testdir with files and empty children directories, then test the fuse interface for proper listings and size metadata.
nejucomo [Wed, 30 Jan 2008 09:57:54 +0000 (02:57 -0700)]
tahoe_fuse: system test: Create a separate directory for each test and pass the cap and local path to each test. Add two basic sanity tests for empty directories.
nejucomo [Wed, 30 Jan 2008 09:56:25 +0000 (02:56 -0700)]
tahoe_fuse: system test: Replace repeated attempts at webapi calls with single calls, abstract webapi calls into a single function.
nejucomo [Wed, 30 Jan 2008 09:55:53 +0000 (02:55 -0700)]
tahoe_fuse: system test: Remove some needless comments.
nejucomo [Wed, 30 Jan 2008 09:46:24 +0000 (02:46 -0700)]
tahoe_fuse: system test: Move test summary to end of output.
nejucomo [Wed, 30 Jan 2008 09:45:54 +0000 (02:45 -0700)]
tahoe_fuse: system tests: Update comments.
nejucomo [Wed, 30 Jan 2008 08:10:53 +0000 (01:10 -0700)]
tahoe_fuse: system test: Make output checking into non-fatal warnings, and make patterns looser.
nejucomo [Wed, 30 Jan 2008 08:27:42 +0000 (01:27 -0700)]
docs: webapi: Add concise shorthand for options, input, output, and statuses to the operation descriptions...
I'm not convinced if this is the best way to do this, but I find it
handy to have a conscise "quick reference" for the webapi operations
which summarize all related I/O.
Another possibility is to reject this patch, but create a separate
"webapi_quickref.html" with a concise table.
nejucomo [Tue, 29 Jan 2008 05:42:28 +0000 (22:42 -0700)]
tahoe_fuse.py: system test: Distinguish between TestFailures and unexpected exceptions during testing (and fix a typo).
nejucomo [Tue, 29 Jan 2008 05:40:47 +0000 (22:40 -0700)]
tahoe_fuse.py: system test: setup: lexically sort test names, create a TestFailure class, implement an empty directory listing test.
nejucomo [Tue, 29 Jan 2008 05:39:13 +0000 (22:39 -0700)]
tahoe_fuse.py: system test: setup: fixed a bug in which the mointpoint was not created before mounting.
nejucomo [Tue, 29 Jan 2008 05:27:19 +0000 (22:27 -0700)]
tahoe_fuse.py: system test: Many changes to framework...
The flow control has been de-obfuscated a bit.
Some output changes.
The test framework has quite a few race conditions, but it does a reasonable job of setting up and cleaning up.
nejucomo [Tue, 29 Jan 2008 05:25:11 +0000 (22:25 -0700)]
Individual tests run after all the setup layers are in place.
Brian Warner [Sat, 2 Feb 2008 03:28:15 +0000 (20:28 -0700)]
client.py: hush pyflakes
Brian Warner [Sat, 2 Feb 2008 03:07:08 +0000 (20:07 -0700)]
client.py: touch BASEDIR/no_storage to not publish a storage server. Addresses #271
Brian Warner [Sat, 2 Feb 2008 02:48:38 +0000 (19:48 -0700)]
introducer: allow nodes to refrain from publishing themselves, by passing furl=None. This would be useful for clients who do not run storage servers.
Zooko O'Whielacronx [Fri, 1 Feb 2008 19:39:06 +0000 (12:39 -0700)]
docs: architecture.txt: some edits with Amber
Zooko O'Whielacronx [Fri, 1 Feb 2008 19:27:37 +0000 (12:27 -0700)]
rename storage_index_chk_hash() to storage_index_hash() and add TODO about how our use of it now includes keys that are not CHKs
Zooko O'Whielacronx [Fri, 1 Feb 2008 18:39:23 +0000 (11:39 -0700)]
docs: update docs/about.html with Amber
robk-tahoe [Fri, 1 Feb 2008 04:10:15 +0000 (21:10 -0700)]
stats: added IStatsProducer interface, fixed stats provider startup
this adds an interface, IStatsProducer, defining the get_stats() method
which the stats provider calls upon and registered producer, and made the
register_producer() method check that interface is implemented.
also refine the startup logic, so that the stats provider doesn't try and
connect out to the stats gatherer until after the node declares the tub
'ready'. this is to address an issue whereby providers would attach to
the gatherer without providing a valid furl, and hence the gatherer would
be unable to determine the tubid of the connected client, leading to lost
samples.
robk-tahoe [Fri, 1 Feb 2008 04:04:23 +0000 (21:04 -0700)]
munin stats: suppress output of data more that 5min old
if a node fails to report stats, the natural thing to do in re munin is to
supress the data for that data series. the previous tahoe-stats would output
whatever data was present in the stats_gatherer's stats.pickle, regardless of
how old.
this change means that if the gatherer hasn't received data within the last
5 min, then no data is reported to munin for that node.
robk-tahoe [Fri, 1 Feb 2008 02:21:17 +0000 (19:21 -0700)]
munin stats: change truncated nodeid to 4 char (not 8)