Brian Warner [Tue, 12 Jan 2010 01:33:43 +0000 (17:33 -0800)]
Clean up log.err calls, for one of the issues in #889.
allmydata.util.log.err() either takes a Failure as the first positional
argument, or takes no positional arguments and must be invoked in an
exception handler. Fixed its signature to match both foolscap.logging.log.err
and twisted.python.log.err . Included a brief unit test.
Brian Warner [Tue, 12 Jan 2010 00:07:23 +0000 (16:07 -0800)]
tidy up DeadReferenceError handling, ignore them in add_lease calls
Stop checking separately for ConnectionDone/ConnectionLost, since those have
been folded into DeadReferenceError since foolscap-0.3.1 . Write
rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
code coverage.
Brian Warner [Sun, 10 Jan 2010 20:35:52 +0000 (12:35 -0800)]
test_repairer: add (commented-out) test_each_byte, to see exactly what the
Verifier misses
The results (described in #819) match our expectations: it misses corruption
in unused share fields and in most container fields (which are only visible
to the storage server, not the client). 1265 bytes of a 2753 byte
share (hosting a 56-byte file with an artifically small segment size) are
unused, mostly in the unused tail of the overallocated UEB space (765 bytes),
and the allocated-but-unwritten plaintext_hash_tree (480 bytes).
francois [Sat, 9 Jan 2010 17:59:46 +0000 (09:59 -0800)]
contrib/fuse/runtests.py: Tolerate the tahoe CLI returning deprecation warnings
Depending on the versions of external libraries such as Twisted of Foolscap,
the tahoe CLI can display deprecation warnings on stdout. The tests should
not interpret those warnings as a failure if the node is in fact correctly
started.
See http://allmydata.org/trac/tahoe/ticket/859 for an example of deprecation
warnings.
Brian Warner [Sat, 2 Jan 2010 22:08:41 +0000 (14:08 -0800)]
mutable/publish: don't loop() right away upon DeadReferenceError. Closes #877
The bug was that a disconnected server could cause us to re-enter the initial
loop() call, sending multiple queries to a single server, provoking an
incorrect UCWE. To fix it, stall the loop() with an eventual.fireEventually()
Brian Warner [Tue, 29 Dec 2009 23:37:46 +0000 (15:37 -0800)]
mutable repair: return successful=False when numshares<k (thus repair fails),
instead of weird errors. Closes #874 and #786.
Previously, if the file had 0 shares, this would raise TypeError as it tried
to call download_version(None). If the file had some shares but fewer than
'k', it would incorrectly raise MustForceRepairError.
Added get_successful() to the IRepairResults API, to give repair() a place to
report non-code-bug problems like this.
Brian Warner [Tue, 29 Dec 2009 23:01:08 +0000 (15:01 -0800)]
checker: don't let failures in add-lease affect checker results. Closes #875.
Mutable servermap updates and the immutable checker, when run with
add_lease=True, send both the do-you-have-block and add-lease commands in
parallel, to avoid an extra round trip time. Many older servers have problems
with add-lease and raise various exceptions, which don't generally matter.
The client-side code was catching+ignoring some of them, but unrecognized
exceptions were passed through to the DYHB code, concealing the DYHB results
from the checker, making it think the server had no shares.
The fix is to separate the code paths. Both commands are sent at the same
time, but the errback path from add-lease is handled separately. Known
exceptions are ignored, the others (both unknown-remote and all-local) are
logged (log.WEIRD, which will trigger an Incident), but neither will affect
the DYHB results.
The add-lease message is sent first, and we know that the server handles them
synchronously. So when the checker is done, we can be sure that all the
add-lease messages have been retired. This makes life easier for unit tests.
Brian Warner [Sun, 27 Dec 2009 22:54:43 +0000 (17:54 -0500)]
Fix 'tahoe ls' on files (#771). Patch adapted from Kevan Carstensen.
web/filenode.py: also serve edge metadata when using t=json on a
DIRCAP/childname object.
tahoe_ls.py: list file objects as if we were listing one-entry directories.
Show edge metadata if we have it, which will be true when doing
'tahoe ls DIRCAP/filename' and false when doing 'tahoe ls
FILECAP'
Brian Warner [Sun, 27 Dec 2009 20:10:43 +0000 (15:10 -0500)]
webapi: don't accept zero-length childnames during traversal. Closes #358, #676.
This forbids operations that would implicitly create a directory with a
zero-length (empty string) name, like what you'd get if you did "tahoe put
local /oops/blah" (#358) or "POST /uri/CAP//?t=mkdir" (#676). The error
message is fairly friendly too.
Also added code to "tahoe put" to catch this error beforehand and suggest the
correct syntax (i.e. without the leading slash).
Brian Warner [Sun, 27 Dec 2009 19:58:28 +0000 (14:58 -0500)]
CLI: send 'Accept:' header to ask for text/plain tracebacks. Closes #646.
The webapi has been looking for an Accept header since 1.4.0, but it treats a
missing header as equal to */* (to honor RFC2616). This change finally
modifies our CLI tools to ask for "text/plain, application/octet-stream",
which seems roughly correct (we either want a plain-text traceback or error
message, or an uninterpreted chunk of binary data to save to disk). Some day
we'll figure out how JSON fits into this scheme.
Brian Warner [Sun, 27 Dec 2009 16:37:03 +0000 (11:37 -0500)]
Makefile: upload-tarballs: switch from xfer-client to flappclient, closes #350
I've also set up a new flappserver on source@allmydata.org to receive the
tarballs. We still need to replace the gutsy buildslave (which is where the
tarballs used to be generated+uploaded) and give it the new FURL.
docs: remove obsolete doc file "codemap.txt"
I started to update this to reflect the current codebase, but then I thought (a) nobody seemed to notice that it hasn't been updated since December 2007, and (b) it will just bit-rot again, so I'm removing it.
Brian Warner [Tue, 8 Dec 2009 17:11:56 +0000 (09:11 -0800)]
mutable/servermap.py: oops, query N+e servers in MODE_WRITE, not k+e
under normal conditions, this wouldn't cause any problems, but if the shares
are really sparse (perhaps because new servers were added), then
file-modifies might stop looking too early and leave old shares in place
Brian Warner [Tue, 1 Dec 2009 22:44:35 +0000 (17:44 -0500)]
Simplify immutable download API: use just filenode.read(consumer, offset, size)
* remove Downloader.download_to_data/download_to_filename/download_to_filehandle
* remove download.Data/FileName/FileHandle targets
* remove filenode.download/download_to_data/download_to_filename methods
* leave Downloader.download (the whole Downloader will go away eventually)
* add util.consumer.MemoryConsumer/download_to_data, for convenience
(this is mostly used by unit tests, but it gets used by enough non-test
code to warrant putting it in allmydata.util)
* update tests
* removes about 180 lines of code. Yay negative code days!
Overall plan is to rewrite immutable/download.py and leave filenode.read() as
the sole read-side API.
Brian Warner [Thu, 26 Nov 2009 23:42:57 +0000 (15:42 -0800)]
Add dirnodes to backupdb and "tahoe backup", closes #606.
* backups now share dirnodes with any previous backup, in any location,
so renames and moves are handled very efficiently
* "tahoe backup" no longer bothers reading the previous snapshot
* if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite,
to force new uploads of all files and directories
Brian Warner [Wed, 18 Nov 2009 19:16:24 +0000 (11:16 -0800)]
make get_size/get_current_size consistent for all IFilesystemNode classes
* stop caching most_recent_size in dirnode, rely upon backing filenode for it
* start caching most_recent_size in MutableFileNode
* return None when you don't know, not "?"
* only render None as "?" in the web "more info" page
* add get_size/get_current_size to UnknownNode
Brian Warner [Wed, 18 Nov 2009 07:09:00 +0000 (23:09 -0800)]
Add t=mkdir-immutable to the webapi. Closes #607.
* change t=mkdir-with-children to not use multipart/form encoding. Instead,
the request body is all JSON. t=mkdir-immutable uses this format too.
* make nodemaker.create_immutable_dirnode() get convergence from SecretHolder,
but let callers override it
* raise NotDeepImmutableError instead of using assert()
* add mutable= argument to DirectoryNode.create_subdirectory(), default True
Brian Warner [Wed, 11 Nov 2009 22:25:42 +0000 (14:25 -0800)]
clean up uri-vs-cap terminology, emphasize cap instances instead of URI strings
* "cap" means a python instance which encapsulates a filecap/dircap (uri.py)
* "uri" means a string with a "URI:" prefix
* FileNode instances are created with (and retain) a cap instance, and
generate uri strings on demand
* .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances
* .get_uri/get_readonly_uri return uri strings
* add filenode.download_to_filename() for control.py, should find a better way
* use MutableFileNode.init_from_cap, not .init_from_uri
* directory URI instances: use get_filenode_cap, not get_filenode_uri
* update/cleanup bench_dirnode.py to match, add Makefile target to run it
Brian Warner [Mon, 26 Oct 2009 01:13:21 +0000 (18:13 -0700)]
webapi: use t=mkdir-with-children instead of a children= arg to t=mkdir .
This is safer: in the earlier API, an old webapi server would silently ignore
the initial children, and clients trying to set them would have to fetch the
newly-created directory to discover the incompatibility. In the new API,
clients using t=mkdir-with-children against an old webapi server will get a
clear error.
Brian Warner [Sat, 17 Oct 2009 20:31:46 +0000 (13:31 -0700)]
nodemaker.create_new_mutable_directory: pack_children() in initial_contents=
instead of creating an empty file and then adding the children later.
This should speed up mkdir(initial_children) considerably, removing two
roundtrips and an entire read-modify-write cycle, probably bringing it down
to a single roundtrip. A quick test (against the volunteergrid) suggests a
30% speedup.
test_dirnode: add new tests to enforce the restrictions that interfaces.py
claims for create_new_mutable_directory(): no UnknownNodes, metadata dicts
Brian Warner [Sat, 17 Oct 2009 19:28:29 +0000 (12:28 -0700)]
update many dirnode interfaces to accept dict-of-nodes instead of dict-of-caps
interfaces.py: define INodeMaker, document argument values, change
create_new_mutable_directory() to take dict-of-nodes. Change
dirnode.set_nodes() and dirnode.create_subdirectory() too.
nodemaker.py: use INodeMaker, update create_new_mutable_directory()
client.py: have create_dirnode() delegate initial_children= to nodemaker
dirnode.py (Adder): take dict-of-nodes instead of list-of-nodes, which
updates set_nodes() and create_subdirectory()
web/common.py (convert_initial_children_json): create dict-of-nodes
web/directory.py: same
web/unlinked.py: same
test_dirnode.py: update tests to match
Brian Warner [Sat, 17 Oct 2009 18:00:05 +0000 (11:00 -0700)]
move dirnode.CachingDict to dictutil.AuxValueDict, generalize method names,
improve tests. Let dirnode _pack_children accept either dict or AuxValueDict.
Brian Warner [Tue, 13 Oct 2009 03:12:32 +0000 (20:12 -0700)]
client.create_mutable_file(contents=) now accepts a callable, which is
invoked with the new MutableFileNode and is supposed to return the initial
contents. This can be used by e.g. a new dirnode which needs the filenode's
writekey to encrypt its initial children.
create_mutable_file() still accepts a bytestring too, or None for an empty
file.