Brian Warner [Thu, 16 Aug 2007 07:52:21 +0000 (00:52 -0700)]
setup.py: disable building RSA for now, since it requires GMP. We'll refrain
from adding it to the build-dependencies until we actually want to use RSA.
refactor reading of configuration files in client.py
This makes it so that an optional file which is unreadable or is rm'ed
at the wrong moment will be ignored instead of raising an exception.
It also bums out a couple of unnecessary lines of code (the explicit
".close()" call).
command-line: fix all three commands and all two ways to invoke them to require node-url and give a useful usage string if node-url is absent or of the wrong form
incomplete version of tahoe-put.py
It doesn't exit properly afterward, and it might not do the best things with non-success responses from the server.
(See tahoe-put-web2ish.py for an example of better response handling.)
a first crack at the "put" command-line
There are actually two versions in this patch, one of which requires twisted.web2 and the other of which uses the Python standard library's socket module. The socketish one doesn't know when the web server is done so it hangs after doing its thing. (Oh -- maybe I should add an HTTP header asking the web server to close the connection when finished.) The web2ish one works better in that respect. Neither of them handle error responses from the server very well yet.
After lunch I intend to finish the socketish one.
To try one, mv src/allmydata/scripts/tahoe_put-{socketish,web2ish}.py src/allmydata/scripts/tahoe_put.py .
If you want to try the web2ish one, and you can't find a web2 package to install, you can get one from:
Brian Warner [Wed, 15 Aug 2007 19:55:11 +0000 (12:55 -0700)]
check_memory.py: Disable the 100MB test for now: our buildslave can't currently handle it because the testnet/framework processes uses something like 600M of RSS.
Brian Warner [Thu, 16 Aug 2007 02:04:47 +0000 (19:04 -0700)]
webapi.txt: clear up underspecified items, replace 'webpassword' paragraph
with a section about our expected plans for #98, add more introductory text
to the sections on manipulate-file vs manipulate-directory.
webapi.txt: shorter and hopefully clearer description of names vs. identifiers
Brian (and anyone who has an interest in the API and documentation): please review.
Brian Warner [Tue, 14 Aug 2007 00:45:02 +0000 (17:45 -0700)]
webish: look for when_done= in POST fields as well as queryargs. Closes #101.
We need to look in the fields because that's how we build the mkdir/upload
forms. Without this, uploading or creating directories would leave us on a
page that had just a URI, instead of something actually useful to a human.
foolscap: change using-foolscap.xhtml to say "public key" instead of "private key"
In the first instance, I hope that the tubid is generated from the hash of the public key and not the hash of the private key!
In the latter instances, the phrase "public key" as in "public key certificate" is more common.
Brian Warner [Sat, 11 Aug 2007 00:25:33 +0000 (17:25 -0700)]
webish: reduce POST memory footprint by overriding http.Request
The original twisted.web.http.Request class has a requestReceived method
that parses the form body (in the request's .content filehandle) using
the stdlib cgi.parse_multipart() function. parse_multipart() consumes a
lot of memory when handling large file uploads, because it holds the
arguments entirely in RAM. Nevow's subclass of Request uses cgi.FieldStorage
instead, which is much more memory-efficient.
This patch uses a local subclass of Request and a modified copy of the
requestReceived method. It disables the cgi.parse_multipart parsing and
instead relies upon Nevow's use of FieldStorage. Our code must look for
form elements (which come from the body of the POST request) in req.fields,
rather than assuming that they will be copied into req.args (which now
only contains the query arguments that appear in the URL).
As a result, memory usage is no longer inflated by the size of the file
being uploaded in a POST upload request. Note that cgi.FieldStorage uses
temporary files (tempfile.TemporaryFile) to hold the data.
Brian Warner [Fri, 10 Aug 2007 01:26:17 +0000 (18:26 -0700)]
encode.py: add a reactor turn barrier between segments, to allow Deferreds to retire and free their arguments, all in the name of reducing memory footprint
However, I haven't heard back from Brian yet, so I'm actually going to put them back in the next patch so that I can compare the two versions of webapi.txt side by side.