Zooko O'Whielacronx [Thu, 9 Aug 2007 19:40:47 +0000 (12:40 -0700)]
merge duplicate patches re: zfec setuptools
Some patches were done in the tahoe darcs repo and some in the zfec darcs repo.
Zooko O'Whielacronx [Thu, 9 Aug 2007 18:33:25 +0000 (11:33 -0700)]
merge duplicate patches to shebang /usr/bin/python
one patch from the tahoe darcs repo and one patch from the zfec darcs repo
Zooko O'Whielacronx [Thu, 9 Aug 2007 18:10:58 +0000 (11:10 -0700)]
zfec: fix bug in argparse
Brian Warner [Thu, 23 Aug 2007 20:00:39 +0000 (13:00 -0700)]
webish: modify JSON to match zooko's proposed API changes in #118
Zooko O'Whielacronx [Thu, 23 Aug 2007 20:27:00 +0000 (13:27 -0700)]
command-line: remove the --vdrive option (it is now hardcoded to "global")
Zooko O'Whielacronx [Thu, 23 Aug 2007 20:09:44 +0000 (13:09 -0700)]
webapi.txt: minor edits
Zooko O'Whielacronx [Thu, 23 Aug 2007 20:06:06 +0000 (13:06 -0700)]
webapi.txt edits (thanks to Brian Warner)
Zooko O'Whielacronx [Thu, 23 Aug 2007 20:03:26 +0000 (13:03 -0700)]
new improved webapi.txt
As per ticket #118, this refactors the explanation of URIs and paths and changes the JSON metadata schema.
http://allmydata.org/trac/tahoe/ticket/118
Brian Warner [Thu, 23 Aug 2007 00:35:01 +0000 (17:35 -0700)]
web: replace FILE links with /uri -based ones, to prevent an XSS attack against the secret vdrive URI contained in the current URL
Brian Warner [Wed, 22 Aug 2007 22:08:41 +0000 (15:08 -0700)]
MANIFEST.in: include .html and .css files from allmydata/web/ too
Brian Warner [Wed, 22 Aug 2007 22:08:00 +0000 (15:08 -0700)]
web: oops, forgot to add start.html
Brian Warner [Wed, 22 Aug 2007 21:54:34 +0000 (14:54 -0700)]
web: remove /vdrive/private, replace with a start.html file that points at the /uri/PRIVATE_URI, to prevent XSRF attacks
Zooko O'Whielacronx [Wed, 22 Aug 2007 17:29:57 +0000 (10:29 -0700)]
refactor reading of configuration files in client.py
This makes it so that an optional file which is unreadable or is rm'ed
at the wrong moment will be ignored instead of raising an exception.
It also bums out a couple of unnecessary lines of code (the explicit
".close()" call).
Brian Warner [Sat, 18 Aug 2007 06:28:22 +0000 (23:28 -0700)]
client.py: add a newline to "myself.furl" contents
Zooko O'Whielacronx [Fri, 17 Aug 2007 21:59:49 +0000 (14:59 -0700)]
command-line: fix ticket #111 by requiring input to be a local file and sending Content-Length header
Zooko O'Whielacronx [Fri, 17 Aug 2007 21:17:31 +0000 (14:17 -0700)]
command-line: fix a few bugs in the "execute this python file" way to execute rm
Zooko O'Whielacronx [Fri, 17 Aug 2007 20:33:03 +0000 (13:33 -0700)]
relnotes.txt a few tiny branding edits
Uncertain about the branding, but what the heck.
Brian Warner [Fri, 17 Aug 2007 00:47:52 +0000 (17:47 -0700)]
tahoe_put-web2ish.py: clean up imports to be pyflakes-compliant, add newline to output
Brian Warner [Fri, 17 Aug 2007 00:47:24 +0000 (17:47 -0700)]
cli.py: fix typo in synopsis
Brian Warner [Fri, 17 Aug 2007 00:05:26 +0000 (17:05 -0700)]
test_system: weaken the assertion about the 410 error message when downloading a bogus URI, since some systems emit error messages differently
Brian Warner [Fri, 17 Aug 2007 00:03:50 +0000 (17:03 -0700)]
test_system.py: verify that we can replace files in place
Brian Warner [Fri, 17 Aug 2007 00:03:19 +0000 (17:03 -0700)]
dirnode: change the defined behavior of RIVirtualDriveServer.set to allow replace-in-place without raising an exception
Brian Warner [Thu, 16 Aug 2007 23:49:40 +0000 (16:49 -0700)]
test_system.py: many (failing) web tests were accidentally bypassed, fix those. Add some PUT tests.
Brian Warner [Thu, 16 Aug 2007 22:59:15 +0000 (15:59 -0700)]
boringfile: generated file 'version.py' was changed to _version.py
Brian Warner [Thu, 16 Aug 2007 22:43:43 +0000 (15:43 -0700)]
misc/get-version.py: get the version string from the new location, _version.py
Zooko O'Whielacronx [Fri, 17 Aug 2007 20:23:16 +0000 (13:23 -0700)]
command-line: add "rm", and tidy-up variable names, and make it so "allmydata-tahoe spam" prints a usage message instead of returning silently
Zooko O'Whielacronx [Fri, 17 Aug 2007 20:06:43 +0000 (13:06 -0700)]
command-line: remove some redundant options checking
Zooko O'Whielacronx [Fri, 17 Aug 2007 19:54:47 +0000 (12:54 -0700)]
command-line: fix all three commands and all two ways to invoke them to require node-url and give a useful usage string if node-url is absent or of the wrong form
Zooko O'Whielacronx [Fri, 17 Aug 2007 19:25:09 +0000 (12:25 -0700)]
relnotes.txt: edit and update relnotes and clarify licence
Zooko O'Whielacronx [Fri, 17 Aug 2007 19:06:41 +0000 (12:06 -0700)]
tahoe_put.py: don't treat "^HTTP 200 OK" appearing somewhere in the body as if it appeared in the header
Zooko O'Whielacronx [Fri, 17 Aug 2007 19:06:11 +0000 (12:06 -0700)]
tahoe_put.py: require node-url to be provided
Zooko O'Whielacronx [Fri, 17 Aug 2007 19:06:00 +0000 (12:06 -0700)]
CREDITS: update Arno's entry
Zooko O'Whielacronx [Thu, 16 Aug 2007 23:53:27 +0000 (16:53 -0700)]
cmdline: change "--server" to "--node-url" and make it have no default value
Zooko O'Whielacronx [Thu, 16 Aug 2007 23:52:22 +0000 (16:52 -0700)]
relnotes.txt: a few more updates to relnotes.txt
Zooko O'Whielacronx [Thu, 16 Aug 2007 23:30:39 +0000 (16:30 -0700)]
put now exits after doing its work, and it prints a terse message if 200 or 201, and a full dump of response elsewise
Zooko O'Whielacronx [Thu, 16 Aug 2007 23:03:02 +0000 (16:03 -0700)]
relnotes.txt: incomplete update to describe v0.5
more to come...
Zooko O'Whielacronx [Thu, 16 Aug 2007 23:01:01 +0000 (16:01 -0700)]
incomplete version of tahoe-put.py
It doesn't exit properly afterward, and it might not do the best things with non-success responses from the server.
(See tahoe-put-web2ish.py for an example of better response handling.)
Zooko O'Whielacronx [Thu, 16 Aug 2007 22:53:53 +0000 (15:53 -0700)]
webapi.txt: s/dirnodes/directories/
Zooko O'Whielacronx [Thu, 16 Aug 2007 22:38:01 +0000 (15:38 -0700)]
fix version class to preferred format and correct parsing
Zooko O'Whielacronx [Thu, 16 Aug 2007 23:16:41 +0000 (16:16 -0700)]
version_class.py: if you don't have pkg_resources for comparing version numbers, use distutils.version.LooseVersion
Brian Warner [Fri, 17 Aug 2007 03:30:07 +0000 (20:30 -0700)]
README: update/clarify the build-depends packages need to create .debs
Brian Warner [Thu, 16 Aug 2007 19:50:19 +0000 (12:50 -0700)]
scripts: rearrange Options, make --version behave the same for all commands
Zooko O'Whielacronx [Thu, 16 Aug 2007 21:14:41 +0000 (14:14 -0700)]
fix small bug in unit tests which caused spurious failures on Windows
Zooko O'Whielacronx [Thu, 16 Aug 2007 21:10:48 +0000 (14:10 -0700)]
in-line doc edit: s/pyutil/allmydata.util/
Zooko O'Whielacronx [Thu, 16 Aug 2007 21:09:30 +0000 (14:09 -0700)]
import version class and make-version script from pyutil -- fixes win32 build, improves error handling, and eliminates unused features
Zooko O'Whielacronx [Thu, 16 Aug 2007 19:15:38 +0000 (12:15 -0700)]
a first crack at the "put" command-line
There are actually two versions in this patch, one of which requires twisted.web2 and the other of which uses the Python standard library's socket module. The socketish one doesn't know when the web server is done so it hangs after doing its thing. (Oh -- maybe I should add an HTTP header asking the web server to close the connection when finished.) The web2ish one works better in that respect. Neither of them handle error responses from the server very well yet.
After lunch I intend to finish the socketish one.
To try one, mv src/allmydata/scripts/tahoe_put-{socketish,web2ish}.py src/allmydata/scripts/tahoe_put.py .
If you want to try the web2ish one, and you can't find a web2 package to install, you can get one from:
http://allmydata.org/~zooko/repos/twistedweb2snarf/
Brian Warner [Thu, 16 Aug 2007 00:47:10 +0000 (17:47 -0700)]
Makefile: add build_ext target, for convenience
Brian Warner [Thu, 16 Aug 2007 06:14:05 +0000 (23:14 -0700)]
webish.py: allow users to delete (but not create) empty-named files. Closes #94.
Brian Warner [Wed, 15 Aug 2007 22:21:38 +0000 (15:21 -0700)]
webish: implement replace= for POST commands
Brian Warner [Wed, 15 Aug 2007 20:22:23 +0000 (13:22 -0700)]
webish: implement replace= for PUT commands
Brian Warner [Wed, 15 Aug 2007 20:22:01 +0000 (13:22 -0700)]
IDirectoryNode: add has_child() method
Brian Warner [Wed, 15 Aug 2007 19:55:11 +0000 (12:55 -0700)]
check_memory.py: Disable the 100MB test for now: our buildslave can't currently handle it because the testnet/framework processes uses something like 600M of RSS.
Brian Warner [Thu, 16 Aug 2007 02:31:49 +0000 (19:31 -0700)]
webapi.txt: specify replace= behavior on all PUT and POST commands
Brian Warner [Thu, 16 Aug 2007 02:04:47 +0000 (19:04 -0700)]
webapi.txt: clear up underspecified items, replace 'webpassword' paragraph
with a section about our expected plans for #98, add more introductory text
to the sections on manipulate-file vs manipulate-directory.
Zooko O'Whielacronx [Wed, 15 Aug 2007 19:28:04 +0000 (12:28 -0700)]
webapi.txt: shorter and hopefully clearer description of names vs. identifiers
Brian (and anyone who has an interest in the API and documentation): please review.
Brian Warner [Tue, 14 Aug 2007 09:12:30 +0000 (02:12 -0700)]
client.py: make a note in the logs when the auto-shutdown feature is in use
Brian Warner [Tue, 14 Aug 2007 00:45:02 +0000 (17:45 -0700)]
webish: look for when_done= in POST fields as well as queryargs. Closes #101.
We need to look in the fields because that's how we build the mkdir/upload
forms. Without this, uploading or creating directories would leave us on a
page that had just a URI, instead of something actually useful to a human.
Brian Warner [Mon, 13 Aug 2007 20:28:40 +0000 (13:28 -0700)]
docs/configuration.txt: explain the files in the node's basedir, which ones are useful to modify, etc
Brian Warner [Sun, 12 Aug 2007 23:33:51 +0000 (16:33 -0700)]
fix pyflakes warnings from recent b32decode change
Brian Warner [Sun, 12 Aug 2007 23:29:34 +0000 (16:29 -0700)]
test_system.py: add coverage for get_permuted_peers()
Brian Warner [Sun, 12 Aug 2007 23:24:51 +0000 (16:24 -0700)]
Client.get_permuted_peers: use self.nodeid now that it's fixed
tahoe [Sun, 12 Aug 2007 18:53:18 +0000 (11:53 -0700)]
shorten ids
Zooko O'Whielacronx [Sun, 12 Aug 2007 17:29:38 +0000 (10:29 -0700)]
don't over-encode the nodeid many times with ascii-encoding
tahoe [Fri, 10 Aug 2007 21:51:47 +0000 (14:51 -0700)]
ymy name...
Zooko O'Whielacronx [Sat, 11 Aug 2007 16:45:03 +0000 (09:45 -0700)]
foolscap: change using-foolscap.xhtml to say "public key" instead of "private key"
In the first instance, I hope that the tubid is generated from the hash of the public key and not the hash of the private key!
In the latter instances, the phrase "public key" as in "public key certificate" is more common.
Brian Warner [Sat, 11 Aug 2007 01:21:22 +0000 (18:21 -0700)]
webish: localfile=/localdir= are now disabled by default, a special switch is required to enable them
Brian Warner [Sat, 11 Aug 2007 01:20:22 +0000 (18:20 -0700)]
webapi.txt: separate out debug/test commands, indicate that localfile=/localdir= requires special activation
Brian Warner [Sat, 11 Aug 2007 00:25:33 +0000 (17:25 -0700)]
webish: reduce POST memory footprint by overriding http.Request
The original twisted.web.http.Request class has a requestReceived method
that parses the form body (in the request's .content filehandle) using
the stdlib cgi.parse_multipart() function. parse_multipart() consumes a
lot of memory when handling large file uploads, because it holds the
arguments entirely in RAM. Nevow's subclass of Request uses cgi.FieldStorage
instead, which is much more memory-efficient.
This patch uses a local subclass of Request and a modified copy of the
requestReceived method. It disables the cgi.parse_multipart parsing and
instead relies upon Nevow's use of FieldStorage. Our code must look for
form elements (which come from the body of the POST request) in req.fields,
rather than assuming that they will be copied into req.args (which now
only contains the query arguments that appear in the URL).
As a result, memory usage is no longer inflated by the size of the file
being uploaded in a POST upload request. Note that cgi.FieldStorage uses
temporary files (tempfile.TemporaryFile) to hold the data.
This closes #29.
Brian Warner [Sat, 11 Aug 2007 00:25:28 +0000 (17:25 -0700)]
webish.POSTHandler: fix typo that meant we didn't look for 'name' in req.fields
Brian Warner [Fri, 10 Aug 2007 22:52:27 +0000 (15:52 -0700)]
webapi.txt: minor clarifications and examples
Brian Warner [Fri, 10 Aug 2007 02:45:18 +0000 (19:45 -0700)]
boringfile: ignore the setuptools .egg that ez_setup.py likes to download/build/install during compilation
Brian Warner [Fri, 10 Aug 2007 08:41:34 +0000 (01:41 -0700)]
Makefile: add 'upload-self' pass to check-memory
Brian Warner [Fri, 10 Aug 2007 01:30:24 +0000 (18:30 -0700)]
#96: add flag to enable pushing data to ourselves, defaulting to False
Brian Warner [Fri, 10 Aug 2007 01:28:45 +0000 (18:28 -0700)]
encode.py: log a percentage complete as well as the raw byte counts
Brian Warner [Fri, 10 Aug 2007 01:26:56 +0000 (18:26 -0700)]
Encoder.__repr__: mention the file being encoded
Brian Warner [Fri, 10 Aug 2007 01:26:17 +0000 (18:26 -0700)]
encode.py: add a reactor turn barrier between segments, to allow Deferreds to retire and free their arguments, all in the name of reducing memory footprint
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:58:33 +0000 (12:58 -0700)]
webapi.txt: put back the manifest feature
So that we can compare versions of webapi.txt with and without this documentation, side by side.
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:52:37 +0000 (12:52 -0700)]
webapi.txt: put back the localfile feature
So that we can compare versions webapi.txt with and without this documentation side by side.
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:33:29 +0000 (12:33 -0700)]
webapi.txt: add URI-based GET variants
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:24:13 +0000 (12:24 -0700)]
webapi.txt: some editing, and remove the localfile feature and the manifest feature
My motivation to remove these features is as per:
http://allmydata.org/pipermail/tahoe-dev/2007-August/000067.html
However, I haven't heard back from Brian yet, so I'm actually going to put them back in the next patch so that I can compare the two versions of webapi.txt side by side.
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:04:30 +0000 (12:04 -0700)]
webapi.txt: further refactoring and add a section explaining TOCTTOU bugs and how to avoid them by using URIs
Zooko O'Whielacronx [Fri, 10 Aug 2007 17:19:27 +0000 (10:19 -0700)]
webapi.txt: further refactoring and editing to clarify the fact that you don't know whether a thing is a file or a directory before you fetch it
Zooko O'Whielacronx [Fri, 10 Aug 2007 16:43:52 +0000 (09:43 -0700)]
webapi.txt: add "?t=file" flag and reorganize doc to discourage people from thinking that they know before hand the file-or-dir type of the thing that they are naming
Zooko O'Whielacronx [Fri, 10 Aug 2007 15:40:02 +0000 (08:40 -0700)]
fix test_web refactoring so that the WebMixin class isn't a TestCase
Thanks, Brian.
wilcoxjg [Thu, 9 Aug 2007 15:57:45 +0000 (08:57 -0700)]
If do_stop attempts to stop a nonexistent process the OSError exception is caught (by errno) and handled tersely.
Zooko O'Whielacronx [Thu, 9 Aug 2007 20:08:42 +0000 (13:08 -0700)]
refactor test_web so that other tests can use the part of test_web that sets up a simple filesystem
Brian Warner [Thu, 9 Aug 2007 19:53:44 +0000 (12:53 -0700)]
introducer: don't log.err() an initial connection failure, since that flunks tests. Use self.log() instead. Also improve test_client.py to always trigger this case, before it was finishing to quickly to always hit the problem.
Brian Warner [Thu, 9 Aug 2007 18:30:33 +0000 (11:30 -0700)]
check_memory.py: include a single 100MB test
Brian Warner [Fri, 10 Aug 2007 01:26:57 +0000 (18:26 -0700)]
munin/tahoe-storagespace.py: use /bin/du instead of our python version, since
it a) is faster and b) will include filesystem overhead that is a real cost.
Brian Warner [Thu, 9 Aug 2007 18:24:35 +0000 (11:24 -0700)]
webapi.txt: update rfc reference
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:29:03 +0000 (10:29 -0700)]
zfec: required version of setuptools is 0.6c6 on cygwin, 0.6a9 on other platforms
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:28:38 +0000 (10:28 -0700)]
zfec: default setuptools version is now 0.6c6
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:19:13 +0000 (10:19 -0700)]
zfec: add benchmark utility
Zooko O'Whielacronx [Fri, 20 Jul 2007 22:51:24 +0000 (15:51 -0700)]
zfec: add to README.txt how to run the unit tests
Zooko O'Whielacronx [Fri, 20 Jul 2007 20:57:47 +0000 (13:57 -0700)]
zfec: fix link to zfec-dev mailing list
Zooko O'Whielacronx [Fri, 20 Jul 2007 20:57:01 +0000 (13:57 -0700)]
zfec: fix README.txt to explain how to use setuptools with GNU stow better
Also remove requirement of setuptools >= v0.6c3. I *think* that using setuptools in the modern way relieves this requirement, but I haven't tested it on dapper yet.
Zooko O'Whielacronx [Fri, 20 Jul 2007 20:55:46 +0000 (13:55 -0700)]
zfec: clarify licensing requirements (transitive grace period)
Zooko O'Whielacronx [Mon, 4 Jun 2007 11:45:17 +0000 (04:45 -0700)]
zfec: shebang usr bin env python in ez_setup.py
Zooko O'Whielacronx [Mon, 4 Jun 2007 11:44:44 +0000 (04:44 -0700)]
zfec: update doc strings in fileutil
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:56:47 +0000 (10:56 -0700)]
fix iputil so that it doesn't launch dozens of processes when you give it a full path and so that it tries executables in preference order and stops as soon as one gives out a dotted-quad string