Brian Warner [Sun, 12 Aug 2007 23:24:51 +0000 (16:24 -0700)]
Client.get_permuted_peers: use self.nodeid now that it's fixed
tahoe [Sun, 12 Aug 2007 18:53:18 +0000 (11:53 -0700)]
shorten ids
Zooko O'Whielacronx [Sun, 12 Aug 2007 17:29:38 +0000 (10:29 -0700)]
don't over-encode the nodeid many times with ascii-encoding
tahoe [Fri, 10 Aug 2007 21:51:47 +0000 (14:51 -0700)]
ymy name...
Zooko O'Whielacronx [Sat, 11 Aug 2007 16:45:03 +0000 (09:45 -0700)]
foolscap: change using-foolscap.xhtml to say "public key" instead of "private key"
In the first instance, I hope that the tubid is generated from the hash of the public key and not the hash of the private key!
In the latter instances, the phrase "public key" as in "public key certificate" is more common.
Brian Warner [Sat, 11 Aug 2007 01:21:22 +0000 (18:21 -0700)]
webish: localfile=/localdir= are now disabled by default, a special switch is required to enable them
Brian Warner [Sat, 11 Aug 2007 01:20:22 +0000 (18:20 -0700)]
webapi.txt: separate out debug/test commands, indicate that localfile=/localdir= requires special activation
Brian Warner [Sat, 11 Aug 2007 00:25:33 +0000 (17:25 -0700)]
webish: reduce POST memory footprint by overriding http.Request
The original twisted.web.http.Request class has a requestReceived method
that parses the form body (in the request's .content filehandle) using
the stdlib cgi.parse_multipart() function. parse_multipart() consumes a
lot of memory when handling large file uploads, because it holds the
arguments entirely in RAM. Nevow's subclass of Request uses cgi.FieldStorage
instead, which is much more memory-efficient.
This patch uses a local subclass of Request and a modified copy of the
requestReceived method. It disables the cgi.parse_multipart parsing and
instead relies upon Nevow's use of FieldStorage. Our code must look for
form elements (which come from the body of the POST request) in req.fields,
rather than assuming that they will be copied into req.args (which now
only contains the query arguments that appear in the URL).
As a result, memory usage is no longer inflated by the size of the file
being uploaded in a POST upload request. Note that cgi.FieldStorage uses
temporary files (tempfile.TemporaryFile) to hold the data.
This closes #29.
Brian Warner [Sat, 11 Aug 2007 00:25:28 +0000 (17:25 -0700)]
webish.POSTHandler: fix typo that meant we didn't look for 'name' in req.fields
Brian Warner [Fri, 10 Aug 2007 22:52:27 +0000 (15:52 -0700)]
webapi.txt: minor clarifications and examples
Brian Warner [Fri, 10 Aug 2007 02:45:18 +0000 (19:45 -0700)]
boringfile: ignore the setuptools .egg that ez_setup.py likes to download/build/install during compilation
Brian Warner [Fri, 10 Aug 2007 08:41:34 +0000 (01:41 -0700)]
Makefile: add 'upload-self' pass to check-memory
Brian Warner [Fri, 10 Aug 2007 01:30:24 +0000 (18:30 -0700)]
#96: add flag to enable pushing data to ourselves, defaulting to False
Brian Warner [Fri, 10 Aug 2007 01:28:45 +0000 (18:28 -0700)]
encode.py: log a percentage complete as well as the raw byte counts
Brian Warner [Fri, 10 Aug 2007 01:26:56 +0000 (18:26 -0700)]
Encoder.__repr__: mention the file being encoded
Brian Warner [Fri, 10 Aug 2007 01:26:17 +0000 (18:26 -0700)]
encode.py: add a reactor turn barrier between segments, to allow Deferreds to retire and free their arguments, all in the name of reducing memory footprint
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:58:33 +0000 (12:58 -0700)]
webapi.txt: put back the manifest feature
So that we can compare versions of webapi.txt with and without this documentation, side by side.
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:52:37 +0000 (12:52 -0700)]
webapi.txt: put back the localfile feature
So that we can compare versions webapi.txt with and without this documentation side by side.
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:33:29 +0000 (12:33 -0700)]
webapi.txt: add URI-based GET variants
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:24:13 +0000 (12:24 -0700)]
webapi.txt: some editing, and remove the localfile feature and the manifest feature
My motivation to remove these features is as per:
http://allmydata.org/pipermail/tahoe-dev/2007-August/000067.html
However, I haven't heard back from Brian yet, so I'm actually going to put them back in the next patch so that I can compare the two versions of webapi.txt side by side.
Zooko O'Whielacronx [Fri, 10 Aug 2007 19:04:30 +0000 (12:04 -0700)]
webapi.txt: further refactoring and add a section explaining TOCTTOU bugs and how to avoid them by using URIs
Zooko O'Whielacronx [Fri, 10 Aug 2007 17:19:27 +0000 (10:19 -0700)]
webapi.txt: further refactoring and editing to clarify the fact that you don't know whether a thing is a file or a directory before you fetch it
Zooko O'Whielacronx [Fri, 10 Aug 2007 16:43:52 +0000 (09:43 -0700)]
webapi.txt: add "?t=file" flag and reorganize doc to discourage people from thinking that they know before hand the file-or-dir type of the thing that they are naming
Zooko O'Whielacronx [Fri, 10 Aug 2007 15:40:02 +0000 (08:40 -0700)]
fix test_web refactoring so that the WebMixin class isn't a TestCase
Thanks, Brian.
wilcoxjg [Thu, 9 Aug 2007 15:57:45 +0000 (08:57 -0700)]
If do_stop attempts to stop a nonexistent process the OSError exception is caught (by errno) and handled tersely.
Zooko O'Whielacronx [Thu, 9 Aug 2007 20:08:42 +0000 (13:08 -0700)]
refactor test_web so that other tests can use the part of test_web that sets up a simple filesystem
Brian Warner [Thu, 9 Aug 2007 19:53:44 +0000 (12:53 -0700)]
introducer: don't log.err() an initial connection failure, since that flunks tests. Use self.log() instead. Also improve test_client.py to always trigger this case, before it was finishing to quickly to always hit the problem.
Brian Warner [Thu, 9 Aug 2007 18:30:33 +0000 (11:30 -0700)]
check_memory.py: include a single 100MB test
Brian Warner [Fri, 10 Aug 2007 01:26:57 +0000 (18:26 -0700)]
munin/tahoe-storagespace.py: use /bin/du instead of our python version, since
it a) is faster and b) will include filesystem overhead that is a real cost.
Brian Warner [Thu, 9 Aug 2007 18:24:35 +0000 (11:24 -0700)]
webapi.txt: update rfc reference
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:29:03 +0000 (10:29 -0700)]
zfec: required version of setuptools is 0.6c6 on cygwin, 0.6a9 on other platforms
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:28:38 +0000 (10:28 -0700)]
zfec: default setuptools version is now 0.6c6
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:19:13 +0000 (10:19 -0700)]
zfec: add benchmark utility
Zooko O'Whielacronx [Fri, 20 Jul 2007 22:51:24 +0000 (15:51 -0700)]
zfec: add to README.txt how to run the unit tests
Zooko O'Whielacronx [Fri, 20 Jul 2007 20:57:47 +0000 (13:57 -0700)]
zfec: fix link to zfec-dev mailing list
Zooko O'Whielacronx [Fri, 20 Jul 2007 20:57:01 +0000 (13:57 -0700)]
zfec: fix README.txt to explain how to use setuptools with GNU stow better
Also remove requirement of setuptools >= v0.6c3. I *think* that using setuptools in the modern way relieves this requirement, but I haven't tested it on dapper yet.
Zooko O'Whielacronx [Fri, 20 Jul 2007 20:55:46 +0000 (13:55 -0700)]
zfec: clarify licensing requirements (transitive grace period)
Zooko O'Whielacronx [Mon, 4 Jun 2007 11:45:17 +0000 (04:45 -0700)]
zfec: shebang usr bin env python in ez_setup.py
Zooko O'Whielacronx [Mon, 4 Jun 2007 11:44:44 +0000 (04:44 -0700)]
zfec: update doc strings in fileutil
Zooko O'Whielacronx [Thu, 9 Aug 2007 17:56:47 +0000 (10:56 -0700)]
fix iputil so that it doesn't launch dozens of processes when you give it a full path and so that it tries executables in preference order and stops as soon as one gives out a dotted-quad string
wilcoxjg [Thu, 9 Aug 2007 04:17:54 +0000 (21:17 -0700)]
in --> across
wilcoxjg [Thu, 9 Aug 2007 04:11:54 +0000 (21:11 -0700)]
cleaning grammar
wilcoxjg [Thu, 9 Aug 2007 03:53:28 +0000 (20:53 -0700)]
/twistd.log -> /logs/twistd.log
Zooko O'Whielacronx [Thu, 9 Aug 2007 05:31:12 +0000 (22:31 -0700)]
misc/sizes.py: rename "block" to "share" and "subblock" to "block"
This renaming is already in place in all of our other documentation and code, but misc/sizes.py got overlooked.
Zooko O'Whielacronx [Thu, 9 Aug 2007 05:31:05 +0000 (22:31 -0700)]
architecture.txt: small edits
Zooko O'Whielacronx [Thu, 9 Aug 2007 04:43:48 +0000 (21:43 -0700)]
startstop_node now works if the logs directory already exists
Thanks to Arno and Arch for pointing this out.
Zooko O'Whielacronx [Thu, 9 Aug 2007 03:47:42 +0000 (20:47 -0700)]
README: formatting (line-wrapping) of arch_o_median's edits to README
wilcoxjg [Thu, 9 Aug 2007 03:43:40 +0000 (20:43 -0700)]
changed README to inslude simplejson in "The Python Way" install
wilcoxjg [Thu, 9 Aug 2007 03:41:26 +0000 (20:41 -0700)]
changed MakeFile to install simplejson
wilcoxjg [Thu, 9 Aug 2007 03:32:09 +0000 (20:32 -0700)]
edited README "The Python Way" installs to /usr/bin
Zooko O'Whielacronx [Thu, 9 Aug 2007 03:44:58 +0000 (20:44 -0700)]
README: clarify/edit build, install, and test instructions
Zooko O'Whielacronx [Thu, 9 Aug 2007 03:44:42 +0000 (20:44 -0700)]
clarify licence
Zooko O'Whielacronx [Thu, 9 Aug 2007 03:44:03 +0000 (20:44 -0700)]
README: point people to allmydata.org if they are looking for precompiled packages
Zooko O'Whielacronx [Thu, 9 Aug 2007 03:32:45 +0000 (20:32 -0700)]
README: edit to arch_o_median's contribution re: installing the Python way
Zooko O'Whielacronx [Thu, 9 Aug 2007 03:28:32 +0000 (20:28 -0700)]
README: add note about what "darcs get" will do and how to run "darcs pull"
wilcoxjg [Thu, 9 Aug 2007 03:00:35 +0000 (20:00 -0700)]
changed README install "Python Way" section
Brian Warner [Thu, 9 Aug 2007 08:32:52 +0000 (01:32 -0700)]
now that the buildslave is moved to a new machine, enable the 50MB test
warner-tahoe [Wed, 8 Aug 2007 02:00:08 +0000 (19:00 -0700)]
foolscap: fix .deb-breaking bug (foolscap.lothar.com: #8)
warner-tahoe [Wed, 8 Aug 2007 01:55:47 +0000 (18:55 -0700)]
update foolscap to foolscap-0.1.5, the latest release
Zooko O'Whielacronx [Wed, 8 Aug 2007 21:37:40 +0000 (14:37 -0700)]
shebang /usr/bin/env python in ez_setup.py
This makes it so that "./ez_setup.py" works on more platforms.
Zooko O'Whielacronx [Tue, 7 Aug 2007 21:25:44 +0000 (14:25 -0700)]
fix missing import -- this bug was introduced by the recent patch to relax setuptools version requirements
Zooko O'Whielacronx [Tue, 7 Aug 2007 21:23:23 +0000 (14:23 -0700)]
in relnotes.txt, make the safety warnings slightly less scary
and update the reference to the wiki's UseCases page
Zooko O'Whielacronx [Tue, 7 Aug 2007 21:15:54 +0000 (14:15 -0700)]
require setuptools >= 0.6c6 on cygwin, and >= 0.6a9 on other platforms
Earlier I tried 0.6a9 (which comes in .deb format on Dapper) and something
didn't work, so I raised it to 0.6c3. However, I didn't make a note of what
failed, and so now I'm hoping that what failed was cygwin-specific. Anyway,
everyone look out for setuptools compatibility problems on the your favorite
platform (and I'll check the buildslaves), and we'll raise the required version
as little as possible and only on the problematic platform.
Zooko O'Whielacronx [Tue, 7 Aug 2007 19:18:44 +0000 (12:18 -0700)]
update Arno's entry in CREDITS
Zooko O'Whielacronx [Tue, 7 Aug 2007 19:13:24 +0000 (12:13 -0700)]
update README to reflect existence of "make install" target
Zooko O'Whielacronx [Tue, 7 Aug 2007 19:12:59 +0000 (12:12 -0700)]
update README to reflect requirement of setuptools 0.6c6
tahoe [Tue, 31 Jul 2007 23:55:45 +0000 (16:55 -0700)]
add install target, works at lest on sudo-free cygwin
tahoe [Tue, 31 Jul 2007 23:18:11 +0000 (16:18 -0700)]
setuptools upgrade to 0.6c6
Zooko O'Whielacronx [Thu, 26 Jul 2007 17:48:30 +0000 (10:48 -0700)]
refactor test_uri and add a test of the empty file
Brian Warner [Wed, 25 Jul 2007 03:44:33 +0000 (20:44 -0700)]
move nodelogs from BASEDIR/twistd.log* to BASEDIR/logs/twistd.log*, closes #61
Brian Warner [Wed, 25 Jul 2007 10:21:30 +0000 (03:21 -0700)]
test_web.py: oops, fix syntax error caused by removing comments
Brian Warner [Wed, 25 Jul 2007 03:16:21 +0000 (20:16 -0700)]
test_web.py: remove spurious '# YES' lines, now that all the tests pass
Brian Warner [Wed, 25 Jul 2007 02:44:25 +0000 (19:44 -0700)]
vdrive: eventually create public/private dirnodes event if the vdrive server isn't available at start up
Brian Warner [Wed, 25 Jul 2007 02:43:21 +0000 (19:43 -0700)]
interfaces: move signatures into docstrings, to reduce lines of code and improve code-coverage numbers
Brian Warner [Wed, 25 Jul 2007 01:13:58 +0000 (18:13 -0700)]
remove unused imports: hush pyflakes warnings
Zooko O'Whielacronx [Tue, 24 Jul 2007 20:46:06 +0000 (13:46 -0700)]
switch from rfc 3548 base-32 to z-base-32 except for tubids/nodeids
Brian Warner [Tue, 24 Jul 2007 09:55:42 +0000 (02:55 -0700)]
URI: we now derive the storage index from the encryption key
Brian Warner [Tue, 24 Jul 2007 02:31:53 +0000 (19:31 -0700)]
refactor upload/encode, to split encrypt and encode responsibilities
Brian Warner [Mon, 23 Jul 2007 03:30:05 +0000 (20:30 -0700)]
update architecture.txt a little bit
Brian Warner [Mon, 23 Jul 2007 02:48:44 +0000 (19:48 -0700)]
truncate storage index to 128 bits, since it's derived from a 128 bit AES key
Zooko O'Whielacronx [Mon, 23 Jul 2007 01:23:37 +0000 (18:23 -0700)]
small edit of relnotes
Zooko O'Whielacronx [Mon, 23 Jul 2007 01:21:29 +0000 (18:21 -0700)]
update relnotes.txt to reflect NJS's comments
Brian Warner [Sun, 22 Jul 2007 01:23:15 +0000 (18:23 -0700)]
CHK: remove the storage index from the URI, deriving it from the key instead
Brian Warner [Sun, 22 Jul 2007 00:45:00 +0000 (17:45 -0700)]
uri.py: get keys and index from the URI instance
Brian Warner [Sun, 22 Jul 2007 00:29:01 +0000 (17:29 -0700)]
docs/uri.txt: document current URI formats
Brian Warner [Sat, 21 Jul 2007 22:40:36 +0000 (15:40 -0700)]
uri: implement URI-processing classes, IFileURI/IDirnodeURI, use internally
Brian Warner [Sat, 21 Jul 2007 22:40:13 +0000 (15:40 -0700)]
dirnode.build_manifest(): tolerate cycles in the directory graph
Brian Warner [Fri, 20 Jul 2007 05:53:29 +0000 (22:53 -0700)]
upload: finish refactoring, all unit tests pass now
Brian Warner [Fri, 20 Jul 2007 01:21:44 +0000 (18:21 -0700)]
upload: refactor to enable streaming upload. not all tests pass yet
Brian Warner [Tue, 17 Jul 2007 19:28:29 +0000 (12:28 -0700)]
webish: improve test coverage further
Brian Warner [Tue, 17 Jul 2007 19:16:45 +0000 (12:16 -0700)]
webish: improve test coverage of WebDownloadTarget
Brian Warner [Tue, 17 Jul 2007 17:41:41 +0000 (10:41 -0700)]
Makefile/check-memory: put all stats in ./memstats.out
Brian Warner [Tue, 17 Jul 2007 03:17:51 +0000 (20:17 -0700)]
check_memory.py: test POST-based uploads as well as from-disk ones
Brian Warner [Tue, 17 Jul 2007 03:17:06 +0000 (20:17 -0700)]
vdrive.py: log a note when we successfully retrieve the global root directory
Brian Warner [Tue, 17 Jul 2007 02:47:42 +0000 (19:47 -0700)]
introducer.py: minor rearrangement of methods
Brian Warner [Tue, 17 Jul 2007 02:34:52 +0000 (19:34 -0700)]
check_memory.py: fix benign-but-noisy vdrive.furl handling bug
Brian Warner [Tue, 17 Jul 2007 01:08:55 +0000 (18:08 -0700)]
check_memory: update it, write stats to a file, disable 50MB test for now
Brian Warner [Tue, 17 Jul 2007 01:07:03 +0000 (18:07 -0700)]
client.py: add a 'debug_no_storage' option to throw out all share data
Brian Warner [Tue, 17 Jul 2007 01:06:27 +0000 (18:06 -0700)]
client.py: write control.furl in the right place
robk-org [Tue, 17 Jul 2007 00:05:01 +0000 (17:05 -0700)]
added unit test to webish's rename function
added unit tests to test various permutations of the rename function, and
some sanity checks on the rename-form function.
also added a guard to prevent '/' in from_/to_name in rename calls.