robk-tahoe [Wed, 2 Apr 2008 01:45:13 +0000 (18:45 -0700)]
added offloaded key generation
this adds a new service to pre-generate RSA key pairs. This allows
the expensive (i.e. slow) key generation to be placed into a process
outside the node, so that the node's reactor will not block when it
needs a key pair, but instead can retrieve them from a pool of already
generated key pairs in the key-generator service.
it adds a tahoe create-key-generator command which initialises an
empty dir with a tahoe-key-generator.tac file which can then be run
via twistd. it stashes its .pem and portnum for furl stability and
writes the furl of the key gen service to key_generator.furl, also
printing it to stdout.
by placing a key_generator.furl file into the nodes config directory
(e.g. ~/.tahoe) a node will attempt to connect to such a service, and
will use that when creating mutable files (i.e. directories) whenever
possible. if the keygen service is unavailable, it will perform the
key generation locally instead, as before.
Zooko O'Whielacronx [Tue, 1 Apr 2008 19:36:09 +0000 (12:36 -0700)]
setup: rename GNUmakefile to Makefile
It's evil and wrong to call something a "Makefile" when it contains code that can't be interpreted by POSIX make and requires GNU make.
But everyone else is doing it. ;-)
Brian Warner [Mon, 31 Mar 2008 22:28:45 +0000 (15:28 -0700)]
introducer.py: accelerate reconnection after being offline. Closes #374.
When we establish any new connection, reset the delays on all the other
Reconnectors. This will trigger a new batch of connection attempts. The idea
is to detect when we (the client) have been offline for a while, and to
connect to all servers when we get back online. By accelerating the timers
inside the Reconnectors, we try to avoid spending a long time in a
partially-connected state (which increases the chances of causing problems
with mutable files, by not updating all the shares that we ought to).
Brian Warner [Fri, 28 Mar 2008 23:18:30 +0000 (16:18 -0700)]
munin plugins: add 'graph_category tahoe'
Brian Warner [Fri, 28 Mar 2008 23:18:09 +0000 (16:18 -0700)]
add munin/tahoe-rootdir-space
Brian Warner [Fri, 28 Mar 2008 21:51:37 +0000 (14:51 -0700)]
munin/tahoe-introstats.py: put this graph in the 'tahoe' category
Peter Secor [Fri, 28 Mar 2008 19:25:08 +0000 (12:25 -0700)]
native client - removed unused file (used to be for elevating privileges) and changed installer to start up the AllmydataTray executable directly
Peter Secor [Fri, 28 Mar 2008 06:41:07 +0000 (23:41 -0700)]
native client - changing way the executables ask for elevated privileges
Brian Warner [Fri, 28 Mar 2008 00:33:58 +0000 (17:33 -0700)]
storage: emit log messages on bucket allocate/read and mutable writev
Zooko O'Whielacronx [Fri, 28 Mar 2008 02:01:29 +0000 (19:01 -0700)]
docs: fix anchor text of hyperlink to tarball
Brian Warner [Thu, 27 Mar 2008 23:50:30 +0000 (16:50 -0700)]
helper: add another munin plugin
Brian Warner [Thu, 27 Mar 2008 23:46:08 +0000 (16:46 -0700)]
helper: add more stats to webapi, at /helper_status
Brian Warner [Thu, 27 Mar 2008 22:55:32 +0000 (15:55 -0700)]
helper: add stats for the gatherer, show some on the webish welcome page
Brian Warner [Thu, 27 Mar 2008 18:33:42 +0000 (11:33 -0700)]
add GET /uri/URI/?t=deep-size, to compute the total size of immutable files reachable from a given directory
Peter Secor [Thu, 27 Mar 2008 04:26:31 +0000 (21:26 -0700)]
native client - updated to auto mount drive at start as well as when the drive is opened
Peter Secor [Thu, 27 Mar 2008 04:05:30 +0000 (21:05 -0700)]
native client - updated to automatically create a Backup directory, temp directory cleanup, automatic mounting and unmounting of the drive when starting and stopping the service, lots of Vista backup error fixes
Brian Warner [Thu, 27 Mar 2008 01:37:54 +0000 (18:37 -0700)]
node.py: make twistd.pid world-readable, so stats-gathering tools (specifically memory-measuring munin plugins) can see it
Brian Warner [Thu, 27 Mar 2008 01:22:07 +0000 (18:22 -0700)]
boodlegrid.tac: update sound samples
Brian Warner [Thu, 27 Mar 2008 01:20:07 +0000 (18:20 -0700)]
web-status: client methods like list_all_uploads() return Upload instances,
not status instances. Fix this. The symptom was that following a link like
'up-123' that referred to an old operation (no longer in memory) while an
upload was active would get an ugly traceback instead of a "no such resource"
message.
Brian Warner [Wed, 26 Mar 2008 23:09:34 +0000 (16:09 -0700)]
misc/boodlegrid.tac: tool to monitor a grid through its flogports
Zooko O'Whielacronx [Wed, 26 Mar 2008 19:13:21 +0000 (12:13 -0700)]
setup: remove bundled setuptools-0.6c7
Zooko O'Whielacronx [Wed, 26 Mar 2008 19:13:02 +0000 (12:13 -0700)]
setup: require setuptools >= v0.6c8
Zooko O'Whielacronx [Wed, 26 Mar 2008 19:12:34 +0000 (12:12 -0700)]
setup: bundle setuptools-0.6c8, we need a bugfix in it
Zooko O'Whielacronx [Wed, 26 Mar 2008 19:11:28 +0000 (12:11 -0700)]
setup: merge in changes to ez_setup.py from the upstream setuptools project
Zooko O'Whielacronx [Wed, 26 Mar 2008 17:00:33 +0000 (10:00 -0700)]
setup: simplify makefile's path manipulation now that we rely on setup.py develop
zandr [Wed, 26 Mar 2008 01:42:11 +0000 (18:42 -0700)]
tahoe-nodememory.py: change category to Tahoe
Zooko O'Whielacronx [Wed, 26 Mar 2008 03:22:29 +0000 (20:22 -0700)]
docs: link to the 1.0.0 tarball in docs/install.html
Zooko O'Whielacronx [Wed, 26 Mar 2008 01:29:04 +0000 (18:29 -0700)]
TAG allmydata-tahoe-1.0.0
Zooko O'Whielacronx [Wed, 26 Mar 2008 01:28:00 +0000 (18:28 -0700)]
docs: update relnotes.txt for Tahoe v1.0!
zandr [Wed, 26 Mar 2008 00:50:04 +0000 (17:50 -0700)]
Copy amd-nodememory munin plugin over to tahoe and point at correct pidfile
Peter Secor [Wed, 26 Mar 2008 00:00:59 +0000 (17:00 -0700)]
native client - updated to fix windows vista backup rproblems, edit word documents directly on the drive, requeue files that failed to upload from the node to the helper
Brian Warner [Tue, 25 Mar 2008 20:15:52 +0000 (13:15 -0700)]
add a munin plugin to display introducer stats
Brian Warner [Tue, 25 Mar 2008 19:56:12 +0000 (12:56 -0700)]
introweb.py: add ?t=json, to provide machine-readable subscriber counts
Brian Warner [Tue, 25 Mar 2008 02:08:15 +0000 (19:08 -0700)]
encode.py: also record the size, along with plaintext_hash and SI
Brian Warner [Tue, 25 Mar 2008 01:55:37 +0000 (18:55 -0700)]
encode: log a plaintext hash and SI for each upload. This will allow the log gatherer to correlate the two, to better measure the benefits of convergence
robk-tahoe [Mon, 24 Mar 2008 22:47:12 +0000 (15:47 -0700)]
confwiz: set a convergence domain based on root_dir upon config
when the confwiz configures a node (i.e. typically once on mac, once per
install on windows) in addition to writing the root_dir.cap retrieved from
the native_client backend into a config file, it additionally writes a hash
thereof into the 'convergence' config file.
this causes uploads from this node to use a consistent 'convergence' hashing
value matching any other nodes with the same configured root_dir, i.e. for
the most part other systems installed and configured on the same account.
robk-tahoe [Mon, 24 Mar 2008 22:46:28 +0000 (15:46 -0700)]
mutable: revise a couple of error messages
at brian and zooko's suggestion, reword an error message encountered when
multiple writers are racing to make overlapping changes to a directory
Brian Warner [Mon, 24 Mar 2008 22:51:19 +0000 (15:51 -0700)]
upload.py: remove spurious trailing quotes that confuse syntax-highlighting
Brian Warner [Mon, 24 Mar 2008 20:39:51 +0000 (13:39 -0700)]
disable plaintext hashes in shares, but leave a switch to turn it back on
Brian Warner [Sun, 23 Mar 2008 22:35:54 +0000 (15:35 -0700)]
UNDO: upload: stop putting plaintext and ciphertext hashes in shares.
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
Zooko O'Whielacronx [Tue, 25 Mar 2008 18:47:39 +0000 (11:47 -0700)]
CREDITS: add Paul Gerhardt, who submitted a small patch for make check-deps to be more newbie-friendly
Zooko O'Whielacronx [Tue, 25 Mar 2008 18:45:55 +0000 (11:45 -0700)]
setup: don't echo "signal-error" to stdout when testing for errors
This patch is thanks to Paul Gerhardt.
Zooko O'Whielacronx [Tue, 25 Mar 2008 18:22:41 +0000 (11:22 -0700)]
docs: document the private/convergence configuration file
Zooko O'Whielacronx [Mon, 24 Mar 2008 22:28:04 +0000 (15:28 -0700)]
fix check-memory to use new upload API (which requires a "convergence" argument), and change it to measure convergence instead of random-key, since convergence is the use case we care about more
Zooko O'Whielacronx [Mon, 24 Mar 2008 16:46:06 +0000 (09:46 -0700)]
use added secret to protect convergent encryption
Now upload or encode methods take a required argument named "convergence" which can be either None, indicating no convergent encryption at all, or a string, which is the "added secret" to be mixed in to the content hash key. If you want traditional convergent encryption behavior, set the added secret to be the empty string.
This patch also renames "content hash key" to "convergent encryption" in a argument names and variable names. (A different and larger renaming is needed in order to clarify that Tahoe supports immutable files which are not encrypted content-hash-key a.k.a. convergent encryption.)
This patch also changes a few unit tests to use non-convergent encryption, because it doesn't matter for what they are testing and non-convergent encryption is slightly faster.
Brian Warner [Sun, 23 Mar 2008 22:35:54 +0000 (15:35 -0700)]
upload: stop putting plaintext and ciphertext hashes in shares.
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
Brian Warner [Sun, 23 Mar 2008 21:46:49 +0000 (14:46 -0700)]
download: make plaintext and ciphertext hashes in the UEB optional.
Removing the plaintext hashes can help with the guess-partial-information
attack. This does not affect compatibility, but if and when we actually
remove any hashes from the share, that will introduce a
forwards-compatibility break: tahoe-0.9 will not be able to read such files.
Peter Secor [Thu, 20 Mar 2008 22:30:26 +0000 (15:30 -0700)]
native client - adding support for special icons for shared and recycling directories
Brian Warner [Thu, 20 Mar 2008 19:18:41 +0000 (12:18 -0700)]
docs: add some accounting proposals
robk-tahoe [Wed, 19 Mar 2008 00:38:44 +0000 (17:38 -0700)]
resolve conflict in windows installer conf
robk-tahoe [Wed, 19 Mar 2008 00:06:47 +0000 (17:06 -0700)]
confwiz: refine the descriptive text on the conf wiz pages
robk-tahoe [Tue, 18 Mar 2008 23:18:07 +0000 (16:18 -0700)]
windows installer: use an allmydata logo in setup wizard
robk-tahoe [Tue, 18 Mar 2008 23:15:36 +0000 (16:15 -0700)]
confwiz: reworked confwiz look and feel
this changes the confwiz to have a look and feel much more consistent
with that of the innosetup installer it is launched within the context
of. this applies, naturally, primarily to windows.
robk-tahoe [Tue, 18 Mar 2008 23:13:38 +0000 (16:13 -0700)]
macapp: updated icon for mac build
Peter Secor [Tue, 18 Mar 2008 22:12:59 +0000 (15:12 -0700)]
native client - adding checks for elevating and managing privileges on Vista
robk-tahoe [Tue, 18 Mar 2008 20:11:08 +0000 (13:11 -0700)]
webish mkdir-p: added unit test
added a test for the simple mkdir-p hack I added yesterday
checks that mkdir-p can create a directory hierarchy, and that resubmitting
a request for the same path yields the existing dir's uri
Peter Secor [Tue, 18 Mar 2008 17:44:43 +0000 (10:44 -0700)]
windows installer - changed to reflect the new StartAllmydata.exe executable that is used to launch the various pieces of the native client. Also verified that the SMB service was stopped when uninstalling.
Peter Secor [Tue, 18 Mar 2008 17:18:47 +0000 (10:18 -0700)]
native client - added StartAllmydata.exe so that we can start/stop processes in Vista
robk-tahoe [Tue, 18 Mar 2008 01:13:01 +0000 (18:13 -0700)]
add a mkdir-p POST handler
this adds a t=mkdir-p call to directories (accessed by their uri as
/uri/<URI>?t=mkdir=p&path=/some/path) which returns the uri for a
directory at a specified path before the given uri, regardless of
whether the directory exists or whether intermediate directories
need to be created to satisfy the request.
this is used by the migration code in MV to optimise the work of
path traversal which was other wise done on every file PUT
Peter Secor [Mon, 17 Mar 2008 21:22:03 +0000 (14:22 -0700)]
native client - fixes for drive size (now 1TB), running service as adminstrator to allow client stopping and starting in Vista, large number of files fix
Brian Warner [Mon, 17 Mar 2008 20:31:10 +0000 (13:31 -0700)]
web/status.py: guard against file_size=None, closes #354
Zooko O'Whielacronx [Thu, 20 Mar 2008 19:11:09 +0000 (12:11 -0700)]
refactor one of the dispatch routines in webish.py
The behavior is intended to be unchanged by this refactoring. Unit tests show no change in behavior.
Zooko O'Whielacronx [Thu, 13 Mar 2008 20:02:37 +0000 (13:02 -0700)]
docs: link from install.html to the (imminent) location of allmydata-tahoe-0.9.0.tar.gz
Zooko O'Whielacronx [Thu, 13 Mar 2008 19:09:58 +0000 (12:09 -0700)]
TAG allmydata-tahoe-0.9.0
Peter Secor [Thu, 13 Mar 2008 18:22:54 +0000 (11:22 -0700)]
native client - fixed icon for system tray, improved link batching
Zooko O'Whielacronx [Thu, 13 Mar 2008 19:13:26 +0000 (12:13 -0700)]
docs: small edit to relnotes.txt
Zooko O'Whielacronx [Thu, 13 Mar 2008 19:04:20 +0000 (12:04 -0700)]
docs: link to the current CREDITS file from relnotes.txt
Zooko O'Whielacronx [Thu, 13 Mar 2008 19:01:47 +0000 (12:01 -0700)]
docs: fix typo in relnotes.txt
Zooko O'Whielacronx [Thu, 13 Mar 2008 18:56:55 +0000 (11:56 -0700)]
docs: small edit to relnotes.txt
Zooko O'Whielacronx [Thu, 13 Mar 2008 18:43:26 +0000 (11:43 -0700)]
docs: update relnotes.txt for allmydata.org "Tahoe" v0.9.0 !
Whee!
Brian Warner [Thu, 13 Mar 2008 01:00:43 +0000 (18:00 -0700)]
mutable.py: split replace() into update() and overwrite(). Addresses #328.
Zooko O'Whielacronx [Thu, 13 Mar 2008 16:10:11 +0000 (09:10 -0700)]
storage servers announce that they will support clients as old as v0.8.0
Not that anyone pays attention to what storage servers claim about what versions they will support.
Zooko O'Whielacronx [Thu, 13 Mar 2008 16:04:44 +0000 (09:04 -0700)]
CREDITS
Brian Warner [Wed, 12 Mar 2008 02:33:19 +0000 (19:33 -0700)]
introducer: record a timestamp with each announcement, and display it on the introducer's web page
Brian Warner [Wed, 12 Mar 2008 02:28:37 +0000 (19:28 -0700)]
introducer: record a timestamp with each subscriber, and display it on the introducer's web page
Brian Warner [Wed, 12 Mar 2008 02:21:29 +0000 (19:21 -0700)]
introweb: combine announcement and subscriber information to show version+nickname for each client
Brian Warner [Wed, 12 Mar 2008 02:20:10 +0000 (19:20 -0700)]
client: publish a 'stub client' announcement to the introducer, to provide version/nickname information for each client
Brian Warner [Wed, 12 Mar 2008 02:01:18 +0000 (19:01 -0700)]
webish: put a trailing slash in the node.url file, since our unit tests use it, and some versions of twisted or nevow don't automatically add one
Brian Warner [Wed, 12 Mar 2008 01:09:13 +0000 (18:09 -0700)]
introweb.py: tolerate non-setLocationed client tubs
Brian Warner [Wed, 12 Mar 2008 00:41:03 +0000 (17:41 -0700)]
oops, add introducer.xhtml
Brian Warner [Wed, 12 Mar 2008 00:36:25 +0000 (17:36 -0700)]
add a webserver for the Introducer, showing service announcements and subscriber lists
Peter Secor [Tue, 11 Mar 2008 20:31:26 +0000 (13:31 -0700)]
native client - updated executables and dll's to have green icon, preference for log levels, and force a link every 100 files when uploading large amounts of files
Peter Secor [Tue, 11 Mar 2008 18:10:38 +0000 (11:10 -0700)]
makefile - unreverted Zooko's change to setup.py, this originally was to see if it was causing the build to create a non-running installer on windows, but it wasn't the problem.
Brian Warner [Tue, 11 Mar 2008 17:50:31 +0000 (10:50 -0700)]
increase remote-interface size limits to 16EiB by not casually using 'int' as a constraint
Brian Warner [Tue, 11 Mar 2008 07:26:00 +0000 (00:26 -0700)]
mutable: tolerate multiple encodings, using whichever version is recoverable first. Closes #312
Brian Warner [Tue, 11 Mar 2008 06:47:35 +0000 (23:47 -0700)]
test_mutable: improve multiple-encodings test coverage
Brian Warner [Tue, 11 Mar 2008 06:16:28 +0000 (23:16 -0700)]
test_mutable: make test-multiple-encodings work
Brian Warner [Tue, 11 Mar 2008 05:15:43 +0000 (22:15 -0700)]
test_mutable: more test coverage, building up a framework to cause reads to occur in a specific order
Peter Secor [Tue, 11 Mar 2008 05:10:50 +0000 (22:10 -0700)]
native client - updated .exe's and .dll's with better caching, using new multi-child upload call, preferences dialog
Brian Warner [Tue, 11 Mar 2008 03:40:39 +0000 (20:40 -0700)]
makefile: clean up pyutil.egg too
Brian Warner [Tue, 11 Mar 2008 03:39:55 +0000 (20:39 -0700)]
mutable: increase max segsize to 3.5MB, to allow dirnodes with about 10k entries
Brian Warner [Tue, 11 Mar 2008 03:32:14 +0000 (20:32 -0700)]
.darcs-boringfile: add pyutil, since it is in setup_requires now
Zooko O'Whielacronx [Tue, 11 Mar 2008 13:18:03 +0000 (06:18 -0700)]
make the nodeurl start with '127.0.0.1' instead of with 'localhost'
This is because there exist in the wild computers that are misconfigured so that 'localhost' doesn't resolve to 127.0.0.1. On those computers, using 'localhost' for the nodeurl is a security problem, because the user commonly sends valuable caps to the nodeurl.
Zooko O'Whielacronx [Tue, 11 Mar 2008 03:15:04 +0000 (20:15 -0700)]
setup: use "-rNN" with NN the "revision number" instead of "-NN" with NN the "build number" in the Windows builds
Zooko O'Whielacronx [Tue, 11 Mar 2008 03:13:21 +0000 (20:13 -0700)]
setup_require pyutil >= 1.3.16, as the Windows installer builder's misc/sub-ver.py relies on it
Zooko O'Whielacronx [Tue, 11 Mar 2008 02:59:30 +0000 (19:59 -0700)]
setup: remove old bundled pyutil-1.3.12
Zooko O'Whielacronx [Tue, 11 Mar 2008 02:59:20 +0000 (19:59 -0700)]
setup: remove old bundled darcsver-1.1.1
Zooko O'Whielacronx [Tue, 11 Mar 2008 02:58:18 +0000 (19:58 -0700)]
setup: bundle pyutil-1.3.16.tar
Zooko O'Whielacronx [Tue, 11 Mar 2008 02:57:07 +0000 (19:57 -0700)]
setup: setup_require darcsver >= 1.1.2
Zooko O'Whielacronx [Tue, 11 Mar 2008 02:56:47 +0000 (19:56 -0700)]
setup: bundle darcsver-1.1.2.tar