Brian Warner [Fri, 13 Apr 2007 02:41:48 +0000 (19:41 -0700)]
hash trees: further cleanup, to make sure we're validating the right thing
hashtree.py: improve the methods available for finding out which hash nodes
are needed. Change set_hashes() to require that every hash provided can
be validated up to the root.
download.py: validate from the top down, including the URI-derived roothash
in the share hash tree, and stashing the thus-validated share hash for use
in the block hash tree.
Brian Warner [Thu, 12 Apr 2007 22:18:46 +0000 (15:18 -0700)]
download: always validate the blockhash, and don't let the bucket trick us into not validating hashes
Brian Warner [Thu, 12 Apr 2007 21:24:11 +0000 (14:24 -0700)]
hashtree.py: reindent from 2-spaces to 4-spaces. No functional changes.
Brian Warner [Thu, 12 Apr 2007 20:13:25 +0000 (13:13 -0700)]
rename chunk.py to hashtree.py
Brian Warner [Thu, 12 Apr 2007 20:07:40 +0000 (13:07 -0700)]
verify hash chains on incoming blocks
Implement enough of chunk.IncompleteHashTree to be usable.
Rearrange download: all block/hash requests now go through
a ValidatedBucket instance, which is responsible for retrieving
and verifying hashes before providing validated data. Download
was changed to use ValidatedBuckets everywhere instead of
unwrapped RIBucketReader references.
Zooko O'Whielacronx [Mon, 16 Apr 2007 22:12:01 +0000 (15:12 -0700)]
oops -- the previous commit of iputil wasn't the right version
Too bad synchronizing pyutil and allmydata.util includes a manual step.
Zooko O'Whielacronx [Mon, 16 Apr 2007 21:59:13 +0000 (14:59 -0700)]
port iputil to Windows (and Irix, and NetBSD, and Solaris 2, ...)
Zooko O'Whielacronx [Sun, 15 Apr 2007 19:08:12 +0000 (12:08 -0700)]
pyfec: bump version number to 1.0.0a1-2-STABLE
Zooko O'Whielacronx [Sun, 15 Apr 2007 19:07:35 +0000 (12:07 -0700)]
pyfec: add mention of sha256sum to README
Zooko O'Whielacronx [Sun, 15 Apr 2007 01:03:54 +0000 (18:03 -0700)]
pyfec: bump the performance measurement bragging up higher in the README
Zooko O'Whielacronx [Sun, 15 Apr 2007 00:50:40 +0000 (17:50 -0700)]
pyfec: bump version number to 1.0.0a1-1-STABLE
Zooko O'Whielacronx [Sun, 15 Apr 2007 00:48:32 +0000 (17:48 -0700)]
pyfec: add -f option to fec, add more user-friendly handling of filesystem errors and user errors
Zooko O'Whielacronx [Sun, 15 Apr 2007 00:47:48 +0000 (17:47 -0700)]
pyfec: add bragging about how fec is way faster than "par2"
Zooko O'Whielacronx [Sat, 14 Apr 2007 23:00:59 +0000 (16:00 -0700)]
pyfec: more progress indicators, handling of already-existent outfile
Zooko O'Whielacronx [Sat, 14 Apr 2007 23:00:33 +0000 (16:00 -0700)]
pyfec: argparse: add a feature of exclusivecreate to the argparse FileType
Zooko O'Whielacronx [Sat, 14 Apr 2007 22:43:41 +0000 (15:43 -0700)]
pyfec: argparse: simplify
Zooko O'Whielacronx [Sat, 14 Apr 2007 19:54:51 +0000 (12:54 -0700)]
pyfec: fix up docs, version numbers, bump version to 1.0.0a1-0-STABLE
Zooko O'Whielacronx [Sat, 14 Apr 2007 19:02:18 +0000 (12:02 -0700)]
pyfec: update README and bump version number to 0.9.9-0-STABLE
Zooko O'Whielacronx [Sat, 14 Apr 2007 19:00:10 +0000 (12:00 -0700)]
pyfec: add bin/fec and bin/unfec, do better handling and reporting of various errors
Zooko O'Whielacronx [Sat, 14 Apr 2007 18:19:24 +0000 (11:19 -0700)]
pyfec: new filefec with compressed metadata, better error handling, much better unit tests
Zooko O'Whielacronx [Sat, 14 Apr 2007 18:05:54 +0000 (11:05 -0700)]
pyfec: import a copy of the argparse module
Zooko O'Whielacronx [Sat, 14 Apr 2007 17:54:43 +0000 (10:54 -0700)]
pyfec: import a copy of mathutil.py from the pyutil library
Zooko O'Whielacronx [Sat, 14 Apr 2007 17:54:02 +0000 (10:54 -0700)]
pyfec: import a copy of fileutil.py from the pyutil library
Zooko O'Whielacronx [Sat, 14 Apr 2007 17:52:45 +0000 (10:52 -0700)]
pyfec: add easyfec.Decoder(), which is easier to use than fec.Decoder()
Zooko O'Whielacronx [Sat, 14 Apr 2007 17:51:48 +0000 (10:51 -0700)]
pyfec: tighten internal datatype from int to short and undo attempt to make it unsigned (Python doesn't do unsigned short)
Zooko O'Whielacronx [Wed, 11 Apr 2007 17:49:33 +0000 (10:49 -0700)]
give it a version number -- v0.9
I intend to bump it to 1.0 after adding the cmdline tools.
Zooko O'Whielacronx [Wed, 11 Apr 2007 17:48:44 +0000 (10:48 -0700)]
import Version from pyutil
Zooko O'Whielacronx [Wed, 11 Apr 2007 17:34:27 +0000 (10:34 -0700)]
merge changes and fix wrong type -- k and m need more than 8 bits (because they are the count rather than the index, i.e. they are 1-indexed)
Zooko O'Whielacronx [Thu, 1 Feb 2007 02:02:57 +0000 (19:02 -0700)]
pyfec: make unit test failures print out a few useful bytes instead of kibibytes
Zooko O'Whielacronx [Thu, 1 Feb 2007 05:55:28 +0000 (22:55 -0700)]
pyfec: add variant of randomized test which happens to match the interface to benchutil
Zooko O'Whielacronx [Thu, 1 Feb 2007 02:02:24 +0000 (19:02 -0700)]
pyfec: make randomized unit tests more comprehensive
Zooko O'Whielacronx [Wed, 31 Jan 2007 20:55:31 +0000 (13:55 -0700)]
pyfec: tighten internal C types
Zooko O'Whielacronx [Thu, 1 Feb 2007 02:01:40 +0000 (19:01 -0700)]
pyfec: fix precondition checks on k and m to actually check the value before coercing it into a smaller type (oops)
Zooko O'Whielacronx [Wed, 31 Jan 2007 23:53:16 +0000 (16:53 -0700)]
pyfec: loosen preconditions -- you can have up to 256 total shares, not up to 255 total shares
Zooko O'Whielacronx [Wed, 11 Apr 2007 15:50:59 +0000 (08:50 -0700)]
silence warning when building
Zooko O'Whielacronx [Tue, 10 Apr 2007 19:24:32 +0000 (12:24 -0700)]
port GNUmakefile to Windows, cygwin, Linux, MacOSX/PPC, MacOSX/Intel
And of course it may well work on lots of other modern unixes, too, especially the more GNUish ones.
Zooko O'Whielacronx [Tue, 10 Apr 2007 17:33:38 +0000 (10:33 -0700)]
merge TODO into roadmap.txt
Zooko O'Whielacronx [Tue, 10 Apr 2007 16:27:58 +0000 (09:27 -0700)]
add TODO: port to Windows
Zooko O'Whielacronx [Mon, 9 Apr 2007 23:20:34 +0000 (16:20 -0700)]
update TODO (add operational TODOs)
Zooko O'Whielacronx [Mon, 9 Apr 2007 23:14:44 +0000 (16:14 -0700)]
tweak README
Brian Warner [Sun, 8 Apr 2007 20:02:13 +0000 (13:02 -0700)]
test_util: add more coverage for assertutil.py
Brian Warner [Sun, 8 Apr 2007 19:43:01 +0000 (12:43 -0700)]
test_util: add full coverage for mathutil.py
Brian Warner [Sat, 7 Apr 2007 05:51:19 +0000 (22:51 -0700)]
download: retrieve share hashes when downloading. We don't really do much validation with them yet, though.
Brian Warner [Sat, 7 Apr 2007 03:56:39 +0000 (20:56 -0700)]
codec: remove more unused debugging code
Brian Warner [Sat, 7 Apr 2007 03:55:59 +0000 (20:55 -0700)]
node: enable Tub.logLocalFailures now that we require foolscap-0.1.2
Brian Warner [Sat, 7 Apr 2007 03:55:37 +0000 (20:55 -0700)]
codec: remove some unused debugging code
Brian Warner [Sat, 7 Apr 2007 03:51:32 +0000 (20:51 -0700)]
debian: remove the lines that install the now-removed .tac files
Brian Warner [Sat, 7 Apr 2007 03:37:38 +0000 (20:37 -0700)]
misc cleanup: remove old .tac files, move old stuff into misc/
Brian Warner [Sat, 7 Apr 2007 03:34:32 +0000 (20:34 -0700)]
test_system: bump up timeouts for the sake of slow slave1, give each test a separate base directory
Brian Warner [Sat, 7 Apr 2007 01:17:23 +0000 (18:17 -0700)]
include pyfec in the trial tests
Brian Warner [Sat, 7 Apr 2007 01:04:38 +0000 (18:04 -0700)]
encode: add more logging to investigate occasional test failures
Brian Warner [Sat, 7 Apr 2007 00:41:01 +0000 (17:41 -0700)]
figleaf output: include a stats.out for parsing by a munin plugin
Brian Warner [Sat, 7 Apr 2007 00:21:25 +0000 (17:21 -0700)]
test_util.py: sigh, one last minor python-2.5 issue
Brian Warner [Sat, 7 Apr 2007 00:12:26 +0000 (17:12 -0700)]
test_util.py: fix another minor python-2.5 issue
Brian Warner [Fri, 6 Apr 2007 23:36:22 +0000 (16:36 -0700)]
test_util.py: fix problems
Brian Warner [Fri, 6 Apr 2007 23:33:22 +0000 (16:33 -0700)]
more allmydata.util test coverage, rename the test case to be more correct
Brian Warner [Fri, 6 Apr 2007 23:29:25 +0000 (16:29 -0700)]
add test coverage for stuff in allmydata/utils
Brian Warner [Fri, 6 Apr 2007 22:45:45 +0000 (15:45 -0700)]
encode/upload: add more logging, to understand the test failure on a slow buildslave
Brian Warner [Fri, 6 Apr 2007 16:09:57 +0000 (09:09 -0700)]
chunk: add IncompleteHashTree for download purposes, plus tests
Brian Warner [Fri, 6 Apr 2007 05:36:18 +0000 (22:36 -0700)]
encode: clean up some weirdness that was there to make unit tests easier to write
Brian Warner [Fri, 6 Apr 2007 04:17:42 +0000 (21:17 -0700)]
rename encode_new.py to encode.py, now that there isn't an old one anymore
Brian Warner [Thu, 5 Apr 2007 19:34:22 +0000 (12:34 -0700)]
update roadmap: add download-peer-selection
Brian Warner [Thu, 5 Apr 2007 00:55:35 +0000 (17:55 -0700)]
add more useful text to the README
Brian Warner [Wed, 4 Apr 2007 23:39:25 +0000 (16:39 -0700)]
add .deb targets for edgy
Brian Warner [Wed, 4 Apr 2007 23:09:13 +0000 (16:09 -0700)]
tests: clean up tearDown to use flushEventualQueue instead of hacking fixed-time delays
Brian Warner [Wed, 4 Apr 2007 22:59:36 +0000 (15:59 -0700)]
allmydata.interfaces: remove some of the placeholders now that we require foolscap-0.1.2
Zooko O'Whielacronx [Wed, 4 Apr 2007 23:12:30 +0000 (16:12 -0700)]
use the "binary" flag on open() for files that shouldn't have line-endings automatically converted
Zooko O'Whielacronx [Wed, 4 Apr 2007 18:18:38 +0000 (11:18 -0700)]
port the makefile to Windows and update the README
Zooko O'Whielacronx [Tue, 3 Apr 2007 17:32:47 +0000 (10:32 -0700)]
add Operations/Deployment for first open source release
As per discussion I had with Peter this morning, we should do these three things and then release the first open source version.
Zooko O'Whielacronx [Tue, 3 Apr 2007 17:32:34 +0000 (10:32 -0700)]
roadmap.txt: editing
Zooko O'Whielacronx [Tue, 3 Apr 2007 17:16:44 +0000 (10:16 -0700)]
mark multiple shares per verifierid as done
Zooko O'Whielacronx [Tue, 3 Apr 2007 17:16:20 +0000 (10:16 -0700)]
roadmap: updated who is working on what
Namely that we don't know who is working on what at the moment...
Zooko O'Whielacronx [Tue, 3 Apr 2007 17:14:34 +0000 (10:14 -0700)]
roadmap.txt: mark LFE as done
Zooko O'Whielacronx [Mon, 2 Apr 2007 17:23:24 +0000 (10:23 -0700)]
tidiness: return res from logging of progress
Brian Warner [Sun, 1 Apr 2007 03:38:43 +0000 (20:38 -0700)]
remove util.ring, no longer used. also remove test_ring.py
Zooko O'Whielacronx [Mon, 9 Apr 2007 23:16:18 +0000 (16:16 -0700)]
pyfec: tidy-up: #include the appropriate header
Brian Warner [Sat, 7 Apr 2007 01:16:50 +0000 (18:16 -0700)]
pyfec: trial-ize the unit tests, making sure to keep working if trial is unavailable
Zooko O'Whielacronx [Fri, 30 Mar 2007 18:52:43 +0000 (11:52 -0700)]
pyfec: rename and clarify -- "blocks" are the units of input/output of the codec, "shares" are sequences of blocks (used to process arbitrary-length files)
Zooko O'Whielacronx [Sat, 31 Mar 2007 01:01:56 +0000 (18:01 -0700)]
fix BucketWriter to not create a finalhome until it is complete, and to clean up the empty $VERIFIERID dir under incoming/ when it moves the last share out of it
Zooko O'Whielacronx [Sat, 31 Mar 2007 01:01:37 +0000 (18:01 -0700)]
log a running total of how much of your file has been uploaded
Zooko O'Whielacronx [Sat, 31 Mar 2007 01:00:40 +0000 (18:00 -0700)]
assert that only dicts get passed to _got_response()
Zooko O'Whielacronx [Sat, 31 Mar 2007 00:12:07 +0000 (17:12 -0700)]
fix storage server to handle the case that it has no directory at all when someone asks for buckets
Zooko O'Whielacronx [Sat, 31 Mar 2007 00:07:04 +0000 (17:07 -0700)]
fix bug in storage-server: yes, "0" is a number, Mr. storage server
Brian Warner [Fri, 30 Mar 2007 23:55:04 +0000 (16:55 -0700)]
handle foolscap's annoying tendency to turn everything into a sets.Set, which are not interoperable with 2.4's builtin 'set' type
Brian Warner [Fri, 30 Mar 2007 23:50:50 +0000 (16:50 -0700)]
finish making the new encoder/decoder/upload/download work
Zooko O'Whielacronx [Fri, 30 Mar 2007 21:54:33 +0000 (14:54 -0700)]
add unit tests and fix bugs in upload
Brian Warner [Fri, 30 Mar 2007 20:20:01 +0000 (13:20 -0700)]
add new test for doing an encode/decode round trip, and make it almost work
Brian Warner [Fri, 30 Mar 2007 19:36:16 +0000 (12:36 -0700)]
uri.py: share counts are not base32-encoded
Brian Warner [Fri, 30 Mar 2007 19:30:14 +0000 (12:30 -0700)]
upload: change _compute_uri a bit, get infile in a different way
Brian Warner [Fri, 30 Mar 2007 18:53:03 +0000 (11:53 -0700)]
switch upload to use encode_new, fix a few things (but not nearly all of them)
Brian Warner [Fri, 30 Mar 2007 18:32:57 +0000 (11:32 -0700)]
test_encode: make sure encode_new can produce the data it is supposed to
Brian Warner [Fri, 30 Mar 2007 18:32:47 +0000 (11:32 -0700)]
chunk.py: remove unused non-tagged hash code
Brian Warner [Fri, 30 Mar 2007 18:32:13 +0000 (11:32 -0700)]
encode_new: use tagged (sha256) hashes everywhere
Zooko O'Whielacronx [Fri, 30 Mar 2007 19:12:23 +0000 (12:12 -0700)]
import fileutil, some of which came from amdlib.util and some of which came from the pyutil library
Zooko O'Whielacronx [Fri, 30 Mar 2007 17:52:19 +0000 (10:52 -0700)]
finish storage server and write new download
Zooko O'Whielacronx [Fri, 30 Mar 2007 17:52:11 +0000 (10:52 -0700)]
TODOs
Zooko O'Whielacronx [Fri, 30 Mar 2007 03:19:52 +0000 (20:19 -0700)]
new upload and storage server
Zooko O'Whielacronx [Fri, 30 Mar 2007 03:19:38 +0000 (20:19 -0700)]
add TODO
Brian Warner [Fri, 30 Mar 2007 02:49:40 +0000 (19:49 -0700)]
chunk.py: fix a pyflakes warning
Brian Warner [Thu, 29 Mar 2007 21:48:42 +0000 (14:48 -0700)]
update feisty packaging