Zooko O'Whielacronx [Thu, 1 Feb 2007 07:03:25 +0000 (00:03 -0700)]
pyfec: fix preconditions and typing, remove unused error-checking, tidy-up naming and documentation
Zooko O'Whielacronx [Wed, 31 Jan 2007 20:54:20 +0000 (13:54 -0700)]
pyfec: licensing tweak
Brian Warner [Thu, 1 Feb 2007 10:39:13 +0000 (03:39 -0700)]
Makefile: $(PWD) doesn't always work, in particular the buildslaves don't
update environment variables like $(PWD) when the launch commands.
Brian Warner [Thu, 1 Feb 2007 02:12:33 +0000 (19:12 -0700)]
Makefile: parameterize 'trial' and 'python' to enable a python2.5 builder
Brian Warner [Wed, 31 Jan 2007 01:10:37 +0000 (18:10 -0700)]
Makefile: count-lines: ignore build/* files, also add total number of .py files
Brian Warner [Wed, 31 Jan 2007 01:03:45 +0000 (18:03 -0700)]
add new build/instdir directories to the boringfile
Zooko O'Whielacronx [Tue, 30 Jan 2007 23:33:05 +0000 (16:33 -0700)]
rename Crypto to allmydata.Crypto
This kind of makes me think that there is a general principle that things
shouldn't know their own names. After all, a name is what *other* people use
to refer to you. And in the general case, some other people might refer to you
in ways incompatible with the ways that other people refer to you.
Zooko O'Whielacronx [Tue, 30 Jan 2007 23:32:52 +0000 (16:32 -0700)]
cleanup new "instdir" on "make clean"
Zooko O'Whielacronx [Tue, 30 Jan 2007 22:58:45 +0000 (15:58 -0700)]
pyfec: make it compatible with Python 2.4 as well as Python 2.5
Zooko O'Whielacronx [Tue, 30 Jan 2007 19:08:46 +0000 (12:08 -0700)]
pyfec: silence compiler warnings, add -Wall to debugmode compilation
Zooko O'Whielacronx [Tue, 30 Jan 2007 21:57:59 +0000 (14:57 -0700)]
change build system to use subpackages pristinely and ask them to install themselves into an "instdir"
Zooko O'Whielacronx [Tue, 30 Jan 2007 18:45:28 +0000 (11:45 -0700)]
pyfec: update performance measurements
Zooko O'Whielacronx [Tue, 30 Jan 2007 17:49:00 +0000 (10:49 -0700)]
pyfec: merge conflict
Zooko O'Whielacronx [Tue, 30 Jan 2007 17:46:40 +0000 (10:46 -0700)]
pyfec: fix docstring
Zooko O'Whielacronx [Tue, 30 Jan 2007 17:37:35 +0000 (10:37 -0700)]
pyfec: fix preconditions, tighten internal C types, fix bugs in the file-encoding and benchmarking utility functions
Zooko O'Whielacronx [Tue, 30 Jan 2007 17:36:42 +0000 (10:36 -0700)]
pyfec: documentation and licensing
Zooko O'Whielacronx [Tue, 30 Jan 2007 17:28:26 +0000 (10:28 -0700)]
pyfec: COPYING
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:25:36 +0000 (20:25 -0700)]
pyfec: add fec/__init__.py to import all the right names into the fec module namespace
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:22:13 +0000 (20:22 -0700)]
pyfec: delete m-k of the tempfiles at random in the benchmark of the to/from files
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:17:59 +0000 (20:17 -0700)]
pyfec: execute tests on import of test_pyfec.py only if it is the __main__
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:16:12 +0000 (20:16 -0700)]
pyfec: weaken preconditions -- you can have up to 256 shares, and you don't have to pad the individual shares except to make them all the same length
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:15:49 +0000 (20:15 -0700)]
pyfec: move benchmark code from test_pyfec.py into the new bench_pyfec.py and add more benchmark code
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:15:27 +0000 (20:15 -0700)]
pyfec: add utility functions to encode into a bunch of temp files and to decode from a bunch of tempfiles
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:10:53 +0000 (20:10 -0700)]
pyfec: trivial formatting tweak
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:10:28 +0000 (20:10 -0700)]
pyfec: remove (optimize) unused code
Zooko O'Whielacronx [Sat, 27 Jan 2007 03:08:50 +0000 (20:08 -0700)]
pyfec: make README.txt much more detailed and rename some internal variables and add some docstrings
Zooko O'Whielacronx [Sat, 27 Jan 2007 02:58:04 +0000 (19:58 -0700)]
rename the C extension from "fec" to "_fec"
Brian Warner [Sat, 27 Jan 2007 00:31:24 +0000 (17:31 -0700)]
fix some of the filetree/workqueue interface definitions
Zooko O'Whielacronx [Fri, 26 Jan 2007 01:47:04 +0000 (18:47 -0700)]
pyfec: add precondition checks on the values of k and m to constructors
Zooko O'Whielacronx [Fri, 26 Jan 2007 01:46:57 +0000 (18:46 -0700)]
pyfec: trivial formatting
Zooko O'Whielacronx [Fri, 26 Jan 2007 01:46:37 +0000 (18:46 -0700)]
add README.txt
Zooko O'Whielacronx [Fri, 26 Jan 2007 01:08:29 +0000 (18:08 -0700)]
pyfec: improve package metadata
Zooko O'Whielacronx [Fri, 26 Jan 2007 01:02:16 +0000 (18:02 -0700)]
add copyright notices, license, attributions
Zooko O'Whielacronx [Fri, 26 Jan 2007 00:50:26 +0000 (17:50 -0700)]
add dummy function to see how fast we can read in a file and invoke a Python function on each segment
Zooko O'Whielacronx [Fri, 26 Jan 2007 00:33:48 +0000 (17:33 -0700)]
add benchmark
Zooko O'Whielacronx [Thu, 25 Jan 2007 22:25:34 +0000 (15:25 -0700)]
fix tests to not require pyutil and other tweaks
Zooko O'Whielacronx [Thu, 25 Jan 2007 22:25:19 +0000 (15:25 -0700)]
add utility method file_fec
Zooko O'Whielacronx [Thu, 25 Jan 2007 22:24:01 +0000 (15:24 -0700)]
use buffers as generic "read" buffers instead of char buffers
This is a typing kludge -- if your buffers have elements of size > 1 then we will be processing only a subset of the elements and treating each byte of the element as a separate entry.
Oh well.
Zooko O'Whielacronx [Thu, 25 Jan 2007 22:23:19 +0000 (15:23 -0700)]
allow any sequence as input, not just List
Zooko O'Whielacronx [Thu, 25 Jan 2007 22:22:43 +0000 (15:22 -0700)]
add warning of subtle gotcha that can happen with mutable data buffers
Brian Warner [Thu, 25 Jan 2007 00:23:22 +0000 (17:23 -0700)]
download: update all users to match Zooko's change to ICodecDecoder.decode (as it now returns a list instead of a single string)
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:34:02 +0000 (15:34 -0700)]
update interfaces and docs for codec
It now takes a sequence of buffers instead of a single string for both encode and decode, and it also takes a separate sequence of shareids for decode instead of a sequence of tuples, and it returns a sequence of buffers instead of a single string.
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:21:45 +0000 (15:21 -0700)]
add comment
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:20:53 +0000 (15:20 -0700)]
nicer API -- you don't have to shuffle the shares into place before calling decode_all()
Instead it does that shuffling in C inside fecmodule.
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:17:51 +0000 (15:17 -0700)]
stricter typing -- using unsigned char for indexes into shares
Brian Warner [Wed, 24 Jan 2007 22:29:22 +0000 (15:29 -0700)]
hush pyflakes
Brian Warner [Wed, 24 Jan 2007 22:10:53 +0000 (15:10 -0700)]
filetree: make delete() work
Zooko O'Whielacronx [Tue, 23 Jan 2007 00:17:31 +0000 (17:17 -0700)]
pyfec v0.9
Here is the change history from the first darcs era, in reverse chronological
order:
Mon Jan 22 16:12:56 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* move everything into a subdirectory so that I can merge this darcs repo with the tahoe darcs repo
./fec -> ./pyfec/fec
./setup.py -> ./pyfec/setup.py
A ./pyfec/
Mon Jan 22 16:10:17 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* clean up and minimize fec.c
* strip out unused code
* hard-code GF_BITS to 8
* reindent and reformat curly bracket placement
M ./fec/fec.c -655 +324
M ./fec/fec.h -25
Mon Jan 22 14:24:32 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* change API to allow a subset of the shares to be produced, and to just pass back pointers to primary shares instead of copying them
M ./fec/fec.c -24 +40
M ./fec/fec.h -5 +17
M ./fec/fecmodule.c -63 +144
M ./fec/test/test_pyfec.py -16 +25
M ./setup.py -2 +27
Tue Jan 16 23:01:44 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* split encoder from decoder
M ./fec/fecmodule.c -48 +161
M ./fec/test/test_pyfec.py -3 +4
Tue Jan 16 14:35:25 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* it compiles now!
./fec.c -> ./pyfec/fec.c
./fec.h -> ./pyfec/fec.h
./fecmodule.c -> ./pyfec/fecmodule.c
./pyfec -> ./fec
M ./fec/fec.c -109 +85 r13
M ./fec/fec.h -3 +2 r13
M ./fec/fecmodule.c -23 +241 r13
A ./fec/test/
A ./fec/test/test_pyfec.py
A ./pyfec/
A ./setup.py
Tue Jan 9 10:47:58 MST 2007 zooko@zooko.com
* start of new fecmodule.c
A ./fecmodule.c
Mon Jan 1 15:00:04 MST 2007 zooko@zooko.com
* tidy up error handling
M ./fec.c -26 +16
Mon Jan 1 14:06:30 MST 2007 zooko@zooko.com
* remove the on-the-fly encoding option
We don't currently need it.
M ./fec.c -68
M ./fec.h -22
Mon Jan 1 13:53:28 MST 2007 zooko@zooko.com
* original import from Mnet project
A ./fec.c
A ./fec.h
Brian Warner [Mon, 22 Jan 2007 08:17:39 +0000 (01:17 -0700)]
Makefile: add target to count lines of code and TODOs
Brian Warner [Mon, 22 Jan 2007 08:06:09 +0000 (01:06 -0700)]
filetree: change the way addpath works, now we add workqueue steps for all involved subtrees at about the same time, rather than letting one step add the next when it runs. Finally add a (passing) test for uploading files to CHK-based directories
Brian Warner [Sun, 21 Jan 2007 23:03:15 +0000 (16:03 -0700)]
filetree: add vdrive upload/download test, change workqueue relative-filename semantics
Brian Warner [Sun, 21 Jan 2007 23:01:09 +0000 (16:01 -0700)]
queen.sendOnly: ignore DeadReferenceError too
Brian Warner [Sun, 21 Jan 2007 22:15:31 +0000 (15:15 -0700)]
move IWorkQueue into allmydata.interfaces, give VirtualDrive an uploader
Brian Warner [Sun, 21 Jan 2007 22:01:34 +0000 (15:01 -0700)]
move upload/download interfaces to allmydata.interfaces, let SubTreeMaker assert IDownloader-ness of its 'downloader' argument
Brian Warner [Sun, 21 Jan 2007 20:39:40 +0000 (13:39 -0700)]
test_filetree: more vdrive._get_file_uri() coverage
Brian Warner [Sun, 21 Jan 2007 20:31:16 +0000 (13:31 -0700)]
filetree: mark leaf nodes by adding is_leaf_subtree(), stop traversing when we hit them, to make vdrive._get_file_uri() work
Brian Warner [Sun, 21 Jan 2007 11:50:37 +0000 (04:50 -0700)]
filetree: test NoSuchDirectoryError in vdrive.list()
Brian Warner [Sun, 21 Jan 2007 11:18:54 +0000 (04:18 -0700)]
more filetree, workqueue-boxes now hold serialized Nodes, move NodeMaker out to a separate module
Brian Warner [Sun, 21 Jan 2007 04:14:41 +0000 (21:14 -0700)]
filetree: start testing IVirtualDrive, beginning with list()
Brian Warner [Sun, 21 Jan 2007 00:04:56 +0000 (17:04 -0700)]
filetree: put SubTreeMaker and NodeMaker in separate classes
Brian Warner [Sat, 20 Jan 2007 22:50:21 +0000 (15:50 -0700)]
more filetree work, more tests now pass
Brian Warner [Sat, 20 Jan 2007 21:41:51 +0000 (14:41 -0700)]
more filetree, it's actually starting to make sense now
Brian Warner [Sat, 20 Jan 2007 20:02:26 +0000 (13:02 -0700)]
filetable: oops, os.listdir() does not guarantee sorted results
Brian Warner [Sat, 20 Jan 2007 12:13:15 +0000 (05:13 -0700)]
filetree: more tests, still very early
Brian Warner [Sat, 20 Jan 2007 11:52:53 +0000 (04:52 -0700)]
filetree: refactor INode serialization, start on tests
Brian Warner [Sat, 20 Jan 2007 11:25:20 +0000 (04:25 -0700)]
more filetree hacking, still too early to test
Brian Warner [Sat, 20 Jan 2007 06:22:39 +0000 (23:22 -0700)]
checkpont more filetree stuff
Brian Warner [Fri, 19 Jan 2007 09:35:36 +0000 (02:35 -0700)]
snapshot filetree work: fix pyflakes complaints
Brian Warner [Fri, 19 Jan 2007 09:23:03 +0000 (02:23 -0700)]
snapshot filetree work: it's getting close
Brian Warner [Sat, 20 Jan 2007 03:05:33 +0000 (20:05 -0700)]
rename bin/allmydata to bin/allmydata-tahoe, so the package can co-exist with the regular (amdlib) package
Brian Warner [Fri, 19 Jan 2007 09:17:48 +0000 (02:17 -0700)]
download.py: fix IDownloader to take a URI
Brian Warner [Fri, 19 Jan 2007 09:17:24 +0000 (02:17 -0700)]
update roadmap
Brian Warner [Thu, 18 Jan 2007 00:00:47 +0000 (17:00 -0700)]
filetree.interfaces: remove not-really-code to improve coverage stats
Brian Warner [Thu, 18 Jan 2007 08:00:11 +0000 (01:00 -0700)]
figleaf.el: add code to auto-enable the annotation mode for all source files
Brian Warner [Wed, 17 Jan 2007 22:34:29 +0000 (15:34 -0700)]
improve test coverage a bit
Brian Warner [Wed, 17 Jan 2007 21:46:02 +0000 (14:46 -0700)]
webish: present real URI in directory listing, not an unnecessarily b2a'ed form
Brian Warner [Wed, 17 Jan 2007 20:54:38 +0000 (13:54 -0700)]
split filetree_new.py up into smaller pieces, in a new subpackage
Brian Warner [Wed, 17 Jan 2007 04:34:34 +0000 (21:34 -0700)]
hush pyflakes warnings
Brian Warner [Wed, 17 Jan 2007 04:29:59 +0000 (21:29 -0700)]
update URI format, include codec name
Brian Warner [Wed, 17 Jan 2007 04:02:00 +0000 (21:02 -0700)]
Makefile: use absolute paths when setting PYTHONPATH
Brian Warner [Wed, 17 Jan 2007 04:01:18 +0000 (21:01 -0700)]
webish: add PBURL to the all-peers table on the welcome page
Brian Warner [Wed, 17 Jan 2007 02:56:30 +0000 (19:56 -0700)]
encode_new.py: recent Foolscap accepts 'None' as a constraint
Brian Warner [Wed, 17 Jan 2007 02:55:53 +0000 (19:55 -0700)]
webish: more verifierid-to-uri transition
Brian Warner [Wed, 17 Jan 2007 02:43:13 +0000 (19:43 -0700)]
vdrive/webish: finish verifierid/uri transition
Brian Warner [Wed, 17 Jan 2007 02:42:28 +0000 (19:42 -0700)]
upload: fix typo in debug messages
Brian Warner [Wed, 17 Jan 2007 02:35:12 +0000 (19:35 -0700)]
upload: rearrange peer-selection code to be more readable, and fix a silly bug
Brian Warner [Wed, 17 Jan 2007 02:07:23 +0000 (19:07 -0700)]
oops, fix that NotEnoughPeersError instrumentation
Brian Warner [Wed, 17 Jan 2007 01:47:52 +0000 (18:47 -0700)]
add better error reporting to NotEnoughPeersError
Brian Warner [Tue, 16 Jan 2007 22:15:26 +0000 (15:15 -0700)]
change upload to push 2 shares instead of 3
Now that peers can talk to themselves, the 5-node system test won't fail
just because one of the shares was hosted on the downloader (and thus
inaccessible until recently). The 3-share push was there to avoid this
problem.
Brian Warner [Tue, 16 Jan 2007 22:12:49 +0000 (15:12 -0700)]
now that foolscap-0.0.7 accepts connect-to-self, allow peers to know about themselves. We now require foolscap-0.0.7
Brian Warner [Tue, 16 Jan 2007 04:31:10 +0000 (21:31 -0700)]
hush pyflakes warnings
Brian Warner [Tue, 16 Jan 2007 04:22:22 +0000 (21:22 -0700)]
rearrange encode/upload, add URIs, switch to ReplicatingEncoder
Added metadata to the bucket store, which is used to hold the share number
(but the bucket doesn't know that, it just gets a string).
Modified the codec interfaces a bit.
Try to pass around URIs to/from download/upload instead of verifierids.
URI format is still in flux.
Change the current (primitive) file encoder to use a ReplicatingEncoder
because it provides ICodecEncoder. We will be moving to the (less primitive)
file encoder (currently in allmydata.encode_new) eventually, but for now
this change lets us test out PyRS or zooko's upcoming C-based RS codec in
something larger than a single unit test. This primitive file encoder only
uses a single segment, and has no merkle trees.
Also added allmydata.util.deferredutil for a DeferredList wrapper that
errbacks (but only when all component Deferreds have fired) if there were
any errors, which unfortunately is not a behavior available from the standard
DeferredList.
Brian Warner [Mon, 15 Jan 2007 21:01:22 +0000 (14:01 -0700)]
add simple metadata (a single string) to the storage protocol
Brian Warner [Fri, 12 Jan 2007 03:57:14 +0000 (20:57 -0700)]
change IEncoder to ICodecEncoder, to match the previous change
Brian Warner [Fri, 12 Jan 2007 03:51:27 +0000 (20:51 -0700)]
use the word 'codec' for erasure coding, for now. 'encode' is used for file-level segmentation/hashing
Brian Warner [Wed, 10 Jan 2007 04:13:05 +0000 (21:13 -0700)]
sendOnly: oops, I keep forgetting that you can't really use f.trap in a lambda, because it returns the failure it trapped
Brian Warner [Wed, 10 Jan 2007 02:40:36 +0000 (19:40 -0700)]
test_system.py: exercise queen.Roster._lost_node too
Brian Warner [Tue, 9 Jan 2007 08:58:50 +0000 (01:58 -0700)]
workqueue: more tests
Brian Warner [Tue, 9 Jan 2007 08:36:12 +0000 (01:36 -0700)]
workqueue: more tests, coverage now at 63.4%, yay
Brian Warner [Tue, 9 Jan 2007 05:29:42 +0000 (22:29 -0700)]
workqueue: more improvements, more tests