Zooko O'Whielacronx [Thu, 25 Jan 2007 22:24:01 +0000 (15:24 -0700)]
use buffers as generic "read" buffers instead of char buffers
This is a typing kludge -- if your buffers have elements of size > 1 then we will be processing only a subset of the elements and treating each byte of the element as a separate entry.
Oh well.
Zooko O'Whielacronx [Thu, 25 Jan 2007 22:23:19 +0000 (15:23 -0700)]
allow any sequence as input, not just List
Zooko O'Whielacronx [Thu, 25 Jan 2007 22:22:43 +0000 (15:22 -0700)]
add warning of subtle gotcha that can happen with mutable data buffers
Brian Warner [Thu, 25 Jan 2007 00:23:22 +0000 (17:23 -0700)]
download: update all users to match Zooko's change to ICodecDecoder.decode (as it now returns a list instead of a single string)
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:34:02 +0000 (15:34 -0700)]
update interfaces and docs for codec
It now takes a sequence of buffers instead of a single string for both encode and decode, and it also takes a separate sequence of shareids for decode instead of a sequence of tuples, and it returns a sequence of buffers instead of a single string.
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:21:45 +0000 (15:21 -0700)]
add comment
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:20:53 +0000 (15:20 -0700)]
nicer API -- you don't have to shuffle the shares into place before calling decode_all()
Instead it does that shuffling in C inside fecmodule.
Zooko O'Whielacronx [Wed, 24 Jan 2007 22:17:51 +0000 (15:17 -0700)]
stricter typing -- using unsigned char for indexes into shares
Brian Warner [Wed, 24 Jan 2007 22:29:22 +0000 (15:29 -0700)]
hush pyflakes
Brian Warner [Wed, 24 Jan 2007 22:10:53 +0000 (15:10 -0700)]
filetree: make delete() work
Zooko O'Whielacronx [Tue, 23 Jan 2007 00:17:31 +0000 (17:17 -0700)]
pyfec v0.9
Here is the change history from the first darcs era, in reverse chronological
order:
Mon Jan 22 16:12:56 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* move everything into a subdirectory so that I can merge this darcs repo with the tahoe darcs repo
./fec -> ./pyfec/fec
./setup.py -> ./pyfec/setup.py
A ./pyfec/
Mon Jan 22 16:10:17 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* clean up and minimize fec.c
* strip out unused code
* hard-code GF_BITS to 8
* reindent and reformat curly bracket placement
M ./fec/fec.c -655 +324
M ./fec/fec.h -25
Mon Jan 22 14:24:32 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* change API to allow a subset of the shares to be produced, and to just pass back pointers to primary shares instead of copying them
M ./fec/fec.c -24 +40
M ./fec/fec.h -5 +17
M ./fec/fecmodule.c -63 +144
M ./fec/test/test_pyfec.py -16 +25
M ./setup.py -2 +27
Tue Jan 16 23:01:44 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* split encoder from decoder
M ./fec/fecmodule.c -48 +161
M ./fec/test/test_pyfec.py -3 +4
Tue Jan 16 14:35:25 MST 2007 "Zooko O'Whielacronx <zooko@zooko.com>"
* it compiles now!
./fec.c -> ./pyfec/fec.c
./fec.h -> ./pyfec/fec.h
./fecmodule.c -> ./pyfec/fecmodule.c
./pyfec -> ./fec
M ./fec/fec.c -109 +85 r13
M ./fec/fec.h -3 +2 r13
M ./fec/fecmodule.c -23 +241 r13
A ./fec/test/
A ./fec/test/test_pyfec.py
A ./pyfec/
A ./setup.py
Tue Jan 9 10:47:58 MST 2007 zooko@zooko.com
* start of new fecmodule.c
A ./fecmodule.c
Mon Jan 1 15:00:04 MST 2007 zooko@zooko.com
* tidy up error handling
M ./fec.c -26 +16
Mon Jan 1 14:06:30 MST 2007 zooko@zooko.com
* remove the on-the-fly encoding option
We don't currently need it.
M ./fec.c -68
M ./fec.h -22
Mon Jan 1 13:53:28 MST 2007 zooko@zooko.com
* original import from Mnet project
A ./fec.c
A ./fec.h
Brian Warner [Mon, 22 Jan 2007 08:17:39 +0000 (01:17 -0700)]
Makefile: add target to count lines of code and TODOs
Brian Warner [Mon, 22 Jan 2007 08:06:09 +0000 (01:06 -0700)]
filetree: change the way addpath works, now we add workqueue steps for all involved subtrees at about the same time, rather than letting one step add the next when it runs. Finally add a (passing) test for uploading files to CHK-based directories
Brian Warner [Sun, 21 Jan 2007 23:03:15 +0000 (16:03 -0700)]
filetree: add vdrive upload/download test, change workqueue relative-filename semantics
Brian Warner [Sun, 21 Jan 2007 23:01:09 +0000 (16:01 -0700)]
queen.sendOnly: ignore DeadReferenceError too
Brian Warner [Sun, 21 Jan 2007 22:15:31 +0000 (15:15 -0700)]
move IWorkQueue into allmydata.interfaces, give VirtualDrive an uploader
Brian Warner [Sun, 21 Jan 2007 22:01:34 +0000 (15:01 -0700)]
move upload/download interfaces to allmydata.interfaces, let SubTreeMaker assert IDownloader-ness of its 'downloader' argument
Brian Warner [Sun, 21 Jan 2007 20:39:40 +0000 (13:39 -0700)]
test_filetree: more vdrive._get_file_uri() coverage
Brian Warner [Sun, 21 Jan 2007 20:31:16 +0000 (13:31 -0700)]
filetree: mark leaf nodes by adding is_leaf_subtree(), stop traversing when we hit them, to make vdrive._get_file_uri() work
Brian Warner [Sun, 21 Jan 2007 11:50:37 +0000 (04:50 -0700)]
filetree: test NoSuchDirectoryError in vdrive.list()
Brian Warner [Sun, 21 Jan 2007 11:18:54 +0000 (04:18 -0700)]
more filetree, workqueue-boxes now hold serialized Nodes, move NodeMaker out to a separate module
Brian Warner [Sun, 21 Jan 2007 04:14:41 +0000 (21:14 -0700)]
filetree: start testing IVirtualDrive, beginning with list()
Brian Warner [Sun, 21 Jan 2007 00:04:56 +0000 (17:04 -0700)]
filetree: put SubTreeMaker and NodeMaker in separate classes
Brian Warner [Sat, 20 Jan 2007 22:50:21 +0000 (15:50 -0700)]
more filetree work, more tests now pass
Brian Warner [Sat, 20 Jan 2007 21:41:51 +0000 (14:41 -0700)]
more filetree, it's actually starting to make sense now
Brian Warner [Sat, 20 Jan 2007 20:02:26 +0000 (13:02 -0700)]
filetable: oops, os.listdir() does not guarantee sorted results
Brian Warner [Sat, 20 Jan 2007 12:13:15 +0000 (05:13 -0700)]
filetree: more tests, still very early
Brian Warner [Sat, 20 Jan 2007 11:52:53 +0000 (04:52 -0700)]
filetree: refactor INode serialization, start on tests
Brian Warner [Sat, 20 Jan 2007 11:25:20 +0000 (04:25 -0700)]
more filetree hacking, still too early to test
Brian Warner [Sat, 20 Jan 2007 06:22:39 +0000 (23:22 -0700)]
checkpont more filetree stuff
Brian Warner [Fri, 19 Jan 2007 09:35:36 +0000 (02:35 -0700)]
snapshot filetree work: fix pyflakes complaints
Brian Warner [Fri, 19 Jan 2007 09:23:03 +0000 (02:23 -0700)]
snapshot filetree work: it's getting close
Brian Warner [Sat, 20 Jan 2007 03:05:33 +0000 (20:05 -0700)]
rename bin/allmydata to bin/allmydata-tahoe, so the package can co-exist with the regular (amdlib) package
Brian Warner [Fri, 19 Jan 2007 09:17:48 +0000 (02:17 -0700)]
download.py: fix IDownloader to take a URI
Brian Warner [Fri, 19 Jan 2007 09:17:24 +0000 (02:17 -0700)]
update roadmap
Brian Warner [Thu, 18 Jan 2007 00:00:47 +0000 (17:00 -0700)]
filetree.interfaces: remove not-really-code to improve coverage stats
Brian Warner [Thu, 18 Jan 2007 08:00:11 +0000 (01:00 -0700)]
figleaf.el: add code to auto-enable the annotation mode for all source files
Brian Warner [Wed, 17 Jan 2007 22:34:29 +0000 (15:34 -0700)]
improve test coverage a bit
Brian Warner [Wed, 17 Jan 2007 21:46:02 +0000 (14:46 -0700)]
webish: present real URI in directory listing, not an unnecessarily b2a'ed form
Brian Warner [Wed, 17 Jan 2007 20:54:38 +0000 (13:54 -0700)]
split filetree_new.py up into smaller pieces, in a new subpackage
Brian Warner [Wed, 17 Jan 2007 04:34:34 +0000 (21:34 -0700)]
hush pyflakes warnings
Brian Warner [Wed, 17 Jan 2007 04:29:59 +0000 (21:29 -0700)]
update URI format, include codec name
Brian Warner [Wed, 17 Jan 2007 04:02:00 +0000 (21:02 -0700)]
Makefile: use absolute paths when setting PYTHONPATH
Brian Warner [Wed, 17 Jan 2007 04:01:18 +0000 (21:01 -0700)]
webish: add PBURL to the all-peers table on the welcome page
Brian Warner [Wed, 17 Jan 2007 02:56:30 +0000 (19:56 -0700)]
encode_new.py: recent Foolscap accepts 'None' as a constraint
Brian Warner [Wed, 17 Jan 2007 02:55:53 +0000 (19:55 -0700)]
webish: more verifierid-to-uri transition
Brian Warner [Wed, 17 Jan 2007 02:43:13 +0000 (19:43 -0700)]
vdrive/webish: finish verifierid/uri transition
Brian Warner [Wed, 17 Jan 2007 02:42:28 +0000 (19:42 -0700)]
upload: fix typo in debug messages
Brian Warner [Wed, 17 Jan 2007 02:35:12 +0000 (19:35 -0700)]
upload: rearrange peer-selection code to be more readable, and fix a silly bug
Brian Warner [Wed, 17 Jan 2007 02:07:23 +0000 (19:07 -0700)]
oops, fix that NotEnoughPeersError instrumentation
Brian Warner [Wed, 17 Jan 2007 01:47:52 +0000 (18:47 -0700)]
add better error reporting to NotEnoughPeersError
Brian Warner [Tue, 16 Jan 2007 22:15:26 +0000 (15:15 -0700)]
change upload to push 2 shares instead of 3
Now that peers can talk to themselves, the 5-node system test won't fail
just because one of the shares was hosted on the downloader (and thus
inaccessible until recently). The 3-share push was there to avoid this
problem.
Brian Warner [Tue, 16 Jan 2007 22:12:49 +0000 (15:12 -0700)]
now that foolscap-0.0.7 accepts connect-to-self, allow peers to know about themselves. We now require foolscap-0.0.7
Brian Warner [Tue, 16 Jan 2007 04:31:10 +0000 (21:31 -0700)]
hush pyflakes warnings
Brian Warner [Tue, 16 Jan 2007 04:22:22 +0000 (21:22 -0700)]
rearrange encode/upload, add URIs, switch to ReplicatingEncoder
Added metadata to the bucket store, which is used to hold the share number
(but the bucket doesn't know that, it just gets a string).
Modified the codec interfaces a bit.
Try to pass around URIs to/from download/upload instead of verifierids.
URI format is still in flux.
Change the current (primitive) file encoder to use a ReplicatingEncoder
because it provides ICodecEncoder. We will be moving to the (less primitive)
file encoder (currently in allmydata.encode_new) eventually, but for now
this change lets us test out PyRS or zooko's upcoming C-based RS codec in
something larger than a single unit test. This primitive file encoder only
uses a single segment, and has no merkle trees.
Also added allmydata.util.deferredutil for a DeferredList wrapper that
errbacks (but only when all component Deferreds have fired) if there were
any errors, which unfortunately is not a behavior available from the standard
DeferredList.
Brian Warner [Mon, 15 Jan 2007 21:01:22 +0000 (14:01 -0700)]
add simple metadata (a single string) to the storage protocol
Brian Warner [Fri, 12 Jan 2007 03:57:14 +0000 (20:57 -0700)]
change IEncoder to ICodecEncoder, to match the previous change
Brian Warner [Fri, 12 Jan 2007 03:51:27 +0000 (20:51 -0700)]
use the word 'codec' for erasure coding, for now. 'encode' is used for file-level segmentation/hashing
Brian Warner [Wed, 10 Jan 2007 04:13:05 +0000 (21:13 -0700)]
sendOnly: oops, I keep forgetting that you can't really use f.trap in a lambda, because it returns the failure it trapped
Brian Warner [Wed, 10 Jan 2007 02:40:36 +0000 (19:40 -0700)]
test_system.py: exercise queen.Roster._lost_node too
Brian Warner [Tue, 9 Jan 2007 08:58:50 +0000 (01:58 -0700)]
workqueue: more tests
Brian Warner [Tue, 9 Jan 2007 08:36:12 +0000 (01:36 -0700)]
workqueue: more tests, coverage now at 63.4%, yay
Brian Warner [Tue, 9 Jan 2007 05:29:42 +0000 (22:29 -0700)]
workqueue: more improvements, more tests
Brian Warner [Tue, 9 Jan 2007 04:29:33 +0000 (21:29 -0700)]
workqueue: start adding tests
Brian Warner [Tue, 9 Jan 2007 04:29:03 +0000 (21:29 -0700)]
figleaf.el: when converting to the elisp-readable format, use our in-tree version of figleaf rather than one found on the system. Also change the keybinding to toggle annotations to C-cC-a, which is easier to type
Brian Warner [Tue, 9 Jan 2007 04:28:08 +0000 (21:28 -0700)]
figleaf: delete coverage file (.figleaf) before each test run, otherwise I suspect coverage data will accumulate from one test run to the next
Brian Warner [Mon, 8 Jan 2007 23:29:14 +0000 (16:29 -0700)]
rename the new filetable code to 'filetree', since robk astutely pointed out that 'table' is misleading and implies a flat list of files in a single directory
Brian Warner [Sun, 7 Jan 2007 20:10:15 +0000 (13:10 -0700)]
figleaf.el: add some emacs22 comments
Brian Warner [Sun, 7 Jan 2007 20:09:37 +0000 (13:09 -0700)]
Makefile: add a default target to just build the tree
Brian Warner [Sat, 6 Jan 2007 04:15:59 +0000 (21:15 -0700)]
fix pyflakes warning in debugshell, by providing a dummy value for 'app' that will be overwritten when the manhole connection is established
Brian Warner [Sat, 6 Jan 2007 04:12:26 +0000 (21:12 -0700)]
use absolute import of 'allmydata.Crypto' rather than a relative import of just 'Crypto', to make it clear that we're using our own form rather than relying upon the system version
Brian Warner [Sat, 6 Jan 2007 04:11:19 +0000 (21:11 -0700)]
workqueue: fix pyflakes warnings, code is still quite incomplete
Brian Warner [Sat, 6 Jan 2007 03:06:51 +0000 (20:06 -0700)]
debian: give incrementing version numbers to .deb packages, using timestamps
Brian Warner [Sat, 6 Jan 2007 01:53:22 +0000 (18:53 -0700)]
test_encode_share.py: fix some pyflakes warnings
Brian Warner [Sat, 6 Jan 2007 01:46:03 +0000 (18:46 -0700)]
add some (disabled) encoder benchmarking code
Brian Warner [Sat, 6 Jan 2007 01:12:45 +0000 (18:12 -0700)]
change Encoder to use py_ecc, now that it works
Brian Warner [Sat, 6 Jan 2007 01:12:04 +0000 (18:12 -0700)]
disable figleaf tracing during py_ecc, since it takes *forever*, especially on the slow buildslave
Brian Warner [Fri, 5 Jan 2007 07:48:42 +0000 (00:48 -0700)]
encode.py: add some timing comments
Brian Warner [Fri, 5 Jan 2007 07:42:52 +0000 (00:42 -0700)]
only run a single (short) py_ecc test on slave3, since it is so slow the tests timeout
Brian Warner [Fri, 5 Jan 2007 07:06:42 +0000 (00:06 -0700)]
encoding: fix the last py_ecc problem, tests pass now
Brian Warner [Fri, 5 Jan 2007 06:51:35 +0000 (23:51 -0700)]
change encode_new to use IEncoder
Brian Warner [Fri, 5 Jan 2007 06:50:21 +0000 (23:50 -0700)]
fix our use of py_ecc (set log2FieldSize=8 explicitly)
Brian Warner [Fri, 5 Jan 2007 04:52:51 +0000 (21:52 -0700)]
establish IEncoder/IDecoder, create suitable interfaces for both the simple replicating encoder and the py_ecc one, add a (failing) unit test for it
Brian Warner [Fri, 5 Jan 2007 04:52:40 +0000 (21:52 -0700)]
figleaf.el: announce annotation/deannotation
Brian Warner [Fri, 5 Jan 2007 01:06:20 +0000 (18:06 -0700)]
more pyflakes cleanups
Brian Warner [Fri, 5 Jan 2007 01:04:43 +0000 (18:04 -0700)]
fix pyflakes warnings in figleaf
Zooko O'Whielacronx [Fri, 5 Jan 2007 00:58:14 +0000 (17:58 -0700)]
add padding to segments and convert type from string to list of integers before passing to py_ecc
But then I get an error that I don't understand about ECC math...
Brian Warner [Thu, 4 Jan 2007 08:26:43 +0000 (01:26 -0700)]
improve figleaf: fix some line-numbering bugs, add an alphabetically-sorted index HTML page
Brian Warner [Thu, 4 Jan 2007 08:25:36 +0000 (01:25 -0700)]
more figleaf emacs work: enable the minor mode, then type C-cA to toggle annotations
Brian Warner [Thu, 4 Jan 2007 05:23:25 +0000 (22:23 -0700)]
modify figleaf2html to show module names instead of .py filenames, also add a --root argument to restrict coverage to a specific parent directory
Brian Warner [Thu, 4 Jan 2007 05:06:51 +0000 (22:06 -0700)]
move figleaf2html and figleaf_htmlizer.py into our tree, for easier customization
Brian Warner [Thu, 4 Jan 2007 04:38:29 +0000 (21:38 -0700)]
figleaf: move a copy into allmydata.util.figleaf, update Makefile/trial stuff
Brian Warner [Tue, 2 Jan 2007 06:48:42 +0000 (23:48 -0700)]
add some experimental emacs test-coverage-annotation tools, still in development
Brian Warner [Tue, 2 Jan 2007 06:47:16 +0000 (23:47 -0700)]
checkpoint work-in-progress for WorkQueue, a disk-persistent list of work to be done
Brian Warner [Tue, 2 Jan 2007 06:44:12 +0000 (23:44 -0700)]
filetable_new: import py_ecc, make the tests get a little bit further
Brian Warner [Tue, 2 Jan 2007 06:43:24 +0000 (23:43 -0700)]
add __init__ and setup.py glue for py_ecc, so we can import it
Zooko O'Whielacronx [Fri, 29 Dec 2006 20:50:53 +0000 (13:50 -0700)]
make encode_new use py_ecc for real live erasure coding
(This patch is not tested -- I'm working on a Mac which doesn't have gcc
installed... (gcc is necessary for the crypto module.) I will now attempt to
connect to a better development computer over my mom's staticy, failure-prone,
14.4 K modem...)
Zooko O'Whielacronx [Fri, 29 Dec 2006 20:50:42 +0000 (13:50 -0700)]
import mathutil from pyutil
Zooko O'Whielacronx [Fri, 29 Dec 2006 20:46:45 +0000 (13:46 -0700)]
import py_ecc, a pure python fec by Emin Martinian, which is under a permissive licence
It is too slow for a real product, but is a quick way to get a working prototype, and also is freely redistributable by us...
Zooko O'Whielacronx [Fri, 29 Dec 2006 19:40:10 +0000 (12:40 -0700)]
encode_new: fix a bug in subshare hashes and tidy-up a couple of things