From: zooko Date: Fri, 26 Jan 2007 00:46:37 +0000 (+0530) Subject: add README.txt X-Git-Url: https://git.rkrishnan.org/?a=commitdiff_plain;h=7c294f7411a2567cd5a4c5f329ee10e528b7424a;p=tahoe-lafs%2Fzfec.git add README.txt darcs-hash:688027fdbf8d8e4e3c2e691814f6052de81a64f4 --- diff --git a/pyfec/README.txt b/pyfec/README.txt new file mode 100644 index 0000000..e01310c --- /dev/null +++ b/pyfec/README.txt @@ -0,0 +1,63 @@ +This package provides an "erasure code", or "forward error correction code". +It is licensed under the GNU General Public License (see the COPYING file for +details). + +The most widely known example of an erasure code is the RAID-5 algorithm which +makes it so that in the event of the loss of any one hard drive, the stored +data can be completely recovered. The algorithm in the pyfec package has a +similar effect, but instead of recovering from the loss of any one element, it +can be parameterized to choose in advance the number of elements whose loss it +can recover from. + +This package is largely based on the old "fec" library by Luigi Rizzo et al., +which is a simple, fast, mature, and optimized implementation of erasure +coding. The pyfec package makes several changes from the original "fec" +package, including addition of the Python API, refactoring of the C API to be +faster (for the way that I use it, at least), and a few clean-ups and +micro-optimizations of the core code itself. + +This package performs two operations, encoding and decoding. Encoding takes +some input data and expands its size by producing extra "check blocks". +Decoding takes some blocks -- any combination of original blocks of data (also +called "primary shares") and check blocks (also called "secondary shares"), and +produces the original data. + +The encoding is parameterized by two integers, k and m. m is the total number +of shares produced, and k is how many of those shares are necessary to +reconstruct the original data. m is required to be at least 1 and at most 255, +and k is required to be at least 1 and at most m. (Note that when k == m then +there is no point in doing erasure coding.) + +Note that each "primary share" is a segment of the original data, so its size +is 1/k'th of the size of original data, and each "secondary share" is of the +same size, so the total space used by all the shares is about m/k times the +size of the original data. + +The decoding step requires as input k of the shares which were produced by the +encoding step. The decoding step produces as output the data that was earlier +input to the encoding step. + +This package also includes a Python interface. See the Python docstrings for +usage details. + +See also the filefec.py module which has a utility function for efficiently +reading a file and encoding it piece by piece. + +Beware of a "gotcha" that can result from the combination of mutable buffers +and the fact that pyfec never makes an unnecessary data copy. That is: +whenever one of the shares produced from a call to encode() or decode() has the +same contents as one of the shares passed as input, then pyfec will return as +output a pointer (in the C API) or a Python reference (in the Python API) to +the object which was passed to it as input. This is efficient as it avoids +making an unnecessary copy of the data. But if the object which was passed as +input is mutable and if that object is mutated after the call to pyfec returns, +then the result from pyfec -- which is just a reference to that same object -- +will also be mutated. This subtlety is the price you pay for avoiding data +copying. If you don't want to have to worry about this, then simply use +immutable objects (e.g. Python strings) to hold the data that you pass to +pyfec. + +Enjoy! + +Zooko Wilcox-O'Hearn +