1 User visible changes in Tahoe-LAFS. -*- outline; coding: utf-8 -*-
3 * Release 1.8.1 (2010-10-28)
5 ** Bugfixes and Improvements
7 - Allow the repairer to improve the health of a file by uploading
8 some shares, even if it cannot achieve the configured happiness
9 threshold. This fixes a regression introduced between v1.7.1 and
11 - Fix a memory leak in the ResponseCache which is used during mutable
12 file/directory operations. (#1045)
13 - Fix a regression and add a performance improvement in the downloader.
14 This issue caused repair to fail in some special cases. (#1223)
15 - Fix a bug that caused 'tahoe cp' to fail for a grid-to-grid copy
16 involving a non-ASCII filename. (#1224)
17 - Fix a rarely-encountered bug involving printing large strings to
18 the console on Windows. (#1232)
19 - Perform ~ expansion in the --exclude-from filename argument to
20 'tahoe backup'. (#1241)
21 - The CLI's 'tahoe mv' and 'tahoe ln' commands previously would try
22 to use an HTTP proxy if the HTTP_PROXY environment variable was set.
23 These now always connect directly to the WAPI, thus avoiding giving
24 caps to the HTTP proxy (and also avoiding failures in the case that
25 the proxy is failing or requires authentication). (#1253)
26 - The CLI now correctly reports failure in the case that 'tahoe mv'
27 fails to unlink the file from its old location. (#1255)
28 - 'tahoe start' now gives a more positive indication that the node
33 - Use a newer version of a dependency if needed, even if an older
34 version is installed. This would previously cause a VersionConflict
36 - Use a precompiled binary of a dependency if one with a sufficiently
37 high version number is available, instead of attempting to compile
38 the dependency from source, even if the source version has a higher
39 version number. (#1233)
43 - All current documentation in .txt format has been converted to
45 - Added docs/backdoors.rst declaring that we won't add backdoors to
46 Tahoe-LAFS, or add anything to facilitate government access to data.
49 * Release 1.8.0 (2010-09-23)
53 - A completely new downloader which improves performance and
54 robustness of immutable-file downloads. It uses the fastest K
55 servers to download the data in K-way parallel. It automatically
56 fails over to alternate servers if servers fail in mid-download. It
57 allows seeking to arbitrary locations in the file (the previous
58 downloader which would only read the entire file sequentially from
59 beginning to end). It minimizes unnecessary round trips and
60 unnecessary bytes transferred to improve performance. It sends
61 requests to fewer servers to reduce the load on servers (the
62 previous one would send a small request to every server for every
63 download) (#287, #288, #448, #798, #800, #990, #1170, #1191)
65 - Non-ASCII command-line arguments and non-ASCII outputs now work on
66 Windows. In addition, the command-line tool now works on 64-bit
69 ** Bugfixes and Improvements
71 - Document and clean up the command-line options for specifying the
72 node's base directory. (#188, #706, #715, #772, #1108)
73 - The default node directory for Windows is ".tahoe" in the user's
74 home directory, the same as on other platforms. (#890)
75 - Fix a case in which full cap URIs could be logged. (#685, #1155)
76 - Fix bug in WUI in Python 2.5 when the system clock is set back to
77 1969. Now you can use Tahoe-LAFS with Python 2.5 and set your
78 system clock to 1969 and still use the WUI. (#1055)
79 - Many improvements in code organization, tests, logging,
80 documentation, and packaging. (#983, #1074, #1108, #1127, #1129,
85 - on x86 and x86-64 platforms, pycryptopp >= 0.5.20
86 - pycrypto 2.2 is excluded due to a bug
88 * Release 1.7.1 (2010-07-18)
90 ** Bugfixes and Improvements
92 - Fix bug in which uploader could fail with AssertionFailure or
93 report that it had achieved servers-of-happiness when it
95 - Fix bug in which servers could get into a state where they would
96 refuse to accept shares of a certain file (#1117)
97 - Add init scripts for managing the gateway server on Debian/Ubuntu
99 - Fix bug where server version number was always 0 on the welcome
101 - Add new command-line command "tahoe unlink" as a synonym for "tahoe
103 - The FTP frontend now encrypts its temporary files, protecting their
104 contents from an attacker who is able to read the disk. (#1083)
105 - Fix IP address detection on FreeBSD 7, 8, and 9 (#1098)
106 - Fix minor layout issue in the Web User Interface with Internet
108 - Fix rarely-encountered incompatibility between Twisted logging
109 utility and the new unicode support added in v1.7.0 (#1099)
110 - Forward-compatibility improvements for non-ASCII caps (#1051)
114 - Simplify and tidy-up directories, unicode support, test code (#923, #967,
117 * Release 1.7.0 (2010-06-18)
123 Your Tahoe-LAFS gateway now acts like a full-fledged SFTP server. It has been
124 tested with sshfs to provide a virtual filesystem in Linux. Many users have
125 asked for this feature. We hope that it serves them well! See the
126 docs/frontends/FTP-and-SFTP.txt document to get started.
128 *** support for non-ASCII character encodings
130 Tahoe-LAFS now correctly handles filenames containing non-ASCII characters on
131 all supported platforms:
133 - when reading files in from the local filesystem (such as when you run "tahoe
134 backup" to back up your local files to a Tahoe-LAFS grid);
136 - when writing files out to the local filesystem (such as when you run "tahoe
137 cp -r" to recursively copy files out of a Tahoe-LAFS grid);
139 - when displaying filenames to the terminal (such as when you run "tahoe ls"),
140 subject to limitations of the terminal and locale;
142 - when parsing command-line arguments, except on Windows.
144 *** Servers of Happiness
146 Tahoe-LAFS now measures during immutable file upload to see how well
147 distributed it is across multiple servers. It aborts the upload if the pieces
148 of the file are not sufficiently well-distributed.
150 This behavior is controlled by a configuration parameter called "servers of
151 happiness". With the default settings for its erasure coding, Tahoe-LAFS
152 generates 10 shares for each file, such that any 3 of those shares are
153 sufficient to recover the file. The default value of "servers of happiness" is
154 7, which means that Tahoe-LAFS will guarantee that there are at least 7 servers
155 holding some of the shares, such that any 3 of those servers can completely
158 The new upload code also distributes the shares better than the previous
159 version in some cases and takes better advantage of pre-existing shares (when a
160 file has already been previously uploaded). See the architecture.txt document
163 ** Bugfixes and Improvements
165 - Premature abort of upload if some shares were already present and some
167 - python ./setup.py install -- can't create or remove files in install
169 - Network failure => internal TypeError. (#902)
170 - Install of Tahoe on CentOS 5.4. (#933)
171 - CLI option --node-url now supports https url. (#1028)
172 - HTML/CSS template files were not correctly installed under Windows. (#1033)
173 - MetadataSetter does not enforce restriction on setting "tahoe" subkeys.
175 - ImportError: No module named setuptools_darcs.setuptools_darcs. (#1054)
176 - Renamed Title in xhtml files. (#1062)
177 - Increase Python version dependency to 2.4.4, to avoid a critical CPython
178 security bug. (#1066)
179 - Typo correction for the munin plugin tahoe_storagespace. (#968)
180 - Fix warnings found by pylint. (#973)
181 - Changing format of some documentation files. (#1027)
182 - the misc/ directory was tied up. (#1068)
183 - The 'ctime' and 'mtime' metadata fields are no longer written except by
184 "tahoe backup". (#924)
185 - Unicode filenames in Tahoe-LAFS directories are normalized so that names
186 that differ only in how accents are encoded are treated as the same. (#1076)
187 - Various small improvements to documentation. (#937, #911, #1024, #1082)
191 The 'tahoe debug consolidate' subcommand (for converting old allmydata Windows
192 client backups to a newer format) has been removed.
194 ** Dependency Updates
196 the Python version dependency is raised to 2.4.4 in some cases (2.4.3 for
197 Redhat-based Linux distributions, 2.4.2 for UCS-2 builds) (#1066)
200 mock (only required by unit tests)
202 * Release 1.6.1 (2010-02-27)
206 *** Correct handling of Small Immutable Directories
208 Immutable directories can now be deep-checked and listed in the web UI in
209 all cases. (In v1.6.0, some operations, such as deep-check, on a directory
210 graph that included very small immutable directories, would result in an
211 exception causing the whole operation to abort.) (#948)
213 ** Usability Improvements
215 Improved user interface messages and error reporting. (#681, #837, #939)
217 The timeouts for operation handles have been greatly increased, so that
218 you can view the results of an operation up to 4 days after it has
219 completed. After viewing them for the first time, the results are
220 retained for a further day. (#577)
222 * Release 1.6.0 (2010-02-01)
226 *** Immutable Directories
228 Tahoe-LAFS can now create and handle immutable directories. (#607, #833, #931)
229 These are read just like normal directories, but are "deep-immutable", meaning
230 that all their children (and everything reachable from those children) must be
231 immutable objects (i.e. immutable or literal files, and other immutable
234 These directories must be created in a single webapi call that provides all
235 of the children at once. (Since they cannot be changed after creation, the
236 usual create/add/add sequence cannot be used.) They have URIs that start with
237 "URI:DIR2-CHK:" or "URI:DIR2-LIT:", and are described on the human-facing web
238 interface (aka the "WUI") with a "DIR-IMM" abbreviation (as opposed to "DIR"
239 for the usual read-write directories and "DIR-RO" for read-only directories).
241 Tahoe-LAFS releases before 1.6.0 cannot read the contents of an immutable
242 directory. 1.5.0 will tolerate their presence in a directory listing (and
243 display it as "unknown"). 1.4.1 and earlier cannot tolerate them: a DIR-IMM
244 child in any directory will prevent the listing of that directory.
246 Immutable directories are repairable, just like normal immutable files.
248 The webapi "POST t=mkdir-immutable" call is used to create immutable
249 directories. See docs/frontends/webapi.txt for details.
251 *** "tahoe backup" now creates immutable directories, backupdb has dircache
253 The "tahoe backup" command has been enhanced to create immutable directories
254 (in previous releases, it created read-only mutable directories) (#828). This
255 is significantly faster, since it does not need to create an RSA keypair for
256 each new directory. Also "DIR-IMM" immutable directories are repairable, unlike
257 "DIR-RO" read-only mutable directories at present. (A future Tahoe-LAFS release
258 should also be able to repair DIR-RO.)
260 In addition, the backupdb (used by "tahoe backup" to remember what it has
261 already copied) has been enhanced to store information about existing immutable
262 directories. This allows it to re-use directories that have moved but still
263 contain identical contents, or that have been deleted and later replaced. (The
264 1.5.0 "tahoe backup" command could only re-use directories that were in the
265 same place as they were in the immediately previous backup.) With this change,
266 the backup process no longer needs to read the previous snapshot out of the
267 Tahoe-LAFS grid, reducing the network load considerably. (#606)
269 A "null backup" (in which nothing has changed since the previous backup) will
270 require only two Tahoe-side operations: one to add an Archives/$TIMESTAMP
271 entry, and a second to update the Latest/ link. On the local disk side, it
272 will readdir() all your local directories and stat() all your local files.
274 If you've been using "tahoe backup" for a while, you will notice that your
275 first use of it after upgrading to 1.6.0 may take a long time: it must create
276 proper immutable versions of all the old read-only mutable directories. This
277 process won't take as long as the initial backup (where all the file contents
278 had to be uploaded too): it will require time proportional to the number and
279 size of your directories. After this initial pass, all subsequent passes
280 should take a tiny fraction of the time.
282 As noted above, Tahoe-LAFS versions earlier than 1.5.0 cannot list a directory
283 containing an immutable subdirectory. Tahoe-LAFS versions earlier than 1.6.0
284 cannot read the contents of an immutable directory.
286 The "tahoe backup" command has been improved to skip over unreadable objects
287 (like device files, named pipes, and files with permissions that prevent the
288 command from reading their contents), instead of throwing an exception and
289 terminating the backup process. It also skips over symlinks, because these
290 cannot be represented faithfully in the Tahoe-side filesystem. A warning
291 message will be emitted each time something is skipped. (#729, #850, #641)
293 *** "create-node" command added, "create-client" now implies --no-storage
295 The basic idea behind Tahoe-LAFS's client+server and client-only processes is
296 that you are creating a general-purpose Tahoe-LAFS "node" process, which has
297 several components that can be activated. Storage service is one of these
298 optional components, as is the Helper, FTP server, and SFTP server. Web gateway
299 functionality is nominally on this list, but it is always active; a future
300 release will make it optional. There are three special purpose servers that
301 can't currently be run as a component in a node: introducer, key-generator,
304 So now "tahoe create-node" will create a Tahoe-LAFS node process, and after
305 creation you can edit its tahoe.cfg to enable or disable the desired
306 services. It is a more general-purpose replacement for "tahoe create-client".
307 The default configuration has storage service enabled. For convenience, the
308 "--no-storage" argument makes a tahoe.cfg file that disables storage
311 "tahoe create-client" has been changed to create a Tahoe-LAFS node without a
312 storage service. It is equivalent to "tahoe create-node --no-storage". This
313 helps to reduce the confusion surrounding the use of a command with "client" in
314 its name to create a storage *server*. Use "tahoe create-client" to create a
315 purely client-side node. If you want to offer storage to the grid, use
316 "tahoe create-node" instead.
318 In the future, other services will be added to the node, and they will be
319 controlled through options in tahoe.cfg . The most important of these
320 services may get additional --enable-XYZ or --disable-XYZ arguments to
323 ** Performance Improvements
325 Download of immutable files begins as soon as the downloader has located the K
326 necessary shares (#928, #287). In both the previous and current releases, a
327 downloader will first issue queries to all storage servers on the grid to
328 locate shares before it begins downloading the shares. In previous releases of
329 Tahoe-LAFS, download would not begin until all storage servers on the grid had
330 replied to the query, at which point K shares would be chosen for download from
331 among the shares that were located. In this release, download begins as soon as
332 any K shares are located. This means that downloads start sooner, which is
333 particularly important if there is a server on the grid that is extremely slow
334 or even hung in such a way that it will never respond. In previous releases
335 such a server would have a negative impact on all downloads from that grid. In
336 this release, such a server will have no impact on downloads, as long as K
337 shares can be found on other, quicker, servers. This also means that
338 downloads now use the "best-alacrity" servers that they talk to, as measured by
339 how quickly the servers reply to the initial query. This might cause downloads
340 to go faster, especially on grids with heterogeneous servers or geographical
345 The webapi acquired a new "t=mkdir-with-children" command, to create and
346 populate a directory in a single call. This is significantly faster than
347 using separate "t=mkdir" and "t=set-children" operations (it uses one
348 gateway-to-grid roundtrip, instead of three or four). (#533)
350 The t=set-children (note the hyphen) operation is now documented in
351 docs/frontends/webapi.txt, and is the new preferred spelling of the old
352 t=set_children (with an underscore). The underscore version remains for
353 backwards compatibility. (#381, #927)
355 The tracebacks produced by errors in CLI tools should now be in plain text,
356 instead of HTML (which is unreadable outside of a browser). (#646)
358 The [storage]reserved_space configuration knob (which causes the storage
359 server to refuse shares when available disk space drops below a threshold)
360 should work on Windows now, not just UNIX. (#637)
362 "tahoe cp" should now exit with status "1" if it cannot figure out a suitable
363 target filename, such as when you copy from a bare filecap. (#761)
365 "tahoe get" no longer creates a zero-length file upon error. (#121)
367 "tahoe ls" can now list single files. (#457)
369 "tahoe deep-check --repair" should tolerate repair failures now, instead of
370 halting traversal. (#874, #786)
372 "tahoe create-alias" no longer corrupts the aliases file if it had
373 previously been edited to have no trailing newline. (#741)
375 Many small packaging improvements were made to facilitate the "tahoe-lafs"
376 package being included in Ubuntu. Several mac/win32 binary libraries were
377 removed, some figleaf code-coverage files were removed, a bundled copy of
378 darcsver-1.2.1 was removed, and additional licensing text was added.
380 Several DeprecationWarnings for python2.6 were silenced. (#859)
382 The checker --add-lease option would sometimes fail for shares stored
383 on old (Tahoe v1.2.0) servers. (#875)
385 The documentation for installing on Windows (docs/install.html) has been
388 For other changes not mentioned here, see
389 <http://allmydata.org/trac/tahoe/query?milestone=1.6.0&keywords=!~news-done>.
390 To include the tickets mentioned above, go to
391 <http://allmydata.org/trac/tahoe/query?milestone=1.6.0>.
394 * Release 1.5.0 (2009-08-01)
398 Uploads of immutable files now use pipelined writes, improving upload speed
399 slightly (10%) over high-latency connections. (#392)
401 Processing large directories has been sped up, by removing a O(N^2) algorithm
402 from the dirnode decoding path and retaining unmodified encrypted entries.
405 The human-facing web interface (aka the "WUI") received a significant CSS
406 makeover by Kevin Reid, making it much prettier and easier to read. The WUI
407 "check" and "deep-check" forms now include a "Renew Lease" checkbox,
408 mirroring the CLI --add-lease option, so leases can be added or renewed from
411 The CLI "tahoe mv" command now refuses to overwrite directories. (#705)
413 The CLI "tahoe webopen" command, when run without arguments, will now bring
414 up the "Welcome Page" (node status and mkdir/upload forms).
416 The 3.5MB limit on mutable files was removed, so it should be possible to
417 upload arbitrarily-sized mutable files. Note, however, that the data format
418 and algorithm remains the same, so using mutable files still requires
419 bandwidth, computation, and RAM in proportion to the size of the mutable file.
422 This version of Tahoe-LAFS will tolerate directory entries that contain filecap
423 formats which it does not recognize: files and directories from the future.
424 This should improve the user experience (for 1.5.0 users) when we add new cap
425 formats in the future. Previous versions would fail badly, preventing the user
426 from seeing or editing anything else in those directories. These unrecognized
427 objects can be renamed and deleted, but obviously not read or written. Also
428 they cannot generally be copied. (#683)
432 deep-check-and-repair now tolerates read-only directories, such as the ones
433 produced by the "tahoe backup" CLI command. Read-only directories and mutable
434 files are checked, but not repaired. Previous versions threw an exception
435 when attempting the repair and failed to process the remaining contents. We
436 cannot yet repair these read-only objects, but at least this version allows
437 the rest of the check+repair to proceed. (#625)
439 A bug in 1.4.1 which caused a server to be listed multiple times (and
440 frequently broke all connections to that server) was fixed. (#653)
442 The plaintext-hashing code was removed from the Helper interface, removing
443 the Helper's ability to mount a partial-information-guessing attack. (#722)
445 ** Platform/packaging changes
447 Tahoe-LAFS now runs on NetBSD, OpenBSD, ArchLinux, and NixOS, and on an
448 embedded system based on an ARM CPU running at 266 MHz.
450 Unit test timeouts have been raised to allow the tests to complete on
451 extremely slow platforms like embedded ARM-based NAS boxes, which may take
452 several hours to run the test suite. An ARM-specific data-corrupting bug in
453 an older version of Crypto++ (5.5.2) was identified: ARM-users are encouraged
454 to use recent Crypto++/pycryptopp which avoids this problem.
456 Tahoe-LAFS now requires a SQLite library, either the sqlite3 that comes
457 built-in with python2.5/2.6, or the add-on pysqlite2 if you're using
458 python2.4. In the previous release, this was only needed for the "tahoe backup"
459 command: now it is mandatory.
461 Several minor documentation updates were made.
463 To help get Tahoe-LAFS into Linux distributions like Fedora and Debian,
464 packaging improvements are being made in both Tahoe-LAFS and related libraries
465 like pycryptopp and zfec.
467 The Crypto++ library included in the pycryptopp package has been upgraded to
468 version 5.6.0 of Crypto++, which includes a more efficient implementation of
469 SHA-256 in assembly for x86 or amd64 architectures.
471 ** dependency updates
474 no python-2.4.0 or 2.4.1 (2.4.2 is good)
475 (they contained a bug in base64.b32decode)
476 avoid python-2.6 on windows with mingw: compiler issues
477 python2.4 requires pysqlite2 (2.5,2.6 does not)
482 * Release 1.4.1 (2009-04-13)
484 ** Garbage Collection
486 The big feature for this release is the implementation of garbage collection,
487 allowing Tahoe storage servers to delete shares for old deleted files. When
488 enabled, this uses a "mark and sweep" process: clients are responsible for
489 updating the leases on their shares (generally by running "tahoe deep-check
490 --add-lease"), and servers are allowed to delete any share which does not
491 have an up-to-date lease. The process is described in detail in
492 docs/garbage-collection.txt .
494 The server must be configured to enable garbage-collection, by adding
495 directives to the [storage] section that define an age limit for shares. The
496 default configuration will not delete any shares.
498 Both servers and clients should be upgraded to this release to make the
499 garbage-collection as pleasant as possible. 1.2.0 servers have code to
500 perform the update-lease operation but it suffers from a fatal bug, while
501 1.3.0 servers have update-lease but will return an exception for unknown
502 storage indices, causing clients to emit an Incident for each exception,
503 slowing the add-lease process down to a crawl. 1.1.0 servers did not have the
504 add-lease operation at all.
506 ** Security/Usability Problems Fixed
508 A super-linear algorithm in the Merkle Tree code was fixed, which previously
509 caused e.g. download of a 10GB file to take several hours before the first
510 byte of plaintext could be produced. The new "alacrity" is about 2 minutes. A
511 future release should reduce this to a few seconds by fixing ticket #442.
513 The previous version permitted a small timing attack (due to our use of
514 strcmp) against the write-enabler and lease-renewal/cancel secrets. An
515 attacker who could measure response-time variations of approximatly 3ns
516 against a very noisy background time of about 15ms might be able to guess
517 these secrets. We do not believe this attack was actually feasible. This
518 release closes the attack by first hashing the two strings to be compared
519 with a random secret.
523 In most cases, HTML tracebacks will only be sent if an "Accept: text/html"
524 header was provided with the HTTP request. This will generally cause browsers
525 to get an HTMLized traceback but send regular text/plain tracebacks to
526 non-browsers (like the CLI clients). More errors have been mapped to useful
529 The streaming webapi operations (deep-check and manifest) now have a way to
530 indicate errors (an output line that starts with "ERROR" instead of being
531 legal JSON). See docs/frontends/webapi.txt for details.
533 The storage server now has its own status page (at /storage), linked from the
534 Welcome page. This page shows progress and results of the two new
535 share-crawlers: one which merely counts shares (to give an estimate of how
536 many files/directories are being stored in the grid), the other examines
537 leases and reports how much space would be freed if GC were enabled. The page
538 also shows how much disk space is present, used, reserved, and available for
539 the Tahoe server, and whether the server is currently running in "read-write"
540 mode or "read-only" mode.
542 When a directory node cannot be read (perhaps because of insufficent shares),
543 a minimal webapi page is created so that the "more-info" links (including a
544 Check/Repair operation) will still be accessible.
546 A new "reliability" page was added, with the beginnings of work on a
547 statistical loss model. You can tell this page how many servers you are using
548 and their independent failure probabilities, and it will tell you the
549 likelihood that an arbitrary file will survive each repair period. The
550 "numpy" package must be installed to access this page. A partial paper,
551 written by Shawn Willden, has been added to docs/proposed/lossmodel.lyx .
555 "tahoe check" and "tahoe deep-check" now accept an "--add-lease" argument, to
556 update a lease on all shares. This is the "mark" side of garbage collection.
558 In many cases, CLI error messages have been improved: the ugly HTMLized
559 traceback has been replaced by a normal python traceback.
561 "tahoe deep-check" and "tahoe manifest" now have better error reporting.
562 "tahoe cp" is now non-verbose by default.
564 "tahoe backup" now accepts several "--exclude" arguments, to ignore certain
565 files (like editor temporary files and version-control metadata) during
568 On windows, the CLI now accepts local paths like "c:\dir\file.txt", which
569 previously was interpreted as a Tahoe path using a "c:" alias.
571 The "tahoe restart" command now uses "--force" by default (meaning it will
572 start a node even if it didn't look like there was one already running).
574 The "tahoe debug consolidate" command was added. This takes a series of
575 independent timestamped snapshot directories (such as those created by the
576 allmydata.com windows backup program, or a series of "tahoe cp -r" commands)
577 and creates new snapshots that used shared read-only directories whenever
578 possible (like the output of "tahoe backup"). In the most common case (when
579 the snapshots are fairly similar), the result will use significantly fewer
580 directories than the original, allowing "deep-check" and similar tools to run
581 much faster. In some cases, the speedup can be an order of magnitude or more.
582 This tool is still somewhat experimental, and only needs to be run on large
583 backups produced by something other than "tahoe backup", so it was placed
584 under the "debug" category.
586 "tahoe cp -r --caps-only tahoe:dir localdir" is a diagnostic tool which,
587 instead of copying the full contents of files into the local directory,
588 merely copies their filecaps. This can be used to verify the results of a
589 "consolidation" operation.
593 The codebase no longer rauses RuntimeError as a kind of assert(). Specific
594 exception classes were created for each previous instance of RuntimeError.
596 Many unit tests were changed to use a non-network test harness, speeding them
599 Deep-traversal operations (manifest and deep-check) now walk individual
600 directories in alphabetical order. Occasional turn breaks are inserted to
601 prevent a stack overflow when traversing directories with hundreds of
604 The experimental SFTP server had its path-handling logic changed slightly, to
605 accomodate more SFTP clients, although there are still issues (#645).
608 * Release 1.3.0 (2009-02-13)
610 ** Checker/Verifier/Repairer
612 The primary focus of this release has been writing a checker / verifier /
613 repairer for files and directories. "Checking" is the act of asking storage
614 servers whether they have a share for the given file or directory: if there
615 are not enough shares available, the file or directory will be
616 unrecoverable. "Verifying" is the act of downloading and cryptographically
617 asserting that the server's share is undamaged: it requires more work
618 (bandwidth and CPU) than checking, but can catch problems that simple
619 checking cannot. "Repair" is the act of replacing missing or damaged shares
622 This release includes a full checker, a partial verifier, and a partial
623 repairer. The repairer is able to handle missing shares: new shares are
624 generated and uploaded to make up for the missing ones. This is currently the
625 best application of the repairer: to replace shares that were lost because of
626 server departure or permanent drive failure.
628 The repairer in this release is somewhat able to handle corrupted shares. The
631 * Immutable verifier is incomplete: not all shares are used, and not all
632 fields of those shares are verified. Therefore the immutable verifier has
633 only a moderate chance of detecting corrupted shares.
634 * The mutable verifier is mostly complete: all shares are examined, and most
635 fields of the shares are validated.
636 * The storage server protocol offers no way for the repairer to replace or
637 delete immutable shares. If corruption is detected, the repairer will
638 upload replacement shares to other servers, but the corrupted shares will
640 * read-only directories and read-only mutable files must be repaired by
641 someone who holds the write-cap: the read-cap is insufficient. Moreover,
642 the deep-check-and-repair operation will halt with an error if it attempts
643 to repair one of these read-only objects.
644 * Some forms of corruption can cause both download and repair operations to
645 fail. A future release will fix this, since download should be tolerant of
646 any corruption as long as there are at least 'k' valid shares, and repair
647 should be able to fix any file that is downloadable.
649 If the downloader, verifier, or repairer detects share corruption, the
650 servers which provided the bad shares will be notified (via a file placed in
651 the BASEDIR/storage/corruption-advisories directory) so their operators can
652 manually delete the corrupted shares and investigate the problem. In
653 addition, the "incident gatherer" mechanism will automatically report share
654 corruption to an incident gatherer service, if one is configured. Note that
655 corrupted shares indicate hardware failures, serious software bugs, or malice
656 on the part of the storage server operator, so a corrupted share should be
657 considered highly unusual.
659 By periodically checking/repairing all files and directories, objects in the
660 Tahoe filesystem remain resistant to recoverability failures due to missing
661 and/or broken servers.
663 This release includes a wapi mechanism to initiate checks on individual
664 files and directories (with or without verification, and with or without
665 automatic repair). A related mechanism is used to initiate a "deep-check" on
666 a directory: recursively traversing the directory and its children, checking
667 (and/or verifying/repairing) everything underneath. Both mechanisms can be
668 run with an "output=JSON" argument, to obtain machine-readable check/repair
669 status results. These results include a copy of the filesystem statistics
670 from the "deep-stats" operation (including total number of files, size
671 histogram, etc). If repair is possible, a "Repair" button will appear on the
674 The client web interface now features some extra buttons to initiate check
675 and deep-check operations. When these operations finish, they display a
676 results page that summarizes any problems that were encountered. All
677 long-running deep-traversal operations, including deep-check, use a
678 start-and-poll mechanism, to avoid depending upon a single long-lived HTTP
679 connection. docs/frontends/webapi.txt has details.
683 The "tahoe backup" command is new in this release, which creates efficient
684 versioned backups of a local directory. Given a local pathname and a target
685 Tahoe directory, this will create a read-only snapshot of the local directory
686 in $target/Archives/$timestamp. It will also create $target/Latest, which is
687 a reference to the latest such snapshot. Each time you run "tahoe backup"
688 with the same source and target, a new $timestamp snapshot will be added.
689 These snapshots will share directories that have not changed since the last
690 backup, to speed up the process and minimize storage requirements. In
691 addition, a small database is used to keep track of which local files have
692 been uploaded already, to avoid uploading them a second time. This
693 drastically reduces the work needed to do a "null backup" (when nothing has
694 changed locally), making "tahoe backup' suitable to run from a daily cronjob.
696 Note that the "tahoe backup" CLI command must be used in conjunction with a
697 1.3.0-or-newer Tahoe client node; there was a bug in the 1.2.0 webapi
698 implementation that would prevent the last step (create $target/Latest) from
703 The 12GiB (approximate) immutable-file-size limitation is lifted. This
704 release knows how to handle so-called "v2 immutable shares", which permit
705 immutable files of up to about 18 EiB (about 3*10^14). These v2 shares are
706 created if the file to be uploaded is too large to fit into v1 shares. v1
707 shares are created if the file is small enough to fit into them, so that
708 files created with tahoe-1.3.0 can still be read by earlier versions if they
709 are not too large. Note that storage servers also had to be changed to
710 support larger files, and this release is the first release in which they are
711 able to do that. Clients will detect which servers are capable of supporting
712 large files on upload and will not attempt to upload shares of a large file
713 to a server which doesn't support it.
717 Tahoe now includes experimental FTP and SFTP servers. When configured with a
718 suitable method to translate username+password into a root directory cap, it
719 provides simple access to the virtual filesystem. Remember that FTP is
720 completely unencrypted: passwords, filenames, and file contents are all sent
721 over the wire in cleartext, so FTP should only be used on a local (127.0.0.1)
722 connection. This feature is still in development: there are no unit tests
723 yet, and behavior with respect to Unicode filenames is uncertain. Please see
724 docs/frontends/FTP-and-SFTP.txt for configuration details. (#512, #531)
728 This release adds the 'tahoe create-alias' command, which is a combination of
729 'tahoe mkdir' and 'tahoe add-alias'. This also allows you to start using a
730 new tahoe directory without exposing its URI in the argv list, which is
731 publicly visible (through the process table) on most unix systems. Thanks to
732 Kevin Reid for bringing this issue to our attention.
734 The single-argument form of "tahoe put" was changed to create an unlinked
735 file. I.e. "tahoe put bar.txt" will take the contents of a local "bar.txt"
736 file, upload them to the grid, and print the resulting read-cap; the file
737 will not be attached to any directories. This seemed a bit more useful than
738 the previous behavior (copy stdin, upload to the grid, attach the resulting
739 file into your default tahoe: alias in a child named 'bar.txt').
741 "tahoe put" was also fixed to handle mutable files correctly: "tahoe put
742 bar.txt URI:SSK:..." will read the contents of the local bar.txt and use them
743 to replace the contents of the given mutable file.
745 The "tahoe webopen" command was modified to accept aliases. This means "tahoe
746 webopen tahoe:" will cause your web browser to open to a "wui" page that
747 gives access to the directory associated with the default "tahoe:" alias. It
748 should also accept leading slashes, like "tahoe webopen tahoe:/stuff".
750 Many esoteric debugging commands were moved down into a "debug" subcommand:
753 tahoe debug dump-share
754 tahoe debug find-shares
755 tahoe debug catalog-shares
756 tahoe debug corrupt-share
758 The last command ("tahoe debug corrupt-share") flips a random bit of the
759 given local sharefile. This is used to test the file verifying/repairing
760 code, and obviously should not be used on user data.
762 The cli might not correctly handle arguments which contain non-ascii
763 characters in Tahoe v1.3 (although depending on your platform it
764 might, especially if your platform can be configured to pass such
765 characters on the command-line in utf-8 encoding). See
766 http://allmydata.org/trac/tahoe/ticket/565 for details.
770 The "default webapi port", used when creating a new client node (and in the
771 getting-started documentation), was changed from 8123 to 3456, to reduce
772 confusion when Tahoe accessed through a Firefox browser on which the
773 "Torbutton" extension has been installed. Port 8123 is occasionally used as a
774 Tor control port, so Torbutton adds 8123 to Firefox's list of "banned ports"
775 to avoid CSRF attacks against Tor. Once 8123 is banned, it is difficult to
776 diagnose why you can no longer reach a Tahoe node, so the Tahoe default was
777 changed. Note that 3456 is reserved by IANA for the "vat" protocol, but there
778 are argueably more Torbutton+Tahoe users than vat users these days. Note that
779 this will only affect newly-created client nodes. Pre-existing client nodes,
780 created by earlier versions of tahoe, may still be listening on 8123.
782 All deep-traversal operations (start-manifest, start-deep-size,
783 start-deep-stats, start-deep-check) now use a start-and-poll approach,
784 instead of using a single (fragile) long-running synchronous HTTP connection.
785 All these "start-" operations use POST instead of GET. The old "GET
786 manifest", "GET deep-size", and "POST deep-check" operations have been
789 The new "POST start-manifest" operation, when it finally completes, results
790 in a table of (path,cap), instead of the list of verifycaps produced by the
791 old "GET manifest". The table is available in several formats: use
792 output=html, output=text, or output=json to choose one. The JSON output also
793 includes stats, and a list of verifycaps and storage-index strings.
795 The "return_to=" and "when_done=" arguments have been removed from the
796 t=check and deep-check operations.
798 The top-level status page (/status) now has a machine-readable form, via
799 "/status/?t=json". This includes information about the currently-active
800 uploads and downloads, which may be useful for frontends that wish to display
801 progress information. There is no easy way to correlate the activities
802 displayed here with recent wapi requests, however.
804 Any files in BASEDIR/public_html/ (configurable) will be served in response
805 to requests in the /static/ portion of the URL space. This will simplify the
806 deployment of javascript-based frontends that can still access wapi calls
807 by conforming to the (regrettable) "same-origin policy".
809 The welcome page now has a "Report Incident" button, which is tied into the
810 "Incident Gatherer" machinery. If the node is attached to an incident
811 gatherer (via log_gatherer.furl), then pushing this button will cause an
812 Incident to be signalled: this means recent log events are aggregated and
813 sent in a bundle to the gatherer. The user can push this button after
814 something strange takes place (and they can provide a short message to go
815 along with it), and the relevant data will be delivered to a centralized
816 incident-gatherer for later processing by operations staff.
818 The "HEAD" method should now work correctly, in addition to the usual "GET",
819 "PUT", and "POST" methods. "HEAD" is supposed to return exactly the same
820 headers as "GET" would, but without any of the actual response body data. For
821 mutable files, this now does a brief mapupdate (to figure out the size of the
822 file that would be returned), without actually retrieving the file's
825 The "GET" operation on files can now support the HTTP "Range:" header,
826 allowing requests for partial content. This allows certain media players to
827 correctly stream audio and movies out of a Tahoe grid. The current
828 implementation uses a disk-based cache in BASEDIR/private/cache/download ,
829 which holds the plaintext of the files being downloaded. Future
830 implementations might not use this cache. GET for immutable files now returns
833 Each file and directory now has a "Show More Info" web page, which contains
834 much of the information that was crammed into the directory page before. This
835 includes readonly URIs, storage index strings, object type, buttons to
836 control checking/verifying/repairing, and deep-check/deep-stats buttons (for
837 directories). For mutable files, the "replace contents" upload form has been
838 moved here too. As a result, the directory page is now much simpler and
839 cleaner, and several potentially-misleading links (like t=uri) are now gone.
841 Slashes are discouraged in Tahoe file/directory names, since they cause
842 problems when accessing the filesystem through the wapi. However, there are
843 a couple of accidental ways to generate such names. This release tries to
844 make it easier to correct such mistakes by escaping slashes in several
845 places, allowing slashes in the t=info and t=delete commands, and in the
846 source (but not the target) of a t=rename command.
850 Tahoe's dependencies have been extended to require the "[secure_connections]"
851 feature from Foolscap, which will cause pyOpenSSL to be required and/or
852 installed. If OpenSSL and its development headers are already installed on
853 your system, this can occur automatically. Tahoe now uses pollreactor
854 (instead of the default selectreactor) to work around a bug between pyOpenSSL
855 and the most recent release of Twisted (8.1.0). This bug only affects unit
856 tests (hang during shutdown), and should not impact regular use.
858 The Tahoe source code tarballs now come in two different forms: regular and
859 "sumo". The regular tarball contains just Tahoe, nothing else. When building
860 from the regular tarball, the build process will download any unmet
861 dependencies from the internet (starting with the index at PyPI) so it can
862 build and install them. The "sumo" tarball contains copies of all the
863 libraries that Tahoe requires (foolscap, twisted, zfec, etc), so using the
864 "sumo" tarball should not require any internet access during the build
865 process. This can be useful if you want to build Tahoe while on an airplane,
866 a desert island, or other bandwidth-limited environments.
868 Similarly, allmydata.org now hosts a "tahoe-deps" tarball which contains the
869 latest versions of all these dependencies. This tarball, located at
870 http://allmydata.org/source/tahoe/deps/tahoe-deps.tar.gz, can be unpacked in
871 the tahoe source tree (or in its parent directory), and the build process
872 should satisfy its downloading needs from it instead of reaching out to PyPI.
873 This can be useful if you want to build Tahoe from a darcs checkout while on
874 that airplane or desert island.
876 Because of the previous two changes ("sumo" tarballs and the "tahoe-deps"
877 bundle), most of the files have been removed from misc/dependencies/ . This
878 brings the regular Tahoe tarball down to 2MB (compressed), and the darcs
879 checkout (without history) to about 7.6MB. A full darcs checkout will still
880 be fairly large (because of the historical patches which included the
881 dependent libraries), but a 'lazy' one should now be small.
883 The default "make" target is now an alias for "setup.py build", which itself
884 is an alias for "setup.py develop --prefix support", with some extra work
885 before and after (see setup.cfg). Most of the complicated platform-dependent
886 code in the Makefile was rewritten in Python and moved into setup.py,
887 simplifying things considerably.
889 Likewise, the "make test" target now delegates most of its work to "setup.py
890 test", which takes care of getting PYTHONPATH configured to access the tahoe
891 code (and dependencies) that gets put in support/lib/ by the build_tahoe
892 step. This should allow unit tests to be run even when trial (which is part
893 of Twisted) wasn't already installed (in this case, trial gets installed to
894 support/bin because Twisted is a dependency of Tahoe).
896 Tahoe is now compatible with the recently-released Python 2.6 , although it
897 is recommended to use Tahoe on Python 2.5, on which it has received more
898 thorough testing and deployment.
900 Tahoe is now compatible with simplejson-2.0.x . The previous release assumed
901 that simplejson.loads always returned unicode strings, which is no longer the
904 ** Grid Management Tools
906 Several tools have been added or updated in the misc/ directory, mostly munin
907 plugins that can be used to monitor a storage grid.
909 The misc/spacetime/ directory contains a "disk watcher" daemon (startable
910 with 'tahoe start'), which can be configured with a set of HTTP URLs
911 (pointing at the wapi '/statistics' page of a bunch of storage servers),
912 and will periodically fetch disk-used/disk-available information from all the
913 servers. It keeps this information in an Axiom database (a sqlite-based
914 library available from divmod.org). The daemon computes time-averaged rates
915 of disk usage, as well as a prediction of how much time is left before the
916 grid is completely full.
918 The misc/munin/ directory contains a new set of munin plugins
919 (tahoe_diskleft, tahoe_diskusage, tahoe_doomsday) which talk to the
920 disk-watcher and provide graphs of its calculations.
922 To support the disk-watcher, the Tahoe statistics component (visible through
923 the wapi at the /statistics/ URL) now includes disk-used and disk-available
924 information. Both are derived through an equivalent of the unix 'df' command
925 (i.e. they ask the kernel for the number of free blocks on the partition that
926 encloses the BASEDIR/storage directory). In the future, the disk-available
927 number will be further influenced by the local storage policy: if that policy
928 says that the server should refuse new shares when less than 5GB is left on
929 the partition, then "disk-available" will report zero even though the kernel
932 The 'tahoe_overhead' munin plugin interacts with an allmydata.com-specific
933 server which reports the total of the 'deep-size' reports for all active user
934 accounts, compares this with the disk-watcher data, to report on overhead
935 percentages. This provides information on how much space could be recovered
936 once Tahoe implements some form of garbage collection.
938 ** Configuration Changes: single INI-format tahoe.cfg file
940 The Tahoe node is now configured with a single INI-format file, named
941 "tahoe.cfg", in the node's base directory. Most of the previous
942 multiple-separate-files are still read for backwards compatibility (the
943 embedded SSH debug server and the advertised_ip_addresses files are the
944 exceptions), but new directives will only be added to tahoe.cfg . The "tahoe
945 create-client" command will create a tahoe.cfg for you, with sample values
946 commented out. (ticket #518)
948 tahoe.cfg now has controls for the foolscap "keepalive" and "disconnect"
951 tahoe.cfg now has controls for the encoding parameters: "shares.needed" and
952 "shares.total" in the "[client]" section. The default parameters are still
955 The inefficient storage 'sizelimit' control (which established an upper bound
956 on the amount of space that a storage server is allowed to consume) has been
957 replaced by a lightweight 'reserved_space' control (which establishes a lower
958 bound on the amount of remaining space). The storage server will reject all
959 writes that would cause the remaining disk space (as measured by a '/bin/df'
960 equivalent) to drop below this value. The "[storage]reserved_space="
961 tahoe.cfg parameter controls this setting. (note that this only affects
962 immutable shares: it is an outstanding bug that reserved_space does not
963 prevent the allocation of new mutable shares, nor does it prevent the growth
964 of existing mutable shares).
968 Clients now declare which versions of the protocols they support. This is
969 part of a new backwards-compatibility system:
970 http://allmydata.org/trac/tahoe/wiki/Versioning .
972 The version strings for human inspection (as displayed on the Welcome web
973 page, and included in logs) now includes a platform identifer (frequently
974 including a linux distribution name, processor architecture, etc).
976 Several bugs have been fixed, including one that would cause an exception (in
977 the logs) if a wapi download operation was cancelled (by closing the TCP
978 connection, or pushing the "stop" button in a web browser).
980 Tahoe now uses Foolscap "Incidents", writing an "incident report" file to
981 logs/incidents/ each time something weird occurs. These reports are available
982 to an "incident gatherer" through the flogtool command. For more details,
983 please see the Foolscap logging documentation. An incident-classifying plugin
984 function is provided in misc/incident-gatherer/classify_tahoe.py .
986 If clients detect corruption in shares, they now automatically report it to
987 the server holding that share, if it is new enough to accept the report.
988 These reports are written to files in BASEDIR/storage/corruption-advisories .
990 The 'nickname' setting is now defined to be a UTF-8 -encoded string, allowing
993 The 'tahoe start' command will now accept a --syslog argument and pass it
994 through to twistd, making it easier to launch non-Tahoe nodes (like the
995 cpu-watcher) and have them log to syslogd instead of a local file. This is
996 useful when running a Tahoe node out of a USB flash drive.
998 The Mac GUI in src/allmydata/gui/ has been improved.
1001 * Release 1.2.0 (2008-07-21)
1005 This release makes the immutable-file "ciphertext hash tree" mandatory.
1006 Previous releases allowed the uploader to decide whether their file would
1007 have an integrity check on the ciphertext or not. A malicious uploader could
1008 use this to create a readcap that would download as one file or a different
1009 one, depending upon which shares the client fetched first, with no errors
1010 raised. There are other integrity checks on the shares themselves, preventing
1011 a storage server or other party from violating the integrity properties of
1012 the read-cap: this failure was only exploitable by the uploader who gives you
1013 a carefully constructed read-cap. If you download the file with Tahoe 1.2.0
1014 or later, you will not be vulnerable to this problem. #491
1016 This change does not introduce a compatibility issue, because all existing
1017 versions of Tahoe will emit the ciphertext hash tree in their shares.
1021 Tahoe now requires Foolscap-0.2.9 . It also requires pycryptopp 0.5 or newer,
1022 since earlier versions had a bug that interacted with specific compiler
1023 versions that could sometimes result in incorrect encryption behavior. Both
1024 packages are included in the Tahoe source tarball in misc/dependencies/ , and
1025 should be built automatically when necessary.
1029 Web API directory pages should now contain properly-slash-terminated links to
1030 other directories. They have also stopped using absolute links in forms and
1031 pages (which interfered with the use of a front-end load-balancing proxy).
1033 The behavior of the "Check This File" button changed, in conjunction with
1034 larger internal changes to file checking/verification. The button triggers an
1035 immediate check as before, but the outcome is shown on its own page, and does
1036 not get stored anywhere. As a result, the web directory page no longer shows
1037 historical checker results.
1039 A new "Deep-Check" button has been added, which allows a user to initiate a
1040 recursive check of the given directory and all files and directories
1041 reachable from it. This can cause quite a bit of work, and has no
1042 intermediate progress information or feedback about the process. In addition,
1043 the results of the deep-check are extremely limited. A later release will
1044 improve this behavior.
1046 The web server's behavior with respect to non-ASCII (unicode) filenames in
1047 the "GET save=true" operation has been improved. To achieve maximum
1048 compatibility with variously buggy web browsers, the server does not try to
1049 figure out the character set of the inbound filename. It just echoes the same
1050 bytes back to the browser in the Content-Disposition header. This seems to
1051 make both IE7 and Firefox work correctly.
1053 ** Checker/Verifier/Repairer
1055 Tahoe is slowly acquiring convenient tools to check up on file health,
1056 examine existing shares for errors, and repair files that are not fully
1057 healthy. This release adds a mutable checker/verifier/repairer, although
1058 testing is very limited, and there are no web interfaces to trigger repair
1059 yet. The "Check" button next to each file or directory on the wapi page
1060 will perform a file check, and the "deep check" button on each directory will
1061 recursively check all files and directories reachable from there (which may
1062 take a very long time).
1064 Future releases will improve access to this functionality.
1066 ** Operations/Packaging
1068 A "check-grid" script has been added, along with a Makefile target. This is
1069 intended (with the help of a pre-configured node directory) to check upon the
1070 health of a Tahoe grid, uploading and downloading a few files. This can be
1071 used as a monitoring tool for a deployed grid, to be run periodically and to
1072 signal an error if it ever fails. It also helps with compatibility testing,
1073 to verify that the latest Tahoe code is still able to handle files created by
1076 The munin plugins from misc/munin/ are now copied into any generated debian
1077 packages, and are made executable (and uncompressed) so they can be symlinked
1078 directly from /etc/munin/plugins/ .
1080 Ubuntu "Hardy" was added as a supported debian platform, with a Makefile
1081 target to produce hardy .deb packages. Some notes have been added to
1082 docs/debian.txt about building Tahoe on a debian/ubuntu system.
1084 Storage servers now measure operation rates and latency-per-operation, and
1085 provides results through the /statistics web page as well as the stats
1086 gatherer. Munin plugins have been added to match.
1090 Tahoe nodes now use Foolscap "incident logging" to record unusual events to
1091 their NODEDIR/logs/incidents/ directory. These incident files can be examined
1092 by Foolscap logging tools, or delivered to an external log-gatherer for
1093 further analysis. Note that Tahoe now requires Foolscap-0.2.9, since 0.2.8
1094 had a bug that complained about "OSError: File exists" when trying to create
1095 the incidents/ directory for a second time.
1097 If no servers are available when retrieving a mutable file (like a
1098 directory), the node now reports an error instead of hanging forever. Earlier
1099 releases would not only hang (causing the wapi directory listing to get
1100 stuck half-way through), but the internal dirnode serialization would cause
1101 all subsequent attempts to retrieve or modify the same directory to hang as
1104 A minor internal exception (reported in logs/twistd.log, in the
1105 "stopProducing" method) was fixed, which complained about "self._paused_at
1106 not defined" whenever a file download was stopped from the web browser end.
1109 * Release 1.1.0 (2008-06-11)
1111 ** CLI: new "alias" model
1113 The new CLI code uses an scp/rsync -like interface, in which directories in
1114 the Tahoe storage grid are referenced by a colon-suffixed alias. The new
1116 tahoe cp local.txt tahoe:virtual.txt
1117 tahoe ls work:subdir
1119 More functionality is available through the CLI: creating unlinked files and
1120 directories, recursive copy in or out of the storage grid, hardlinks, and
1121 retrieving the raw read- or write- caps through the 'ls' command. Please read
1122 docs/CLI.txt for complete details.
1124 ** wapi: new pages, new commands
1126 Several new pages were added to the web API:
1128 /helper_status : to describe what a Helper is doing
1129 /statistics : reports node uptime, CPU usage, other stats
1130 /file : for easy file-download URLs, see #221
1131 /cap == /uri : future compatibility
1133 The localdir=/localfile= and t=download operations were removed. These
1134 required special configuration to enable anyways, but this feature was a
1135 security problem, and was mostly obviated by the new "cp -r" command.
1137 Several new options to the GET command were added:
1139 t=deep-size : add up the size of all immutable files reachable from the directory
1140 t=deep-stats : return a JSON-encoded description of number of files, size
1141 distribution, total size, etc
1143 POST is now preferred over PUT for most operations which cause side-effects.
1145 Most wapi calls now accept overwrite=, and default to overwrite=true .
1147 "POST /uri/DIRCAP/parent/child?t=mkdir" is now the preferred API to create
1148 multiple directories at once, rather than ...?t=mkdir-p .
1150 PUT to a mutable file ("PUT /uri/MUTABLEFILECAP", "PUT /uri/DIRCAP/child")
1151 will modify the file in-place.
1153 ** more munin graphs in misc/munin/
1157 tahoe_estimate_files
1158 mutable files published/retrieved
1167 setuptools (now required at runtime)
1169 ** New Mutable-File Code
1171 The mutable-file handling code (mostly used for directories) has been
1172 completely rewritten. The new scheme has a better API (with a modify()
1173 method) and is less likely to lose data when several uncoordinated writers
1174 change a file at the same time.
1176 In addition, a single Tahoe process will coordinate its own writes. If you
1177 make two concurrent directory-modifying wapi calls to a single tahoe node,
1178 it will internally make one of them wait for the other to complete. This
1179 prevents auto-collision (#391).
1181 The new mutable-file code also detects errors during publish better. Earlier
1182 releases might believe that a mutable file was published when in fact it
1187 The node now monitors its own CPU usage, as a percentage, measured every 60
1188 seconds. 1/5/15 minute moving averages are available on the /statistics web
1189 page and via the stats-gathering interface.
1191 Clients now accelerate reconnection to all servers after being offline
1192 (#374). When a client is offline for a long time, it scales back reconnection
1193 attempts to approximately once per hour, so it may take a while to make the
1194 first attempt, but once any attempt succeeds, the other server connections
1195 will be retried immediately.
1197 A new "offloaded KeyGenerator" facility can be configured, to move RSA key
1198 generation out from, say, a wapi node, into a separate process. RSA keys
1199 can take several seconds to create, and so a wapi node which is being used
1200 for directory creation will be unavailable for anything else during this
1201 time. The Key Generator process will pre-compute a small pool of keys, to
1202 speed things up further. This also takes better advantage of multi-core CPUs,
1205 The node will only use a potentially-slow "du -s" command at startup (to
1206 measure how much space has been used) if the "sizelimit" parameter has been
1207 configured (to limit how much space is used). Large storage servers should
1208 turn off sizelimit until a later release improves the space-management code,
1209 since "du -s" on a terabyte filesystem can take hours.
1211 The Introducer now allows new announcements to replace old ones, to avoid
1212 buildups of obsolete announcements.
1214 Immutable files are limited to about 12GiB (when using the default 3-of-10
1215 encoding), because larger files would be corrupted by the four-byte
1216 share-size field on the storage servers (#439). A later release will remove
1217 this limit. Earlier releases would allow >12GiB uploads, but the resulting
1218 file would be unretrievable.
1220 The docs/ directory has been rearranged, with old docs put in
1221 docs/historical/ and not-yet-implemented ones in docs/proposed/ .
1223 The Mac OS-X FUSE plugin has a significant bug fix: earlier versions would
1224 corrupt writes that used seek() instead of writing the file in linear order.
1225 The rsync tool is known to perform writes in this order. This has been fixed.