1 User visible changes in Tahoe-LAFS. -*- outline; coding: utf-8 -*-
5 ** Bugfixes and Improvements
7 - 'top' on some Unix platforms, e.g. Linux, now shows node processes as
8 'tahoe' instead of 'python'. (#174)
9 - Improve HTML formatting of the WUI. (#1219)
10 - The 'tahoe create-alias' and 'tahoe add-alias' commands now accept a
11 trailing colon on the alias. (#1305)
12 - The default reserved_space setting for newly-created storage nodes
17 - The unmaintained GUI applications for Mac OS X and Windows have been
22 - Some documentation updates missed in 1.8.1 have been completed. (#1225)
29 * Release 1.8.1 (2010-10-28)
31 ** Bugfixes and Improvements
33 - Allow the repairer to improve the health of a file by uploading
34 some shares, even if it cannot achieve the configured happiness
35 threshold. This fixes a regression introduced between v1.7.1 and
37 - Fix a memory leak in the ResponseCache which is used during mutable
38 file/directory operations. (#1045)
39 - Fix a regression and add a performance improvement in the downloader.
40 This issue caused repair to fail in some special cases. (#1223)
41 - Fix a bug that caused 'tahoe cp' to fail for a grid-to-grid copy
42 involving a non-ASCII filename. (#1224)
43 - Fix a rarely-encountered bug involving printing large strings to
44 the console on Windows. (#1232)
45 - Perform ~ expansion in the --exclude-from filename argument to
46 'tahoe backup'. (#1241)
47 - The CLI's 'tahoe mv' and 'tahoe ln' commands previously would try
48 to use an HTTP proxy if the HTTP_PROXY environment variable was set.
49 These now always connect directly to the WAPI, thus avoiding giving
50 caps to the HTTP proxy (and also avoiding failures in the case that
51 the proxy is failing or requires authentication). (#1253)
52 - The CLI now correctly reports failure in the case that 'tahoe mv'
53 fails to unlink the file from its old location. (#1255)
54 - 'tahoe start' now gives a more positive indication that the node
56 - The arguments seen by 'ps' or other tools for node processes are
57 now more useful (in particular, they include the path of the
58 'tahoe' script, rather than an obscure tool named 'twistd'). (#174)
62 - The tahoe start/stop/restart and node creation commands no longer
63 accept the -m or --multiple option, for consistency between platforms.
68 - We now host binary packages so that users on certain operating systems
69 can install without having a compiler.
70 <http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html>
71 - Use a newer version of a dependency if needed, even if an older
72 version is installed. This would previously cause a VersionConflict
74 - Use a precompiled binary of a dependency if one with a sufficiently
75 high version number is available, instead of attempting to compile
76 the dependency from source, even if the source version has a higher
77 version number. (#1233)
81 - All current documentation in .txt format has been converted to
83 - Added docs/backdoors.rst declaring that we won't add backdoors to
84 Tahoe-LAFS, or add anything to facilitate government access to data.
88 * Release 1.8.0 (2010-09-23)
92 - A completely new downloader which improves performance and
93 robustness of immutable-file downloads. It uses the fastest K
94 servers to download the data in K-way parallel. It automatically
95 fails over to alternate servers if servers fail in mid-download. It
96 allows seeking to arbitrary locations in the file (the previous
97 downloader which would only read the entire file sequentially from
98 beginning to end). It minimizes unnecessary round trips and
99 unnecessary bytes transferred to improve performance. It sends
100 requests to fewer servers to reduce the load on servers (the
101 previous one would send a small request to every server for every
102 download) (#287, #288, #448, #798, #800, #990, #1170, #1191)
104 - Non-ASCII command-line arguments and non-ASCII outputs now work on
105 Windows. In addition, the command-line tool now works on 64-bit
108 ** Bugfixes and Improvements
110 - Document and clean up the command-line options for specifying the
111 node's base directory. (#188, #706, #715, #772, #1108)
112 - The default node directory for Windows is ".tahoe" in the user's
113 home directory, the same as on other platforms. (#890)
114 - Fix a case in which full cap URIs could be logged. (#685, #1155)
115 - Fix bug in WUI in Python 2.5 when the system clock is set back to
116 1969. Now you can use Tahoe-LAFS with Python 2.5 and set your
117 system clock to 1969 and still use the WUI. (#1055)
118 - Many improvements in code organization, tests, logging,
119 documentation, and packaging. (#983, #1074, #1108, #1127, #1129,
122 ** Dependency Updates
124 - on x86 and x86-64 platforms, pycryptopp >= 0.5.20
125 - pycrypto 2.2 is excluded due to a bug
128 * Release 1.7.1 (2010-07-18)
130 ** Bugfixes and Improvements
132 - Fix bug in which uploader could fail with AssertionFailure or
133 report that it had achieved servers-of-happiness when it
135 - Fix bug in which servers could get into a state where they would
136 refuse to accept shares of a certain file (#1117)
137 - Add init scripts for managing the gateway server on Debian/Ubuntu
139 - Fix bug where server version number was always 0 on the welcome
141 - Add new command-line command "tahoe unlink" as a synonym for "tahoe
143 - The FTP frontend now encrypts its temporary files, protecting their
144 contents from an attacker who is able to read the disk. (#1083)
145 - Fix IP address detection on FreeBSD 7, 8, and 9 (#1098)
146 - Fix minor layout issue in the Web User Interface with Internet
148 - Fix rarely-encountered incompatibility between Twisted logging
149 utility and the new unicode support added in v1.7.0 (#1099)
150 - Forward-compatibility improvements for non-ASCII caps (#1051)
154 - Simplify and tidy-up directories, unicode support, test code (#923, #967,
158 * Release 1.7.0 (2010-06-18)
164 Your Tahoe-LAFS gateway now acts like a full-fledged SFTP server. It has been
165 tested with sshfs to provide a virtual filesystem in Linux. Many users have
166 asked for this feature. We hope that it serves them well! See the
167 docs/frontends/FTP-and-SFTP.txt document to get started.
169 *** support for non-ASCII character encodings
171 Tahoe-LAFS now correctly handles filenames containing non-ASCII characters on
172 all supported platforms:
174 - when reading files in from the local filesystem (such as when you run "tahoe
175 backup" to back up your local files to a Tahoe-LAFS grid);
177 - when writing files out to the local filesystem (such as when you run "tahoe
178 cp -r" to recursively copy files out of a Tahoe-LAFS grid);
180 - when displaying filenames to the terminal (such as when you run "tahoe ls"),
181 subject to limitations of the terminal and locale;
183 - when parsing command-line arguments, except on Windows.
185 *** Servers of Happiness
187 Tahoe-LAFS now measures during immutable file upload to see how well
188 distributed it is across multiple servers. It aborts the upload if the pieces
189 of the file are not sufficiently well-distributed.
191 This behavior is controlled by a configuration parameter called "servers of
192 happiness". With the default settings for its erasure coding, Tahoe-LAFS
193 generates 10 shares for each file, such that any 3 of those shares are
194 sufficient to recover the file. The default value of "servers of happiness" is
195 7, which means that Tahoe-LAFS will guarantee that there are at least 7 servers
196 holding some of the shares, such that any 3 of those servers can completely
199 The new upload code also distributes the shares better than the previous
200 version in some cases and takes better advantage of pre-existing shares (when a
201 file has already been previously uploaded). See the architecture.txt document
204 ** Bugfixes and Improvements
206 - Premature abort of upload if some shares were already present and some
208 - python ./setup.py install -- can't create or remove files in install
210 - Network failure => internal TypeError. (#902)
211 - Install of Tahoe on CentOS 5.4. (#933)
212 - CLI option --node-url now supports https url. (#1028)
213 - HTML/CSS template files were not correctly installed under Windows. (#1033)
214 - MetadataSetter does not enforce restriction on setting "tahoe" subkeys.
216 - ImportError: No module named setuptools_darcs.setuptools_darcs. (#1054)
217 - Renamed Title in xhtml files. (#1062)
218 - Increase Python version dependency to 2.4.4, to avoid a critical CPython
219 security bug. (#1066)
220 - Typo correction for the munin plugin tahoe_storagespace. (#968)
221 - Fix warnings found by pylint. (#973)
222 - Changing format of some documentation files. (#1027)
223 - the misc/ directory was tied up. (#1068)
224 - The 'ctime' and 'mtime' metadata fields are no longer written except by
225 "tahoe backup". (#924)
226 - Unicode filenames in Tahoe-LAFS directories are normalized so that names
227 that differ only in how accents are encoded are treated as the same. (#1076)
228 - Various small improvements to documentation. (#937, #911, #1024, #1082)
232 The 'tahoe debug consolidate' subcommand (for converting old allmydata Windows
233 client backups to a newer format) has been removed.
235 ** Dependency Updates
237 the Python version dependency is raised to 2.4.4 in some cases (2.4.3 for
238 Redhat-based Linux distributions, 2.4.2 for UCS-2 builds) (#1066)
241 mock (only required by unit tests)
244 * Release 1.6.1 (2010-02-27)
248 *** Correct handling of Small Immutable Directories
250 Immutable directories can now be deep-checked and listed in the web UI in
251 all cases. (In v1.6.0, some operations, such as deep-check, on a directory
252 graph that included very small immutable directories, would result in an
253 exception causing the whole operation to abort.) (#948)
255 ** Usability Improvements
257 Improved user interface messages and error reporting. (#681, #837, #939)
259 The timeouts for operation handles have been greatly increased, so that
260 you can view the results of an operation up to 4 days after it has
261 completed. After viewing them for the first time, the results are
262 retained for a further day. (#577)
265 * Release 1.6.0 (2010-02-01)
269 *** Immutable Directories
271 Tahoe-LAFS can now create and handle immutable directories. (#607, #833, #931)
272 These are read just like normal directories, but are "deep-immutable", meaning
273 that all their children (and everything reachable from those children) must be
274 immutable objects (i.e. immutable or literal files, and other immutable
277 These directories must be created in a single webapi call that provides all
278 of the children at once. (Since they cannot be changed after creation, the
279 usual create/add/add sequence cannot be used.) They have URIs that start with
280 "URI:DIR2-CHK:" or "URI:DIR2-LIT:", and are described on the human-facing web
281 interface (aka the "WUI") with a "DIR-IMM" abbreviation (as opposed to "DIR"
282 for the usual read-write directories and "DIR-RO" for read-only directories).
284 Tahoe-LAFS releases before 1.6.0 cannot read the contents of an immutable
285 directory. 1.5.0 will tolerate their presence in a directory listing (and
286 display it as "unknown"). 1.4.1 and earlier cannot tolerate them: a DIR-IMM
287 child in any directory will prevent the listing of that directory.
289 Immutable directories are repairable, just like normal immutable files.
291 The webapi "POST t=mkdir-immutable" call is used to create immutable
292 directories. See docs/frontends/webapi.txt for details.
294 *** "tahoe backup" now creates immutable directories, backupdb has dircache
296 The "tahoe backup" command has been enhanced to create immutable directories
297 (in previous releases, it created read-only mutable directories) (#828). This
298 is significantly faster, since it does not need to create an RSA keypair for
299 each new directory. Also "DIR-IMM" immutable directories are repairable, unlike
300 "DIR-RO" read-only mutable directories at present. (A future Tahoe-LAFS release
301 should also be able to repair DIR-RO.)
303 In addition, the backupdb (used by "tahoe backup" to remember what it has
304 already copied) has been enhanced to store information about existing immutable
305 directories. This allows it to re-use directories that have moved but still
306 contain identical contents, or that have been deleted and later replaced. (The
307 1.5.0 "tahoe backup" command could only re-use directories that were in the
308 same place as they were in the immediately previous backup.) With this change,
309 the backup process no longer needs to read the previous snapshot out of the
310 Tahoe-LAFS grid, reducing the network load considerably. (#606)
312 A "null backup" (in which nothing has changed since the previous backup) will
313 require only two Tahoe-side operations: one to add an Archives/$TIMESTAMP
314 entry, and a second to update the Latest/ link. On the local disk side, it
315 will readdir() all your local directories and stat() all your local files.
317 If you've been using "tahoe backup" for a while, you will notice that your
318 first use of it after upgrading to 1.6.0 may take a long time: it must create
319 proper immutable versions of all the old read-only mutable directories. This
320 process won't take as long as the initial backup (where all the file contents
321 had to be uploaded too): it will require time proportional to the number and
322 size of your directories. After this initial pass, all subsequent passes
323 should take a tiny fraction of the time.
325 As noted above, Tahoe-LAFS versions earlier than 1.5.0 cannot list a directory
326 containing an immutable subdirectory. Tahoe-LAFS versions earlier than 1.6.0
327 cannot read the contents of an immutable directory.
329 The "tahoe backup" command has been improved to skip over unreadable objects
330 (like device files, named pipes, and files with permissions that prevent the
331 command from reading their contents), instead of throwing an exception and
332 terminating the backup process. It also skips over symlinks, because these
333 cannot be represented faithfully in the Tahoe-side filesystem. A warning
334 message will be emitted each time something is skipped. (#729, #850, #641)
336 *** "create-node" command added, "create-client" now implies --no-storage
338 The basic idea behind Tahoe-LAFS's client+server and client-only processes is
339 that you are creating a general-purpose Tahoe-LAFS "node" process, which has
340 several components that can be activated. Storage service is one of these
341 optional components, as is the Helper, FTP server, and SFTP server. Web gateway
342 functionality is nominally on this list, but it is always active; a future
343 release will make it optional. There are three special purpose servers that
344 can't currently be run as a component in a node: introducer, key-generator,
347 So now "tahoe create-node" will create a Tahoe-LAFS node process, and after
348 creation you can edit its tahoe.cfg to enable or disable the desired
349 services. It is a more general-purpose replacement for "tahoe create-client".
350 The default configuration has storage service enabled. For convenience, the
351 "--no-storage" argument makes a tahoe.cfg file that disables storage
354 "tahoe create-client" has been changed to create a Tahoe-LAFS node without a
355 storage service. It is equivalent to "tahoe create-node --no-storage". This
356 helps to reduce the confusion surrounding the use of a command with "client" in
357 its name to create a storage *server*. Use "tahoe create-client" to create a
358 purely client-side node. If you want to offer storage to the grid, use
359 "tahoe create-node" instead.
361 In the future, other services will be added to the node, and they will be
362 controlled through options in tahoe.cfg . The most important of these
363 services may get additional --enable-XYZ or --disable-XYZ arguments to
366 ** Performance Improvements
368 Download of immutable files begins as soon as the downloader has located the K
369 necessary shares (#928, #287). In both the previous and current releases, a
370 downloader will first issue queries to all storage servers on the grid to
371 locate shares before it begins downloading the shares. In previous releases of
372 Tahoe-LAFS, download would not begin until all storage servers on the grid had
373 replied to the query, at which point K shares would be chosen for download from
374 among the shares that were located. In this release, download begins as soon as
375 any K shares are located. This means that downloads start sooner, which is
376 particularly important if there is a server on the grid that is extremely slow
377 or even hung in such a way that it will never respond. In previous releases
378 such a server would have a negative impact on all downloads from that grid. In
379 this release, such a server will have no impact on downloads, as long as K
380 shares can be found on other, quicker, servers. This also means that
381 downloads now use the "best-alacrity" servers that they talk to, as measured by
382 how quickly the servers reply to the initial query. This might cause downloads
383 to go faster, especially on grids with heterogeneous servers or geographical
388 The webapi acquired a new "t=mkdir-with-children" command, to create and
389 populate a directory in a single call. This is significantly faster than
390 using separate "t=mkdir" and "t=set-children" operations (it uses one
391 gateway-to-grid roundtrip, instead of three or four). (#533)
393 The t=set-children (note the hyphen) operation is now documented in
394 docs/frontends/webapi.txt, and is the new preferred spelling of the old
395 t=set_children (with an underscore). The underscore version remains for
396 backwards compatibility. (#381, #927)
398 The tracebacks produced by errors in CLI tools should now be in plain text,
399 instead of HTML (which is unreadable outside of a browser). (#646)
401 The [storage]reserved_space configuration knob (which causes the storage
402 server to refuse shares when available disk space drops below a threshold)
403 should work on Windows now, not just UNIX. (#637)
405 "tahoe cp" should now exit with status "1" if it cannot figure out a suitable
406 target filename, such as when you copy from a bare filecap. (#761)
408 "tahoe get" no longer creates a zero-length file upon error. (#121)
410 "tahoe ls" can now list single files. (#457)
412 "tahoe deep-check --repair" should tolerate repair failures now, instead of
413 halting traversal. (#874, #786)
415 "tahoe create-alias" no longer corrupts the aliases file if it had
416 previously been edited to have no trailing newline. (#741)
418 Many small packaging improvements were made to facilitate the "tahoe-lafs"
419 package being included in Ubuntu. Several mac/win32 binary libraries were
420 removed, some figleaf code-coverage files were removed, a bundled copy of
421 darcsver-1.2.1 was removed, and additional licensing text was added.
423 Several DeprecationWarnings for python2.6 were silenced. (#859)
425 The checker --add-lease option would sometimes fail for shares stored
426 on old (Tahoe v1.2.0) servers. (#875)
428 The documentation for installing on Windows (docs/install.html) has been
431 For other changes not mentioned here, see
432 <http://allmydata.org/trac/tahoe/query?milestone=1.6.0&keywords=!~news-done>.
433 To include the tickets mentioned above, go to
434 <http://allmydata.org/trac/tahoe/query?milestone=1.6.0>.
437 * Release 1.5.0 (2009-08-01)
441 Uploads of immutable files now use pipelined writes, improving upload speed
442 slightly (10%) over high-latency connections. (#392)
444 Processing large directories has been sped up, by removing a O(N^2) algorithm
445 from the dirnode decoding path and retaining unmodified encrypted entries.
448 The human-facing web interface (aka the "WUI") received a significant CSS
449 makeover by Kevin Reid, making it much prettier and easier to read. The WUI
450 "check" and "deep-check" forms now include a "Renew Lease" checkbox,
451 mirroring the CLI --add-lease option, so leases can be added or renewed from
454 The CLI "tahoe mv" command now refuses to overwrite directories. (#705)
456 The CLI "tahoe webopen" command, when run without arguments, will now bring
457 up the "Welcome Page" (node status and mkdir/upload forms).
459 The 3.5MB limit on mutable files was removed, so it should be possible to
460 upload arbitrarily-sized mutable files. Note, however, that the data format
461 and algorithm remains the same, so using mutable files still requires
462 bandwidth, computation, and RAM in proportion to the size of the mutable file.
465 This version of Tahoe-LAFS will tolerate directory entries that contain filecap
466 formats which it does not recognize: files and directories from the future.
467 This should improve the user experience (for 1.5.0 users) when we add new cap
468 formats in the future. Previous versions would fail badly, preventing the user
469 from seeing or editing anything else in those directories. These unrecognized
470 objects can be renamed and deleted, but obviously not read or written. Also
471 they cannot generally be copied. (#683)
475 deep-check-and-repair now tolerates read-only directories, such as the ones
476 produced by the "tahoe backup" CLI command. Read-only directories and mutable
477 files are checked, but not repaired. Previous versions threw an exception
478 when attempting the repair and failed to process the remaining contents. We
479 cannot yet repair these read-only objects, but at least this version allows
480 the rest of the check+repair to proceed. (#625)
482 A bug in 1.4.1 which caused a server to be listed multiple times (and
483 frequently broke all connections to that server) was fixed. (#653)
485 The plaintext-hashing code was removed from the Helper interface, removing
486 the Helper's ability to mount a partial-information-guessing attack. (#722)
488 ** Platform/packaging changes
490 Tahoe-LAFS now runs on NetBSD, OpenBSD, ArchLinux, and NixOS, and on an
491 embedded system based on an ARM CPU running at 266 MHz.
493 Unit test timeouts have been raised to allow the tests to complete on
494 extremely slow platforms like embedded ARM-based NAS boxes, which may take
495 several hours to run the test suite. An ARM-specific data-corrupting bug in
496 an older version of Crypto++ (5.5.2) was identified: ARM-users are encouraged
497 to use recent Crypto++/pycryptopp which avoids this problem.
499 Tahoe-LAFS now requires a SQLite library, either the sqlite3 that comes
500 built-in with python2.5/2.6, or the add-on pysqlite2 if you're using
501 python2.4. In the previous release, this was only needed for the "tahoe backup"
502 command: now it is mandatory.
504 Several minor documentation updates were made.
506 To help get Tahoe-LAFS into Linux distributions like Fedora and Debian,
507 packaging improvements are being made in both Tahoe-LAFS and related libraries
508 like pycryptopp and zfec.
510 The Crypto++ library included in the pycryptopp package has been upgraded to
511 version 5.6.0 of Crypto++, which includes a more efficient implementation of
512 SHA-256 in assembly for x86 or amd64 architectures.
514 ** dependency updates
517 no python-2.4.0 or 2.4.1 (2.4.2 is good)
518 (they contained a bug in base64.b32decode)
519 avoid python-2.6 on windows with mingw: compiler issues
520 python2.4 requires pysqlite2 (2.5,2.6 does not)
525 * Release 1.4.1 (2009-04-13)
527 ** Garbage Collection
529 The big feature for this release is the implementation of garbage collection,
530 allowing Tahoe storage servers to delete shares for old deleted files. When
531 enabled, this uses a "mark and sweep" process: clients are responsible for
532 updating the leases on their shares (generally by running "tahoe deep-check
533 --add-lease"), and servers are allowed to delete any share which does not
534 have an up-to-date lease. The process is described in detail in
535 docs/garbage-collection.txt .
537 The server must be configured to enable garbage-collection, by adding
538 directives to the [storage] section that define an age limit for shares. The
539 default configuration will not delete any shares.
541 Both servers and clients should be upgraded to this release to make the
542 garbage-collection as pleasant as possible. 1.2.0 servers have code to
543 perform the update-lease operation but it suffers from a fatal bug, while
544 1.3.0 servers have update-lease but will return an exception for unknown
545 storage indices, causing clients to emit an Incident for each exception,
546 slowing the add-lease process down to a crawl. 1.1.0 servers did not have the
547 add-lease operation at all.
549 ** Security/Usability Problems Fixed
551 A super-linear algorithm in the Merkle Tree code was fixed, which previously
552 caused e.g. download of a 10GB file to take several hours before the first
553 byte of plaintext could be produced. The new "alacrity" is about 2 minutes. A
554 future release should reduce this to a few seconds by fixing ticket #442.
556 The previous version permitted a small timing attack (due to our use of
557 strcmp) against the write-enabler and lease-renewal/cancel secrets. An
558 attacker who could measure response-time variations of approximatly 3ns
559 against a very noisy background time of about 15ms might be able to guess
560 these secrets. We do not believe this attack was actually feasible. This
561 release closes the attack by first hashing the two strings to be compared
562 with a random secret.
566 In most cases, HTML tracebacks will only be sent if an "Accept: text/html"
567 header was provided with the HTTP request. This will generally cause browsers
568 to get an HTMLized traceback but send regular text/plain tracebacks to
569 non-browsers (like the CLI clients). More errors have been mapped to useful
572 The streaming webapi operations (deep-check and manifest) now have a way to
573 indicate errors (an output line that starts with "ERROR" instead of being
574 legal JSON). See docs/frontends/webapi.txt for details.
576 The storage server now has its own status page (at /storage), linked from the
577 Welcome page. This page shows progress and results of the two new
578 share-crawlers: one which merely counts shares (to give an estimate of how
579 many files/directories are being stored in the grid), the other examines
580 leases and reports how much space would be freed if GC were enabled. The page
581 also shows how much disk space is present, used, reserved, and available for
582 the Tahoe server, and whether the server is currently running in "read-write"
583 mode or "read-only" mode.
585 When a directory node cannot be read (perhaps because of insufficent shares),
586 a minimal webapi page is created so that the "more-info" links (including a
587 Check/Repair operation) will still be accessible.
589 A new "reliability" page was added, with the beginnings of work on a
590 statistical loss model. You can tell this page how many servers you are using
591 and their independent failure probabilities, and it will tell you the
592 likelihood that an arbitrary file will survive each repair period. The
593 "numpy" package must be installed to access this page. A partial paper,
594 written by Shawn Willden, has been added to docs/proposed/lossmodel.lyx .
598 "tahoe check" and "tahoe deep-check" now accept an "--add-lease" argument, to
599 update a lease on all shares. This is the "mark" side of garbage collection.
601 In many cases, CLI error messages have been improved: the ugly HTMLized
602 traceback has been replaced by a normal python traceback.
604 "tahoe deep-check" and "tahoe manifest" now have better error reporting.
605 "tahoe cp" is now non-verbose by default.
607 "tahoe backup" now accepts several "--exclude" arguments, to ignore certain
608 files (like editor temporary files and version-control metadata) during
611 On windows, the CLI now accepts local paths like "c:\dir\file.txt", which
612 previously was interpreted as a Tahoe path using a "c:" alias.
614 The "tahoe restart" command now uses "--force" by default (meaning it will
615 start a node even if it didn't look like there was one already running).
617 The "tahoe debug consolidate" command was added. This takes a series of
618 independent timestamped snapshot directories (such as those created by the
619 allmydata.com windows backup program, or a series of "tahoe cp -r" commands)
620 and creates new snapshots that used shared read-only directories whenever
621 possible (like the output of "tahoe backup"). In the most common case (when
622 the snapshots are fairly similar), the result will use significantly fewer
623 directories than the original, allowing "deep-check" and similar tools to run
624 much faster. In some cases, the speedup can be an order of magnitude or more.
625 This tool is still somewhat experimental, and only needs to be run on large
626 backups produced by something other than "tahoe backup", so it was placed
627 under the "debug" category.
629 "tahoe cp -r --caps-only tahoe:dir localdir" is a diagnostic tool which,
630 instead of copying the full contents of files into the local directory,
631 merely copies their filecaps. This can be used to verify the results of a
632 "consolidation" operation.
636 The codebase no longer rauses RuntimeError as a kind of assert(). Specific
637 exception classes were created for each previous instance of RuntimeError.
639 Many unit tests were changed to use a non-network test harness, speeding them
642 Deep-traversal operations (manifest and deep-check) now walk individual
643 directories in alphabetical order. Occasional turn breaks are inserted to
644 prevent a stack overflow when traversing directories with hundreds of
647 The experimental SFTP server had its path-handling logic changed slightly, to
648 accomodate more SFTP clients, although there are still issues (#645).
651 * Release 1.3.0 (2009-02-13)
653 ** Checker/Verifier/Repairer
655 The primary focus of this release has been writing a checker / verifier /
656 repairer for files and directories. "Checking" is the act of asking storage
657 servers whether they have a share for the given file or directory: if there
658 are not enough shares available, the file or directory will be
659 unrecoverable. "Verifying" is the act of downloading and cryptographically
660 asserting that the server's share is undamaged: it requires more work
661 (bandwidth and CPU) than checking, but can catch problems that simple
662 checking cannot. "Repair" is the act of replacing missing or damaged shares
665 This release includes a full checker, a partial verifier, and a partial
666 repairer. The repairer is able to handle missing shares: new shares are
667 generated and uploaded to make up for the missing ones. This is currently the
668 best application of the repairer: to replace shares that were lost because of
669 server departure or permanent drive failure.
671 The repairer in this release is somewhat able to handle corrupted shares. The
674 * Immutable verifier is incomplete: not all shares are used, and not all
675 fields of those shares are verified. Therefore the immutable verifier has
676 only a moderate chance of detecting corrupted shares.
677 * The mutable verifier is mostly complete: all shares are examined, and most
678 fields of the shares are validated.
679 * The storage server protocol offers no way for the repairer to replace or
680 delete immutable shares. If corruption is detected, the repairer will
681 upload replacement shares to other servers, but the corrupted shares will
683 * read-only directories and read-only mutable files must be repaired by
684 someone who holds the write-cap: the read-cap is insufficient. Moreover,
685 the deep-check-and-repair operation will halt with an error if it attempts
686 to repair one of these read-only objects.
687 * Some forms of corruption can cause both download and repair operations to
688 fail. A future release will fix this, since download should be tolerant of
689 any corruption as long as there are at least 'k' valid shares, and repair
690 should be able to fix any file that is downloadable.
692 If the downloader, verifier, or repairer detects share corruption, the
693 servers which provided the bad shares will be notified (via a file placed in
694 the BASEDIR/storage/corruption-advisories directory) so their operators can
695 manually delete the corrupted shares and investigate the problem. In
696 addition, the "incident gatherer" mechanism will automatically report share
697 corruption to an incident gatherer service, if one is configured. Note that
698 corrupted shares indicate hardware failures, serious software bugs, or malice
699 on the part of the storage server operator, so a corrupted share should be
700 considered highly unusual.
702 By periodically checking/repairing all files and directories, objects in the
703 Tahoe filesystem remain resistant to recoverability failures due to missing
704 and/or broken servers.
706 This release includes a wapi mechanism to initiate checks on individual
707 files and directories (with or without verification, and with or without
708 automatic repair). A related mechanism is used to initiate a "deep-check" on
709 a directory: recursively traversing the directory and its children, checking
710 (and/or verifying/repairing) everything underneath. Both mechanisms can be
711 run with an "output=JSON" argument, to obtain machine-readable check/repair
712 status results. These results include a copy of the filesystem statistics
713 from the "deep-stats" operation (including total number of files, size
714 histogram, etc). If repair is possible, a "Repair" button will appear on the
717 The client web interface now features some extra buttons to initiate check
718 and deep-check operations. When these operations finish, they display a
719 results page that summarizes any problems that were encountered. All
720 long-running deep-traversal operations, including deep-check, use a
721 start-and-poll mechanism, to avoid depending upon a single long-lived HTTP
722 connection. docs/frontends/webapi.txt has details.
726 The "tahoe backup" command is new in this release, which creates efficient
727 versioned backups of a local directory. Given a local pathname and a target
728 Tahoe directory, this will create a read-only snapshot of the local directory
729 in $target/Archives/$timestamp. It will also create $target/Latest, which is
730 a reference to the latest such snapshot. Each time you run "tahoe backup"
731 with the same source and target, a new $timestamp snapshot will be added.
732 These snapshots will share directories that have not changed since the last
733 backup, to speed up the process and minimize storage requirements. In
734 addition, a small database is used to keep track of which local files have
735 been uploaded already, to avoid uploading them a second time. This
736 drastically reduces the work needed to do a "null backup" (when nothing has
737 changed locally), making "tahoe backup' suitable to run from a daily cronjob.
739 Note that the "tahoe backup" CLI command must be used in conjunction with a
740 1.3.0-or-newer Tahoe client node; there was a bug in the 1.2.0 webapi
741 implementation that would prevent the last step (create $target/Latest) from
746 The 12GiB (approximate) immutable-file-size limitation is lifted. This
747 release knows how to handle so-called "v2 immutable shares", which permit
748 immutable files of up to about 18 EiB (about 3*10^14). These v2 shares are
749 created if the file to be uploaded is too large to fit into v1 shares. v1
750 shares are created if the file is small enough to fit into them, so that
751 files created with tahoe-1.3.0 can still be read by earlier versions if they
752 are not too large. Note that storage servers also had to be changed to
753 support larger files, and this release is the first release in which they are
754 able to do that. Clients will detect which servers are capable of supporting
755 large files on upload and will not attempt to upload shares of a large file
756 to a server which doesn't support it.
760 Tahoe now includes experimental FTP and SFTP servers. When configured with a
761 suitable method to translate username+password into a root directory cap, it
762 provides simple access to the virtual filesystem. Remember that FTP is
763 completely unencrypted: passwords, filenames, and file contents are all sent
764 over the wire in cleartext, so FTP should only be used on a local (127.0.0.1)
765 connection. This feature is still in development: there are no unit tests
766 yet, and behavior with respect to Unicode filenames is uncertain. Please see
767 docs/frontends/FTP-and-SFTP.txt for configuration details. (#512, #531)
771 This release adds the 'tahoe create-alias' command, which is a combination of
772 'tahoe mkdir' and 'tahoe add-alias'. This also allows you to start using a
773 new tahoe directory without exposing its URI in the argv list, which is
774 publicly visible (through the process table) on most unix systems. Thanks to
775 Kevin Reid for bringing this issue to our attention.
777 The single-argument form of "tahoe put" was changed to create an unlinked
778 file. I.e. "tahoe put bar.txt" will take the contents of a local "bar.txt"
779 file, upload them to the grid, and print the resulting read-cap; the file
780 will not be attached to any directories. This seemed a bit more useful than
781 the previous behavior (copy stdin, upload to the grid, attach the resulting
782 file into your default tahoe: alias in a child named 'bar.txt').
784 "tahoe put" was also fixed to handle mutable files correctly: "tahoe put
785 bar.txt URI:SSK:..." will read the contents of the local bar.txt and use them
786 to replace the contents of the given mutable file.
788 The "tahoe webopen" command was modified to accept aliases. This means "tahoe
789 webopen tahoe:" will cause your web browser to open to a "wui" page that
790 gives access to the directory associated with the default "tahoe:" alias. It
791 should also accept leading slashes, like "tahoe webopen tahoe:/stuff".
793 Many esoteric debugging commands were moved down into a "debug" subcommand:
796 tahoe debug dump-share
797 tahoe debug find-shares
798 tahoe debug catalog-shares
799 tahoe debug corrupt-share
801 The last command ("tahoe debug corrupt-share") flips a random bit of the
802 given local sharefile. This is used to test the file verifying/repairing
803 code, and obviously should not be used on user data.
805 The cli might not correctly handle arguments which contain non-ascii
806 characters in Tahoe v1.3 (although depending on your platform it
807 might, especially if your platform can be configured to pass such
808 characters on the command-line in utf-8 encoding). See
809 http://allmydata.org/trac/tahoe/ticket/565 for details.
813 The "default webapi port", used when creating a new client node (and in the
814 getting-started documentation), was changed from 8123 to 3456, to reduce
815 confusion when Tahoe accessed through a Firefox browser on which the
816 "Torbutton" extension has been installed. Port 8123 is occasionally used as a
817 Tor control port, so Torbutton adds 8123 to Firefox's list of "banned ports"
818 to avoid CSRF attacks against Tor. Once 8123 is banned, it is difficult to
819 diagnose why you can no longer reach a Tahoe node, so the Tahoe default was
820 changed. Note that 3456 is reserved by IANA for the "vat" protocol, but there
821 are argueably more Torbutton+Tahoe users than vat users these days. Note that
822 this will only affect newly-created client nodes. Pre-existing client nodes,
823 created by earlier versions of tahoe, may still be listening on 8123.
825 All deep-traversal operations (start-manifest, start-deep-size,
826 start-deep-stats, start-deep-check) now use a start-and-poll approach,
827 instead of using a single (fragile) long-running synchronous HTTP connection.
828 All these "start-" operations use POST instead of GET. The old "GET
829 manifest", "GET deep-size", and "POST deep-check" operations have been
832 The new "POST start-manifest" operation, when it finally completes, results
833 in a table of (path,cap), instead of the list of verifycaps produced by the
834 old "GET manifest". The table is available in several formats: use
835 output=html, output=text, or output=json to choose one. The JSON output also
836 includes stats, and a list of verifycaps and storage-index strings.
838 The "return_to=" and "when_done=" arguments have been removed from the
839 t=check and deep-check operations.
841 The top-level status page (/status) now has a machine-readable form, via
842 "/status/?t=json". This includes information about the currently-active
843 uploads and downloads, which may be useful for frontends that wish to display
844 progress information. There is no easy way to correlate the activities
845 displayed here with recent wapi requests, however.
847 Any files in BASEDIR/public_html/ (configurable) will be served in response
848 to requests in the /static/ portion of the URL space. This will simplify the
849 deployment of javascript-based frontends that can still access wapi calls
850 by conforming to the (regrettable) "same-origin policy".
852 The welcome page now has a "Report Incident" button, which is tied into the
853 "Incident Gatherer" machinery. If the node is attached to an incident
854 gatherer (via log_gatherer.furl), then pushing this button will cause an
855 Incident to be signalled: this means recent log events are aggregated and
856 sent in a bundle to the gatherer. The user can push this button after
857 something strange takes place (and they can provide a short message to go
858 along with it), and the relevant data will be delivered to a centralized
859 incident-gatherer for later processing by operations staff.
861 The "HEAD" method should now work correctly, in addition to the usual "GET",
862 "PUT", and "POST" methods. "HEAD" is supposed to return exactly the same
863 headers as "GET" would, but without any of the actual response body data. For
864 mutable files, this now does a brief mapupdate (to figure out the size of the
865 file that would be returned), without actually retrieving the file's
868 The "GET" operation on files can now support the HTTP "Range:" header,
869 allowing requests for partial content. This allows certain media players to
870 correctly stream audio and movies out of a Tahoe grid. The current
871 implementation uses a disk-based cache in BASEDIR/private/cache/download ,
872 which holds the plaintext of the files being downloaded. Future
873 implementations might not use this cache. GET for immutable files now returns
876 Each file and directory now has a "Show More Info" web page, which contains
877 much of the information that was crammed into the directory page before. This
878 includes readonly URIs, storage index strings, object type, buttons to
879 control checking/verifying/repairing, and deep-check/deep-stats buttons (for
880 directories). For mutable files, the "replace contents" upload form has been
881 moved here too. As a result, the directory page is now much simpler and
882 cleaner, and several potentially-misleading links (like t=uri) are now gone.
884 Slashes are discouraged in Tahoe file/directory names, since they cause
885 problems when accessing the filesystem through the wapi. However, there are
886 a couple of accidental ways to generate such names. This release tries to
887 make it easier to correct such mistakes by escaping slashes in several
888 places, allowing slashes in the t=info and t=delete commands, and in the
889 source (but not the target) of a t=rename command.
893 Tahoe's dependencies have been extended to require the "[secure_connections]"
894 feature from Foolscap, which will cause pyOpenSSL to be required and/or
895 installed. If OpenSSL and its development headers are already installed on
896 your system, this can occur automatically. Tahoe now uses pollreactor
897 (instead of the default selectreactor) to work around a bug between pyOpenSSL
898 and the most recent release of Twisted (8.1.0). This bug only affects unit
899 tests (hang during shutdown), and should not impact regular use.
901 The Tahoe source code tarballs now come in two different forms: regular and
902 "sumo". The regular tarball contains just Tahoe, nothing else. When building
903 from the regular tarball, the build process will download any unmet
904 dependencies from the internet (starting with the index at PyPI) so it can
905 build and install them. The "sumo" tarball contains copies of all the
906 libraries that Tahoe requires (foolscap, twisted, zfec, etc), so using the
907 "sumo" tarball should not require any internet access during the build
908 process. This can be useful if you want to build Tahoe while on an airplane,
909 a desert island, or other bandwidth-limited environments.
911 Similarly, allmydata.org now hosts a "tahoe-deps" tarball which contains the
912 latest versions of all these dependencies. This tarball, located at
913 http://allmydata.org/source/tahoe/deps/tahoe-deps.tar.gz, can be unpacked in
914 the tahoe source tree (or in its parent directory), and the build process
915 should satisfy its downloading needs from it instead of reaching out to PyPI.
916 This can be useful if you want to build Tahoe from a darcs checkout while on
917 that airplane or desert island.
919 Because of the previous two changes ("sumo" tarballs and the "tahoe-deps"
920 bundle), most of the files have been removed from misc/dependencies/ . This
921 brings the regular Tahoe tarball down to 2MB (compressed), and the darcs
922 checkout (without history) to about 7.6MB. A full darcs checkout will still
923 be fairly large (because of the historical patches which included the
924 dependent libraries), but a 'lazy' one should now be small.
926 The default "make" target is now an alias for "setup.py build", which itself
927 is an alias for "setup.py develop --prefix support", with some extra work
928 before and after (see setup.cfg). Most of the complicated platform-dependent
929 code in the Makefile was rewritten in Python and moved into setup.py,
930 simplifying things considerably.
932 Likewise, the "make test" target now delegates most of its work to "setup.py
933 test", which takes care of getting PYTHONPATH configured to access the tahoe
934 code (and dependencies) that gets put in support/lib/ by the build_tahoe
935 step. This should allow unit tests to be run even when trial (which is part
936 of Twisted) wasn't already installed (in this case, trial gets installed to
937 support/bin because Twisted is a dependency of Tahoe).
939 Tahoe is now compatible with the recently-released Python 2.6 , although it
940 is recommended to use Tahoe on Python 2.5, on which it has received more
941 thorough testing and deployment.
943 Tahoe is now compatible with simplejson-2.0.x . The previous release assumed
944 that simplejson.loads always returned unicode strings, which is no longer the
947 ** Grid Management Tools
949 Several tools have been added or updated in the misc/ directory, mostly munin
950 plugins that can be used to monitor a storage grid.
952 The misc/spacetime/ directory contains a "disk watcher" daemon (startable
953 with 'tahoe start'), which can be configured with a set of HTTP URLs
954 (pointing at the wapi '/statistics' page of a bunch of storage servers),
955 and will periodically fetch disk-used/disk-available information from all the
956 servers. It keeps this information in an Axiom database (a sqlite-based
957 library available from divmod.org). The daemon computes time-averaged rates
958 of disk usage, as well as a prediction of how much time is left before the
959 grid is completely full.
961 The misc/munin/ directory contains a new set of munin plugins
962 (tahoe_diskleft, tahoe_diskusage, tahoe_doomsday) which talk to the
963 disk-watcher and provide graphs of its calculations.
965 To support the disk-watcher, the Tahoe statistics component (visible through
966 the wapi at the /statistics/ URL) now includes disk-used and disk-available
967 information. Both are derived through an equivalent of the unix 'df' command
968 (i.e. they ask the kernel for the number of free blocks on the partition that
969 encloses the BASEDIR/storage directory). In the future, the disk-available
970 number will be further influenced by the local storage policy: if that policy
971 says that the server should refuse new shares when less than 5GB is left on
972 the partition, then "disk-available" will report zero even though the kernel
975 The 'tahoe_overhead' munin plugin interacts with an allmydata.com-specific
976 server which reports the total of the 'deep-size' reports for all active user
977 accounts, compares this with the disk-watcher data, to report on overhead
978 percentages. This provides information on how much space could be recovered
979 once Tahoe implements some form of garbage collection.
981 ** Configuration Changes: single INI-format tahoe.cfg file
983 The Tahoe node is now configured with a single INI-format file, named
984 "tahoe.cfg", in the node's base directory. Most of the previous
985 multiple-separate-files are still read for backwards compatibility (the
986 embedded SSH debug server and the advertised_ip_addresses files are the
987 exceptions), but new directives will only be added to tahoe.cfg . The "tahoe
988 create-client" command will create a tahoe.cfg for you, with sample values
989 commented out. (ticket #518)
991 tahoe.cfg now has controls for the foolscap "keepalive" and "disconnect"
994 tahoe.cfg now has controls for the encoding parameters: "shares.needed" and
995 "shares.total" in the "[client]" section. The default parameters are still
998 The inefficient storage 'sizelimit' control (which established an upper bound
999 on the amount of space that a storage server is allowed to consume) has been
1000 replaced by a lightweight 'reserved_space' control (which establishes a lower
1001 bound on the amount of remaining space). The storage server will reject all
1002 writes that would cause the remaining disk space (as measured by a '/bin/df'
1003 equivalent) to drop below this value. The "[storage]reserved_space="
1004 tahoe.cfg parameter controls this setting. (note that this only affects
1005 immutable shares: it is an outstanding bug that reserved_space does not
1006 prevent the allocation of new mutable shares, nor does it prevent the growth
1007 of existing mutable shares).
1011 Clients now declare which versions of the protocols they support. This is
1012 part of a new backwards-compatibility system:
1013 http://allmydata.org/trac/tahoe/wiki/Versioning .
1015 The version strings for human inspection (as displayed on the Welcome web
1016 page, and included in logs) now includes a platform identifer (frequently
1017 including a linux distribution name, processor architecture, etc).
1019 Several bugs have been fixed, including one that would cause an exception (in
1020 the logs) if a wapi download operation was cancelled (by closing the TCP
1021 connection, or pushing the "stop" button in a web browser).
1023 Tahoe now uses Foolscap "Incidents", writing an "incident report" file to
1024 logs/incidents/ each time something weird occurs. These reports are available
1025 to an "incident gatherer" through the flogtool command. For more details,
1026 please see the Foolscap logging documentation. An incident-classifying plugin
1027 function is provided in misc/incident-gatherer/classify_tahoe.py .
1029 If clients detect corruption in shares, they now automatically report it to
1030 the server holding that share, if it is new enough to accept the report.
1031 These reports are written to files in BASEDIR/storage/corruption-advisories .
1033 The 'nickname' setting is now defined to be a UTF-8 -encoded string, allowing
1034 non-ascii nicknames.
1036 The 'tahoe start' command will now accept a --syslog argument and pass it
1037 through to twistd, making it easier to launch non-Tahoe nodes (like the
1038 cpu-watcher) and have them log to syslogd instead of a local file. This is
1039 useful when running a Tahoe node out of a USB flash drive.
1041 The Mac GUI in src/allmydata/gui/ has been improved.
1044 * Release 1.2.0 (2008-07-21)
1048 This release makes the immutable-file "ciphertext hash tree" mandatory.
1049 Previous releases allowed the uploader to decide whether their file would
1050 have an integrity check on the ciphertext or not. A malicious uploader could
1051 use this to create a readcap that would download as one file or a different
1052 one, depending upon which shares the client fetched first, with no errors
1053 raised. There are other integrity checks on the shares themselves, preventing
1054 a storage server or other party from violating the integrity properties of
1055 the read-cap: this failure was only exploitable by the uploader who gives you
1056 a carefully constructed read-cap. If you download the file with Tahoe 1.2.0
1057 or later, you will not be vulnerable to this problem. #491
1059 This change does not introduce a compatibility issue, because all existing
1060 versions of Tahoe will emit the ciphertext hash tree in their shares.
1064 Tahoe now requires Foolscap-0.2.9 . It also requires pycryptopp 0.5 or newer,
1065 since earlier versions had a bug that interacted with specific compiler
1066 versions that could sometimes result in incorrect encryption behavior. Both
1067 packages are included in the Tahoe source tarball in misc/dependencies/ , and
1068 should be built automatically when necessary.
1072 Web API directory pages should now contain properly-slash-terminated links to
1073 other directories. They have also stopped using absolute links in forms and
1074 pages (which interfered with the use of a front-end load-balancing proxy).
1076 The behavior of the "Check This File" button changed, in conjunction with
1077 larger internal changes to file checking/verification. The button triggers an
1078 immediate check as before, but the outcome is shown on its own page, and does
1079 not get stored anywhere. As a result, the web directory page no longer shows
1080 historical checker results.
1082 A new "Deep-Check" button has been added, which allows a user to initiate a
1083 recursive check of the given directory and all files and directories
1084 reachable from it. This can cause quite a bit of work, and has no
1085 intermediate progress information or feedback about the process. In addition,
1086 the results of the deep-check are extremely limited. A later release will
1087 improve this behavior.
1089 The web server's behavior with respect to non-ASCII (unicode) filenames in
1090 the "GET save=true" operation has been improved. To achieve maximum
1091 compatibility with variously buggy web browsers, the server does not try to
1092 figure out the character set of the inbound filename. It just echoes the same
1093 bytes back to the browser in the Content-Disposition header. This seems to
1094 make both IE7 and Firefox work correctly.
1096 ** Checker/Verifier/Repairer
1098 Tahoe is slowly acquiring convenient tools to check up on file health,
1099 examine existing shares for errors, and repair files that are not fully
1100 healthy. This release adds a mutable checker/verifier/repairer, although
1101 testing is very limited, and there are no web interfaces to trigger repair
1102 yet. The "Check" button next to each file or directory on the wapi page
1103 will perform a file check, and the "deep check" button on each directory will
1104 recursively check all files and directories reachable from there (which may
1105 take a very long time).
1107 Future releases will improve access to this functionality.
1109 ** Operations/Packaging
1111 A "check-grid" script has been added, along with a Makefile target. This is
1112 intended (with the help of a pre-configured node directory) to check upon the
1113 health of a Tahoe grid, uploading and downloading a few files. This can be
1114 used as a monitoring tool for a deployed grid, to be run periodically and to
1115 signal an error if it ever fails. It also helps with compatibility testing,
1116 to verify that the latest Tahoe code is still able to handle files created by
1119 The munin plugins from misc/munin/ are now copied into any generated debian
1120 packages, and are made executable (and uncompressed) so they can be symlinked
1121 directly from /etc/munin/plugins/ .
1123 Ubuntu "Hardy" was added as a supported debian platform, with a Makefile
1124 target to produce hardy .deb packages. Some notes have been added to
1125 docs/debian.txt about building Tahoe on a debian/ubuntu system.
1127 Storage servers now measure operation rates and latency-per-operation, and
1128 provides results through the /statistics web page as well as the stats
1129 gatherer. Munin plugins have been added to match.
1133 Tahoe nodes now use Foolscap "incident logging" to record unusual events to
1134 their NODEDIR/logs/incidents/ directory. These incident files can be examined
1135 by Foolscap logging tools, or delivered to an external log-gatherer for
1136 further analysis. Note that Tahoe now requires Foolscap-0.2.9, since 0.2.8
1137 had a bug that complained about "OSError: File exists" when trying to create
1138 the incidents/ directory for a second time.
1140 If no servers are available when retrieving a mutable file (like a
1141 directory), the node now reports an error instead of hanging forever. Earlier
1142 releases would not only hang (causing the wapi directory listing to get
1143 stuck half-way through), but the internal dirnode serialization would cause
1144 all subsequent attempts to retrieve or modify the same directory to hang as
1147 A minor internal exception (reported in logs/twistd.log, in the
1148 "stopProducing" method) was fixed, which complained about "self._paused_at
1149 not defined" whenever a file download was stopped from the web browser end.
1152 * Release 1.1.0 (2008-06-11)
1154 ** CLI: new "alias" model
1156 The new CLI code uses an scp/rsync -like interface, in which directories in
1157 the Tahoe storage grid are referenced by a colon-suffixed alias. The new
1159 tahoe cp local.txt tahoe:virtual.txt
1160 tahoe ls work:subdir
1162 More functionality is available through the CLI: creating unlinked files and
1163 directories, recursive copy in or out of the storage grid, hardlinks, and
1164 retrieving the raw read- or write- caps through the 'ls' command. Please read
1165 docs/CLI.txt for complete details.
1167 ** wapi: new pages, new commands
1169 Several new pages were added to the web API:
1171 /helper_status : to describe what a Helper is doing
1172 /statistics : reports node uptime, CPU usage, other stats
1173 /file : for easy file-download URLs, see #221
1174 /cap == /uri : future compatibility
1176 The localdir=/localfile= and t=download operations were removed. These
1177 required special configuration to enable anyways, but this feature was a
1178 security problem, and was mostly obviated by the new "cp -r" command.
1180 Several new options to the GET command were added:
1182 t=deep-size : add up the size of all immutable files reachable from the directory
1183 t=deep-stats : return a JSON-encoded description of number of files, size
1184 distribution, total size, etc
1186 POST is now preferred over PUT for most operations which cause side-effects.
1188 Most wapi calls now accept overwrite=, and default to overwrite=true .
1190 "POST /uri/DIRCAP/parent/child?t=mkdir" is now the preferred API to create
1191 multiple directories at once, rather than ...?t=mkdir-p .
1193 PUT to a mutable file ("PUT /uri/MUTABLEFILECAP", "PUT /uri/DIRCAP/child")
1194 will modify the file in-place.
1196 ** more munin graphs in misc/munin/
1200 tahoe_estimate_files
1201 mutable files published/retrieved
1210 setuptools (now required at runtime)
1212 ** New Mutable-File Code
1214 The mutable-file handling code (mostly used for directories) has been
1215 completely rewritten. The new scheme has a better API (with a modify()
1216 method) and is less likely to lose data when several uncoordinated writers
1217 change a file at the same time.
1219 In addition, a single Tahoe process will coordinate its own writes. If you
1220 make two concurrent directory-modifying wapi calls to a single tahoe node,
1221 it will internally make one of them wait for the other to complete. This
1222 prevents auto-collision (#391).
1224 The new mutable-file code also detects errors during publish better. Earlier
1225 releases might believe that a mutable file was published when in fact it
1230 The node now monitors its own CPU usage, as a percentage, measured every 60
1231 seconds. 1/5/15 minute moving averages are available on the /statistics web
1232 page and via the stats-gathering interface.
1234 Clients now accelerate reconnection to all servers after being offline
1235 (#374). When a client is offline for a long time, it scales back reconnection
1236 attempts to approximately once per hour, so it may take a while to make the
1237 first attempt, but once any attempt succeeds, the other server connections
1238 will be retried immediately.
1240 A new "offloaded KeyGenerator" facility can be configured, to move RSA key
1241 generation out from, say, a wapi node, into a separate process. RSA keys
1242 can take several seconds to create, and so a wapi node which is being used
1243 for directory creation will be unavailable for anything else during this
1244 time. The Key Generator process will pre-compute a small pool of keys, to
1245 speed things up further. This also takes better advantage of multi-core CPUs,
1248 The node will only use a potentially-slow "du -s" command at startup (to
1249 measure how much space has been used) if the "sizelimit" parameter has been
1250 configured (to limit how much space is used). Large storage servers should
1251 turn off sizelimit until a later release improves the space-management code,
1252 since "du -s" on a terabyte filesystem can take hours.
1254 The Introducer now allows new announcements to replace old ones, to avoid
1255 buildups of obsolete announcements.
1257 Immutable files are limited to about 12GiB (when using the default 3-of-10
1258 encoding), because larger files would be corrupted by the four-byte
1259 share-size field on the storage servers (#439). A later release will remove
1260 this limit. Earlier releases would allow >12GiB uploads, but the resulting
1261 file would be unretrievable.
1263 The docs/ directory has been rearranged, with old docs put in
1264 docs/historical/ and not-yet-implemented ones in docs/proposed/ .
1266 The Mac OS-X FUSE plugin has a significant bug fix: earlier versions would
1267 corrupt writes that used seek() instead of writing the file in linear order.
1268 The rsync tool is known to perform writes in this order. This has been fixed.