From a817163cc698316df42f5ff1bd3e5ff0eb43fd78 Mon Sep 17 00:00:00 2001 From: david-sarah Date: Mon, 16 May 2011 18:02:55 -0700 Subject: [PATCH] docs: convert NEWS to NEWS.rst and change all references to it. --- NEWS | 1271 ----------------- NEWS.rst | 1579 +++++++++++++++++++++ docs/how_to_make_a_tahoe-lafs_release.rst | 2 +- misc/debian/rules.sid | 2 +- misc/debian_helpers/etch/debian/rules | 2 +- misc/debian_helpers/lenny/debian/rules | 2 +- misc/debian_helpers/sid/debian/rules | 2 +- relnotes.txt | 4 +- src/allmydata/test/test_repairer.py | 2 +- 9 files changed, 1587 insertions(+), 1279 deletions(-) delete mode 100644 NEWS create mode 100644 NEWS.rst diff --git a/NEWS b/NEWS deleted file mode 100644 index 3e08bb16..00000000 --- a/NEWS +++ /dev/null @@ -1,1271 +0,0 @@ -User visible changes in Tahoe-LAFS. -*- outline; coding: utf-8 -*- - -* Release 1.8.2 (2011-01-30) - -** Compatibility and Dependencies - - - Tahoe is now compatible with Twisted-10.2 (released last month), as well - as with earlier versions. The previous Tahoe-1.8.1 release failed to run - against Twisted-10.2, raising an AttributeError on - StreamServerEndpointService (#1286) - - Tahoe now depends upon the "mock" testing library, and the foolscap - dependency was raised to 0.6.1 . It no longer requires pywin32 (which - was used only on windows). Future developers should note that - reactor.spawnProcess and derivatives may no longer be used inside - Tahoe code. - -** Other Changes - - - the default reserved_space value for new storage nodes is 1 GB (#1208) - - documentation is now in reStructuredText (.rst) format - - "tahoe cp" should now handle non-ASCII filenames - - the unmaintained Mac/Windows GUI applications have been removed (#1282) - - tahoe processes should appear in top and ps as "tahoe", not "python", - on some unix platforms. (#174) - - "tahoe debug trial" can be used to run the test suite (#1296) - - the SFTP frontend now reports unknown sizes as "0" instead of "?", - to improve compatibility with clients like FileZilla (#1337) - - "tahoe --version" should now report correct values in situations where - 1.8.1 might have been wrong (#1287) - - -* Release 1.8.1 (2010-10-28) - -** Bugfixes and Improvements - - - Allow the repairer to improve the health of a file by uploading - some shares, even if it cannot achieve the configured happiness - threshold. This fixes a regression introduced between v1.7.1 and - v1.8.0. (#1212) - - Fix a memory leak in the ResponseCache which is used during mutable - file/directory operations. (#1045) - - Fix a regression and add a performance improvement in the downloader. - This issue caused repair to fail in some special cases. (#1223) - - Fix a bug that caused 'tahoe cp' to fail for a grid-to-grid copy - involving a non-ASCII filename. (#1224) - - Fix a rarely-encountered bug involving printing large strings to - the console on Windows. (#1232) - - Perform ~ expansion in the --exclude-from filename argument to - 'tahoe backup'. (#1241) - - The CLI's 'tahoe mv' and 'tahoe ln' commands previously would try - to use an HTTP proxy if the HTTP_PROXY environment variable was set. - These now always connect directly to the WAPI, thus avoiding giving - caps to the HTTP proxy (and also avoiding failures in the case that - the proxy is failing or requires authentication). (#1253) - - The CLI now correctly reports failure in the case that 'tahoe mv' - fails to unlink the file from its old location. (#1255) - - 'tahoe start' now gives a more positive indication that the node - has started. (#71) - - The arguments seen by 'ps' or other tools for node processes are - now more useful (in particular, they include the path of the - 'tahoe' script, rather than an obscure tool named 'twistd'). (#174) - -** Removed Features - - - The tahoe start/stop/restart and node creation commands no longer - accept the -m or --multiple option, for consistency between platforms. - (#1262) - -** Packaging - - - We now host binary packages so that users on certain operating systems - can install without having a compiler. - - - Use a newer version of a dependency if needed, even if an older - version is installed. This would previously cause a VersionConflict - error. (#1190) - - Use a precompiled binary of a dependency if one with a sufficiently - high version number is available, instead of attempting to compile - the dependency from source, even if the source version has a higher - version number. (#1233) - -** Documentation - - - All current documentation in .txt format has been converted to - .rst format. (#1225) - - Added docs/backdoors.rst declaring that we won't add backdoors to - Tahoe-LAFS, or add anything to facilitate government access to data. - (#1216) - - -* Release 1.8.0 (2010-09-23) - -** New Features - - - A completely new downloader which improves performance and - robustness of immutable-file downloads. It uses the fastest K - servers to download the data in K-way parallel. It automatically - fails over to alternate servers if servers fail in mid-download. It - allows seeking to arbitrary locations in the file (the previous - downloader which would only read the entire file sequentially from - beginning to end). It minimizes unnecessary round trips and - unnecessary bytes transferred to improve performance. It sends - requests to fewer servers to reduce the load on servers (the - previous one would send a small request to every server for every - download) (#287, #288, #448, #798, #800, #990, #1170, #1191) - - - Non-ASCII command-line arguments and non-ASCII outputs now work on - Windows. In addition, the command-line tool now works on 64-bit - Windows. (#1074) - -** Bugfixes and Improvements - - - Document and clean up the command-line options for specifying the - node's base directory. (#188, #706, #715, #772, #1108) - - The default node directory for Windows is ".tahoe" in the user's - home directory, the same as on other platforms. (#890) - - Fix a case in which full cap URIs could be logged. (#685, #1155) - - Fix bug in WUI in Python 2.5 when the system clock is set back to - 1969. Now you can use Tahoe-LAFS with Python 2.5 and set your - system clock to 1969 and still use the WUI. (#1055) - - Many improvements in code organization, tests, logging, - documentation, and packaging. (#983, #1074, #1108, #1127, #1129, - #1131, #1166, #1175) - -** Dependency Updates - - - on x86 and x86-64 platforms, pycryptopp >= 0.5.20 - - pycrypto 2.2 is excluded due to a bug - - -* Release 1.7.1 (2010-07-18) - -** Bugfixes and Improvements - - - Fix bug in which uploader could fail with AssertionFailure or - report that it had achieved servers-of-happiness when it - hadn't. (#1118) - - Fix bug in which servers could get into a state where they would - refuse to accept shares of a certain file (#1117) - - Add init scripts for managing the gateway server on Debian/Ubuntu - (#961) - - Fix bug where server version number was always 0 on the welcome - page (#1067) - - Add new command-line command "tahoe unlink" as a synonym for "tahoe - rm" (#776) - - The FTP frontend now encrypts its temporary files, protecting their - contents from an attacker who is able to read the disk. (#1083) - - Fix IP address detection on FreeBSD 7, 8, and 9 (#1098) - - Fix minor layout issue in the Web User Interface with Internet - Explorer (#1097) - - Fix rarely-encountered incompatibility between Twisted logging - utility and the new unicode support added in v1.7.0 (#1099) - - Forward-compatibility improvements for non-ASCII caps (#1051) - -** Code improvements - - - Simplify and tidy-up directories, unicode support, test code (#923, #967, - #1072) - - -* Release 1.7.0 (2010-06-18) - -** New Features - -*** SFTP support - -Your Tahoe-LAFS gateway now acts like a full-fledged SFTP server. It has been -tested with sshfs to provide a virtual filesystem in Linux. Many users have -asked for this feature. We hope that it serves them well! See the -docs/frontends/FTP-and-SFTP.txt document to get started. - -*** support for non-ASCII character encodings - -Tahoe-LAFS now correctly handles filenames containing non-ASCII characters on -all supported platforms: - - - when reading files in from the local filesystem (such as when you run "tahoe - backup" to back up your local files to a Tahoe-LAFS grid); - - - when writing files out to the local filesystem (such as when you run "tahoe - cp -r" to recursively copy files out of a Tahoe-LAFS grid); - - - when displaying filenames to the terminal (such as when you run "tahoe ls"), - subject to limitations of the terminal and locale; - - - when parsing command-line arguments, except on Windows. - -*** Servers of Happiness - -Tahoe-LAFS now measures during immutable file upload to see how well -distributed it is across multiple servers. It aborts the upload if the pieces -of the file are not sufficiently well-distributed. - -This behavior is controlled by a configuration parameter called "servers of -happiness". With the default settings for its erasure coding, Tahoe-LAFS -generates 10 shares for each file, such that any 3 of those shares are -sufficient to recover the file. The default value of "servers of happiness" is -7, which means that Tahoe-LAFS will guarantee that there are at least 7 servers -holding some of the shares, such that any 3 of those servers can completely -recover your file. - -The new upload code also distributes the shares better than the previous -version in some cases and takes better advantage of pre-existing shares (when a -file has already been previously uploaded). See the architecture.txt document -[3] for details. - -** Bugfixes and Improvements - - - Premature abort of upload if some shares were already present and some - servers fail. (#608) - - python ./setup.py install -- can't create or remove files in install - directory. (#803) - - Network failure => internal TypeError. (#902) - - Install of Tahoe on CentOS 5.4. (#933) - - CLI option --node-url now supports https url. (#1028) - - HTML/CSS template files were not correctly installed under Windows. (#1033) - - MetadataSetter does not enforce restriction on setting "tahoe" subkeys. - (#1034) - - ImportError: No module named setuptools_darcs.setuptools_darcs. (#1054) - - Renamed Title in xhtml files. (#1062) - - Increase Python version dependency to 2.4.4, to avoid a critical CPython - security bug. (#1066) - - Typo correction for the munin plugin tahoe_storagespace. (#968) - - Fix warnings found by pylint. (#973) - - Changing format of some documentation files. (#1027) - - the misc/ directory was tied up. (#1068) - - The 'ctime' and 'mtime' metadata fields are no longer written except by - "tahoe backup". (#924) - - Unicode filenames in Tahoe-LAFS directories are normalized so that names - that differ only in how accents are encoded are treated as the same. (#1076) - - Various small improvements to documentation. (#937, #911, #1024, #1082) - -** Removals - -The 'tahoe debug consolidate' subcommand (for converting old allmydata Windows -client backups to a newer format) has been removed. - -** Dependency Updates - -the Python version dependency is raised to 2.4.4 in some cases (2.4.3 for - Redhat-based Linux distributions, 2.4.2 for UCS-2 builds) (#1066) -pycrypto >= 2.0.1 -pyasn1 >= 0.0.8a -mock (only required by unit tests) - - -* Release 1.6.1 (2010-02-27) - -** Bugfixes - -*** Correct handling of Small Immutable Directories - -Immutable directories can now be deep-checked and listed in the web UI in -all cases. (In v1.6.0, some operations, such as deep-check, on a directory -graph that included very small immutable directories, would result in an -exception causing the whole operation to abort.) (#948) - -** Usability Improvements - -Improved user interface messages and error reporting. (#681, #837, #939) - -The timeouts for operation handles have been greatly increased, so that -you can view the results of an operation up to 4 days after it has -completed. After viewing them for the first time, the results are -retained for a further day. (#577) - - -* Release 1.6.0 (2010-02-01) - -** New Features - -*** Immutable Directories - -Tahoe-LAFS can now create and handle immutable directories. (#607, #833, #931) -These are read just like normal directories, but are "deep-immutable", meaning -that all their children (and everything reachable from those children) must be -immutable objects (i.e. immutable or literal files, and other immutable -directories). - -These directories must be created in a single webapi call that provides all -of the children at once. (Since they cannot be changed after creation, the -usual create/add/add sequence cannot be used.) They have URIs that start with -"URI:DIR2-CHK:" or "URI:DIR2-LIT:", and are described on the human-facing web -interface (aka the "WUI") with a "DIR-IMM" abbreviation (as opposed to "DIR" -for the usual read-write directories and "DIR-RO" for read-only directories). - -Tahoe-LAFS releases before 1.6.0 cannot read the contents of an immutable -directory. 1.5.0 will tolerate their presence in a directory listing (and -display it as "unknown"). 1.4.1 and earlier cannot tolerate them: a DIR-IMM -child in any directory will prevent the listing of that directory. - -Immutable directories are repairable, just like normal immutable files. - -The webapi "POST t=mkdir-immutable" call is used to create immutable -directories. See docs/frontends/webapi.txt for details. - -*** "tahoe backup" now creates immutable directories, backupdb has dircache - -The "tahoe backup" command has been enhanced to create immutable directories -(in previous releases, it created read-only mutable directories) (#828). This -is significantly faster, since it does not need to create an RSA keypair for -each new directory. Also "DIR-IMM" immutable directories are repairable, unlike -"DIR-RO" read-only mutable directories at present. (A future Tahoe-LAFS release -should also be able to repair DIR-RO.) - -In addition, the backupdb (used by "tahoe backup" to remember what it has -already copied) has been enhanced to store information about existing immutable -directories. This allows it to re-use directories that have moved but still -contain identical contents, or that have been deleted and later replaced. (The -1.5.0 "tahoe backup" command could only re-use directories that were in the -same place as they were in the immediately previous backup.) With this change, -the backup process no longer needs to read the previous snapshot out of the -Tahoe-LAFS grid, reducing the network load considerably. (#606) - -A "null backup" (in which nothing has changed since the previous backup) will -require only two Tahoe-side operations: one to add an Archives/$TIMESTAMP -entry, and a second to update the Latest/ link. On the local disk side, it -will readdir() all your local directories and stat() all your local files. - -If you've been using "tahoe backup" for a while, you will notice that your -first use of it after upgrading to 1.6.0 may take a long time: it must create -proper immutable versions of all the old read-only mutable directories. This -process won't take as long as the initial backup (where all the file contents -had to be uploaded too): it will require time proportional to the number and -size of your directories. After this initial pass, all subsequent passes -should take a tiny fraction of the time. - -As noted above, Tahoe-LAFS versions earlier than 1.5.0 cannot list a directory -containing an immutable subdirectory. Tahoe-LAFS versions earlier than 1.6.0 -cannot read the contents of an immutable directory. - -The "tahoe backup" command has been improved to skip over unreadable objects -(like device files, named pipes, and files with permissions that prevent the -command from reading their contents), instead of throwing an exception and -terminating the backup process. It also skips over symlinks, because these -cannot be represented faithfully in the Tahoe-side filesystem. A warning -message will be emitted each time something is skipped. (#729, #850, #641) - -*** "create-node" command added, "create-client" now implies --no-storage - -The basic idea behind Tahoe-LAFS's client+server and client-only processes is -that you are creating a general-purpose Tahoe-LAFS "node" process, which has -several components that can be activated. Storage service is one of these -optional components, as is the Helper, FTP server, and SFTP server. Web gateway -functionality is nominally on this list, but it is always active; a future -release will make it optional. There are three special purpose servers that -can't currently be run as a component in a node: introducer, key-generator, -and stats-gatherer. - -So now "tahoe create-node" will create a Tahoe-LAFS node process, and after -creation you can edit its tahoe.cfg to enable or disable the desired -services. It is a more general-purpose replacement for "tahoe create-client". -The default configuration has storage service enabled. For convenience, the -"--no-storage" argument makes a tahoe.cfg file that disables storage -service. (#760) - -"tahoe create-client" has been changed to create a Tahoe-LAFS node without a -storage service. It is equivalent to "tahoe create-node --no-storage". This -helps to reduce the confusion surrounding the use of a command with "client" in -its name to create a storage *server*. Use "tahoe create-client" to create a -purely client-side node. If you want to offer storage to the grid, use -"tahoe create-node" instead. - -In the future, other services will be added to the node, and they will be -controlled through options in tahoe.cfg . The most important of these -services may get additional --enable-XYZ or --disable-XYZ arguments to -"tahoe create-node". - -** Performance Improvements - -Download of immutable files begins as soon as the downloader has located the K -necessary shares (#928, #287). In both the previous and current releases, a -downloader will first issue queries to all storage servers on the grid to -locate shares before it begins downloading the shares. In previous releases of -Tahoe-LAFS, download would not begin until all storage servers on the grid had -replied to the query, at which point K shares would be chosen for download from -among the shares that were located. In this release, download begins as soon as -any K shares are located. This means that downloads start sooner, which is -particularly important if there is a server on the grid that is extremely slow -or even hung in such a way that it will never respond. In previous releases -such a server would have a negative impact on all downloads from that grid. In -this release, such a server will have no impact on downloads, as long as K -shares can be found on other, quicker, servers. This also means that -downloads now use the "best-alacrity" servers that they talk to, as measured by -how quickly the servers reply to the initial query. This might cause downloads -to go faster, especially on grids with heterogeneous servers or geographical -dispersion. - -** Minor Changes - -The webapi acquired a new "t=mkdir-with-children" command, to create and -populate a directory in a single call. This is significantly faster than -using separate "t=mkdir" and "t=set-children" operations (it uses one -gateway-to-grid roundtrip, instead of three or four). (#533) - -The t=set-children (note the hyphen) operation is now documented in -docs/frontends/webapi.txt, and is the new preferred spelling of the old -t=set_children (with an underscore). The underscore version remains for -backwards compatibility. (#381, #927) - -The tracebacks produced by errors in CLI tools should now be in plain text, -instead of HTML (which is unreadable outside of a browser). (#646) - -The [storage]reserved_space configuration knob (which causes the storage -server to refuse shares when available disk space drops below a threshold) -should work on Windows now, not just UNIX. (#637) - -"tahoe cp" should now exit with status "1" if it cannot figure out a suitable -target filename, such as when you copy from a bare filecap. (#761) - -"tahoe get" no longer creates a zero-length file upon error. (#121) - -"tahoe ls" can now list single files. (#457) - -"tahoe deep-check --repair" should tolerate repair failures now, instead of -halting traversal. (#874, #786) - -"tahoe create-alias" no longer corrupts the aliases file if it had -previously been edited to have no trailing newline. (#741) - -Many small packaging improvements were made to facilitate the "tahoe-lafs" -package being included in Ubuntu. Several mac/win32 binary libraries were -removed, some figleaf code-coverage files were removed, a bundled copy of -darcsver-1.2.1 was removed, and additional licensing text was added. - -Several DeprecationWarnings for python2.6 were silenced. (#859) - -The checker --add-lease option would sometimes fail for shares stored -on old (Tahoe v1.2.0) servers. (#875) - -The documentation for installing on Windows (docs/quickstart.rst) has been -improved. (#773) - -For other changes not mentioned here, see -. -To include the tickets mentioned above, go to -. - - -* Release 1.5.0 (2009-08-01) - -** Improvements - -Uploads of immutable files now use pipelined writes, improving upload speed -slightly (10%) over high-latency connections. (#392) - -Processing large directories has been sped up, by removing a O(N^2) algorithm -from the dirnode decoding path and retaining unmodified encrypted entries. -(#750, #752) - -The human-facing web interface (aka the "WUI") received a significant CSS -makeover by Kevin Reid, making it much prettier and easier to read. The WUI -"check" and "deep-check" forms now include a "Renew Lease" checkbox, -mirroring the CLI --add-lease option, so leases can be added or renewed from -the web interface. - -The CLI "tahoe mv" command now refuses to overwrite directories. (#705) - -The CLI "tahoe webopen" command, when run without arguments, will now bring -up the "Welcome Page" (node status and mkdir/upload forms). - -The 3.5MB limit on mutable files was removed, so it should be possible to -upload arbitrarily-sized mutable files. Note, however, that the data format -and algorithm remains the same, so using mutable files still requires -bandwidth, computation, and RAM in proportion to the size of the mutable file. -(#694) - -This version of Tahoe-LAFS will tolerate directory entries that contain filecap -formats which it does not recognize: files and directories from the future. -This should improve the user experience (for 1.5.0 users) when we add new cap -formats in the future. Previous versions would fail badly, preventing the user -from seeing or editing anything else in those directories. These unrecognized -objects can be renamed and deleted, but obviously not read or written. Also -they cannot generally be copied. (#683) - -** Bugfixes - -deep-check-and-repair now tolerates read-only directories, such as the ones -produced by the "tahoe backup" CLI command. Read-only directories and mutable -files are checked, but not repaired. Previous versions threw an exception -when attempting the repair and failed to process the remaining contents. We -cannot yet repair these read-only objects, but at least this version allows -the rest of the check+repair to proceed. (#625) - -A bug in 1.4.1 which caused a server to be listed multiple times (and -frequently broke all connections to that server) was fixed. (#653) - -The plaintext-hashing code was removed from the Helper interface, removing -the Helper's ability to mount a partial-information-guessing attack. (#722) - -** Platform/packaging changes - -Tahoe-LAFS now runs on NetBSD, OpenBSD, ArchLinux, and NixOS, and on an -embedded system based on an ARM CPU running at 266 MHz. - -Unit test timeouts have been raised to allow the tests to complete on -extremely slow platforms like embedded ARM-based NAS boxes, which may take -several hours to run the test suite. An ARM-specific data-corrupting bug in -an older version of Crypto++ (5.5.2) was identified: ARM-users are encouraged -to use recent Crypto++/pycryptopp which avoids this problem. - -Tahoe-LAFS now requires a SQLite library, either the sqlite3 that comes -built-in with python2.5/2.6, or the add-on pysqlite2 if you're using -python2.4. In the previous release, this was only needed for the "tahoe backup" -command: now it is mandatory. - -Several minor documentation updates were made. - -To help get Tahoe-LAFS into Linux distributions like Fedora and Debian, -packaging improvements are being made in both Tahoe-LAFS and related libraries -like pycryptopp and zfec. - -The Crypto++ library included in the pycryptopp package has been upgraded to -version 5.6.0 of Crypto++, which includes a more efficient implementation of -SHA-256 in assembly for x86 or amd64 architectures. - -** dependency updates - - foolscap-0.4.1 - no python-2.4.0 or 2.4.1 (2.4.2 is good) - (they contained a bug in base64.b32decode) - avoid python-2.6 on windows with mingw: compiler issues - python2.4 requires pysqlite2 (2.5,2.6 does not) - no python-3.x - pycryptopp-0.5.15 - - -* Release 1.4.1 (2009-04-13) - -** Garbage Collection - -The big feature for this release is the implementation of garbage collection, -allowing Tahoe storage servers to delete shares for old deleted files. When -enabled, this uses a "mark and sweep" process: clients are responsible for -updating the leases on their shares (generally by running "tahoe deep-check ---add-lease"), and servers are allowed to delete any share which does not -have an up-to-date lease. The process is described in detail in -docs/garbage-collection.txt . - -The server must be configured to enable garbage-collection, by adding -directives to the [storage] section that define an age limit for shares. The -default configuration will not delete any shares. - -Both servers and clients should be upgraded to this release to make the -garbage-collection as pleasant as possible. 1.2.0 servers have code to -perform the update-lease operation but it suffers from a fatal bug, while -1.3.0 servers have update-lease but will return an exception for unknown -storage indices, causing clients to emit an Incident for each exception, -slowing the add-lease process down to a crawl. 1.1.0 servers did not have the -add-lease operation at all. - -** Security/Usability Problems Fixed - -A super-linear algorithm in the Merkle Tree code was fixed, which previously -caused e.g. download of a 10GB file to take several hours before the first -byte of plaintext could be produced. The new "alacrity" is about 2 minutes. A -future release should reduce this to a few seconds by fixing ticket #442. - -The previous version permitted a small timing attack (due to our use of -strcmp) against the write-enabler and lease-renewal/cancel secrets. An -attacker who could measure response-time variations of approximatly 3ns -against a very noisy background time of about 15ms might be able to guess -these secrets. We do not believe this attack was actually feasible. This -release closes the attack by first hashing the two strings to be compared -with a random secret. - -** webapi changes - -In most cases, HTML tracebacks will only be sent if an "Accept: text/html" -header was provided with the HTTP request. This will generally cause browsers -to get an HTMLized traceback but send regular text/plain tracebacks to -non-browsers (like the CLI clients). More errors have been mapped to useful -HTTP error codes. - -The streaming webapi operations (deep-check and manifest) now have a way to -indicate errors (an output line that starts with "ERROR" instead of being -legal JSON). See docs/frontends/webapi.txt for details. - -The storage server now has its own status page (at /storage), linked from the -Welcome page. This page shows progress and results of the two new -share-crawlers: one which merely counts shares (to give an estimate of how -many files/directories are being stored in the grid), the other examines -leases and reports how much space would be freed if GC were enabled. The page -also shows how much disk space is present, used, reserved, and available for -the Tahoe server, and whether the server is currently running in "read-write" -mode or "read-only" mode. - -When a directory node cannot be read (perhaps because of insufficent shares), -a minimal webapi page is created so that the "more-info" links (including a -Check/Repair operation) will still be accessible. - -A new "reliability" page was added, with the beginnings of work on a -statistical loss model. You can tell this page how many servers you are using -and their independent failure probabilities, and it will tell you the -likelihood that an arbitrary file will survive each repair period. The -"numpy" package must be installed to access this page. A partial paper, -written by Shawn Willden, has been added to docs/proposed/lossmodel.lyx . - -** CLI changes - -"tahoe check" and "tahoe deep-check" now accept an "--add-lease" argument, to -update a lease on all shares. This is the "mark" side of garbage collection. - -In many cases, CLI error messages have been improved: the ugly HTMLized -traceback has been replaced by a normal python traceback. - -"tahoe deep-check" and "tahoe manifest" now have better error reporting. -"tahoe cp" is now non-verbose by default. - -"tahoe backup" now accepts several "--exclude" arguments, to ignore certain -files (like editor temporary files and version-control metadata) during -backup. - -On windows, the CLI now accepts local paths like "c:\dir\file.txt", which -previously was interpreted as a Tahoe path using a "c:" alias. - -The "tahoe restart" command now uses "--force" by default (meaning it will -start a node even if it didn't look like there was one already running). - -The "tahoe debug consolidate" command was added. This takes a series of -independent timestamped snapshot directories (such as those created by the -allmydata.com windows backup program, or a series of "tahoe cp -r" commands) -and creates new snapshots that used shared read-only directories whenever -possible (like the output of "tahoe backup"). In the most common case (when -the snapshots are fairly similar), the result will use significantly fewer -directories than the original, allowing "deep-check" and similar tools to run -much faster. In some cases, the speedup can be an order of magnitude or more. -This tool is still somewhat experimental, and only needs to be run on large -backups produced by something other than "tahoe backup", so it was placed -under the "debug" category. - -"tahoe cp -r --caps-only tahoe:dir localdir" is a diagnostic tool which, -instead of copying the full contents of files into the local directory, -merely copies their filecaps. This can be used to verify the results of a -"consolidation" operation. - -** other fixes - -The codebase no longer rauses RuntimeError as a kind of assert(). Specific -exception classes were created for each previous instance of RuntimeError. - -Many unit tests were changed to use a non-network test harness, speeding them -up considerably. - -Deep-traversal operations (manifest and deep-check) now walk individual -directories in alphabetical order. Occasional turn breaks are inserted to -prevent a stack overflow when traversing directories with hundreds of -entries. - -The experimental SFTP server had its path-handling logic changed slightly, to -accomodate more SFTP clients, although there are still issues (#645). - - -* Release 1.3.0 (2009-02-13) - -** Checker/Verifier/Repairer - -The primary focus of this release has been writing a checker / verifier / -repairer for files and directories. "Checking" is the act of asking storage -servers whether they have a share for the given file or directory: if there -are not enough shares available, the file or directory will be -unrecoverable. "Verifying" is the act of downloading and cryptographically -asserting that the server's share is undamaged: it requires more work -(bandwidth and CPU) than checking, but can catch problems that simple -checking cannot. "Repair" is the act of replacing missing or damaged shares -with new ones. - -This release includes a full checker, a partial verifier, and a partial -repairer. The repairer is able to handle missing shares: new shares are -generated and uploaded to make up for the missing ones. This is currently the -best application of the repairer: to replace shares that were lost because of -server departure or permanent drive failure. - -The repairer in this release is somewhat able to handle corrupted shares. The -limitations are: - - * Immutable verifier is incomplete: not all shares are used, and not all - fields of those shares are verified. Therefore the immutable verifier has - only a moderate chance of detecting corrupted shares. - * The mutable verifier is mostly complete: all shares are examined, and most - fields of the shares are validated. - * The storage server protocol offers no way for the repairer to replace or - delete immutable shares. If corruption is detected, the repairer will - upload replacement shares to other servers, but the corrupted shares will - be left in place. - * read-only directories and read-only mutable files must be repaired by - someone who holds the write-cap: the read-cap is insufficient. Moreover, - the deep-check-and-repair operation will halt with an error if it attempts - to repair one of these read-only objects. - * Some forms of corruption can cause both download and repair operations to - fail. A future release will fix this, since download should be tolerant of - any corruption as long as there are at least 'k' valid shares, and repair - should be able to fix any file that is downloadable. - -If the downloader, verifier, or repairer detects share corruption, the -servers which provided the bad shares will be notified (via a file placed in -the BASEDIR/storage/corruption-advisories directory) so their operators can -manually delete the corrupted shares and investigate the problem. In -addition, the "incident gatherer" mechanism will automatically report share -corruption to an incident gatherer service, if one is configured. Note that -corrupted shares indicate hardware failures, serious software bugs, or malice -on the part of the storage server operator, so a corrupted share should be -considered highly unusual. - -By periodically checking/repairing all files and directories, objects in the -Tahoe filesystem remain resistant to recoverability failures due to missing -and/or broken servers. - -This release includes a wapi mechanism to initiate checks on individual -files and directories (with or without verification, and with or without -automatic repair). A related mechanism is used to initiate a "deep-check" on -a directory: recursively traversing the directory and its children, checking -(and/or verifying/repairing) everything underneath. Both mechanisms can be -run with an "output=JSON" argument, to obtain machine-readable check/repair -status results. These results include a copy of the filesystem statistics -from the "deep-stats" operation (including total number of files, size -histogram, etc). If repair is possible, a "Repair" button will appear on the -results page. - -The client web interface now features some extra buttons to initiate check -and deep-check operations. When these operations finish, they display a -results page that summarizes any problems that were encountered. All -long-running deep-traversal operations, including deep-check, use a -start-and-poll mechanism, to avoid depending upon a single long-lived HTTP -connection. docs/frontends/webapi.txt has details. - -** Efficient Backup - -The "tahoe backup" command is new in this release, which creates efficient -versioned backups of a local directory. Given a local pathname and a target -Tahoe directory, this will create a read-only snapshot of the local directory -in $target/Archives/$timestamp. It will also create $target/Latest, which is -a reference to the latest such snapshot. Each time you run "tahoe backup" -with the same source and target, a new $timestamp snapshot will be added. -These snapshots will share directories that have not changed since the last -backup, to speed up the process and minimize storage requirements. In -addition, a small database is used to keep track of which local files have -been uploaded already, to avoid uploading them a second time. This -drastically reduces the work needed to do a "null backup" (when nothing has -changed locally), making "tahoe backup' suitable to run from a daily cronjob. - -Note that the "tahoe backup" CLI command must be used in conjunction with a -1.3.0-or-newer Tahoe client node; there was a bug in the 1.2.0 webapi -implementation that would prevent the last step (create $target/Latest) from -working. - -** Large Files - -The 12GiB (approximate) immutable-file-size limitation is lifted. This -release knows how to handle so-called "v2 immutable shares", which permit -immutable files of up to about 18 EiB (about 3*10^14). These v2 shares are -created if the file to be uploaded is too large to fit into v1 shares. v1 -shares are created if the file is small enough to fit into them, so that -files created with tahoe-1.3.0 can still be read by earlier versions if they -are not too large. Note that storage servers also had to be changed to -support larger files, and this release is the first release in which they are -able to do that. Clients will detect which servers are capable of supporting -large files on upload and will not attempt to upload shares of a large file -to a server which doesn't support it. - -** FTP/SFTP Server - -Tahoe now includes experimental FTP and SFTP servers. When configured with a -suitable method to translate username+password into a root directory cap, it -provides simple access to the virtual filesystem. Remember that FTP is -completely unencrypted: passwords, filenames, and file contents are all sent -over the wire in cleartext, so FTP should only be used on a local (127.0.0.1) -connection. This feature is still in development: there are no unit tests -yet, and behavior with respect to Unicode filenames is uncertain. Please see -docs/frontends/FTP-and-SFTP.txt for configuration details. (#512, #531) - -** CLI Changes - -This release adds the 'tahoe create-alias' command, which is a combination of -'tahoe mkdir' and 'tahoe add-alias'. This also allows you to start using a -new tahoe directory without exposing its URI in the argv list, which is -publicly visible (through the process table) on most unix systems. Thanks to -Kevin Reid for bringing this issue to our attention. - -The single-argument form of "tahoe put" was changed to create an unlinked -file. I.e. "tahoe put bar.txt" will take the contents of a local "bar.txt" -file, upload them to the grid, and print the resulting read-cap; the file -will not be attached to any directories. This seemed a bit more useful than -the previous behavior (copy stdin, upload to the grid, attach the resulting -file into your default tahoe: alias in a child named 'bar.txt'). - -"tahoe put" was also fixed to handle mutable files correctly: "tahoe put -bar.txt URI:SSK:..." will read the contents of the local bar.txt and use them -to replace the contents of the given mutable file. - -The "tahoe webopen" command was modified to accept aliases. This means "tahoe -webopen tahoe:" will cause your web browser to open to a "wui" page that -gives access to the directory associated with the default "tahoe:" alias. It -should also accept leading slashes, like "tahoe webopen tahoe:/stuff". - -Many esoteric debugging commands were moved down into a "debug" subcommand: - - tahoe debug dump-cap - tahoe debug dump-share - tahoe debug find-shares - tahoe debug catalog-shares - tahoe debug corrupt-share - -The last command ("tahoe debug corrupt-share") flips a random bit of the -given local sharefile. This is used to test the file verifying/repairing -code, and obviously should not be used on user data. - -The cli might not correctly handle arguments which contain non-ascii -characters in Tahoe v1.3 (although depending on your platform it -might, especially if your platform can be configured to pass such -characters on the command-line in utf-8 encoding). See -http://tahoe-lafs.org/trac/tahoe/ticket/565 for details. - -** Web changes - -The "default webapi port", used when creating a new client node (and in the -getting-started documentation), was changed from 8123 to 3456, to reduce -confusion when Tahoe accessed through a Firefox browser on which the -"Torbutton" extension has been installed. Port 8123 is occasionally used as a -Tor control port, so Torbutton adds 8123 to Firefox's list of "banned ports" -to avoid CSRF attacks against Tor. Once 8123 is banned, it is difficult to -diagnose why you can no longer reach a Tahoe node, so the Tahoe default was -changed. Note that 3456 is reserved by IANA for the "vat" protocol, but there -are argueably more Torbutton+Tahoe users than vat users these days. Note that -this will only affect newly-created client nodes. Pre-existing client nodes, -created by earlier versions of tahoe, may still be listening on 8123. - -All deep-traversal operations (start-manifest, start-deep-size, -start-deep-stats, start-deep-check) now use a start-and-poll approach, -instead of using a single (fragile) long-running synchronous HTTP connection. -All these "start-" operations use POST instead of GET. The old "GET -manifest", "GET deep-size", and "POST deep-check" operations have been -removed. - -The new "POST start-manifest" operation, when it finally completes, results -in a table of (path,cap), instead of the list of verifycaps produced by the -old "GET manifest". The table is available in several formats: use -output=html, output=text, or output=json to choose one. The JSON output also -includes stats, and a list of verifycaps and storage-index strings. - -The "return_to=" and "when_done=" arguments have been removed from the -t=check and deep-check operations. - -The top-level status page (/status) now has a machine-readable form, via -"/status/?t=json". This includes information about the currently-active -uploads and downloads, which may be useful for frontends that wish to display -progress information. There is no easy way to correlate the activities -displayed here with recent wapi requests, however. - -Any files in BASEDIR/public_html/ (configurable) will be served in response -to requests in the /static/ portion of the URL space. This will simplify the -deployment of javascript-based frontends that can still access wapi calls -by conforming to the (regrettable) "same-origin policy". - -The welcome page now has a "Report Incident" button, which is tied into the -"Incident Gatherer" machinery. If the node is attached to an incident -gatherer (via log_gatherer.furl), then pushing this button will cause an -Incident to be signalled: this means recent log events are aggregated and -sent in a bundle to the gatherer. The user can push this button after -something strange takes place (and they can provide a short message to go -along with it), and the relevant data will be delivered to a centralized -incident-gatherer for later processing by operations staff. - -The "HEAD" method should now work correctly, in addition to the usual "GET", -"PUT", and "POST" methods. "HEAD" is supposed to return exactly the same -headers as "GET" would, but without any of the actual response body data. For -mutable files, this now does a brief mapupdate (to figure out the size of the -file that would be returned), without actually retrieving the file's -contents. - -The "GET" operation on files can now support the HTTP "Range:" header, -allowing requests for partial content. This allows certain media players to -correctly stream audio and movies out of a Tahoe grid. The current -implementation uses a disk-based cache in BASEDIR/private/cache/download , -which holds the plaintext of the files being downloaded. Future -implementations might not use this cache. GET for immutable files now returns -an ETag header. - -Each file and directory now has a "Show More Info" web page, which contains -much of the information that was crammed into the directory page before. This -includes readonly URIs, storage index strings, object type, buttons to -control checking/verifying/repairing, and deep-check/deep-stats buttons (for -directories). For mutable files, the "replace contents" upload form has been -moved here too. As a result, the directory page is now much simpler and -cleaner, and several potentially-misleading links (like t=uri) are now gone. - -Slashes are discouraged in Tahoe file/directory names, since they cause -problems when accessing the filesystem through the wapi. However, there are -a couple of accidental ways to generate such names. This release tries to -make it easier to correct such mistakes by escaping slashes in several -places, allowing slashes in the t=info and t=delete commands, and in the -source (but not the target) of a t=rename command. - -** Packaging - -Tahoe's dependencies have been extended to require the "[secure_connections]" -feature from Foolscap, which will cause pyOpenSSL to be required and/or -installed. If OpenSSL and its development headers are already installed on -your system, this can occur automatically. Tahoe now uses pollreactor -(instead of the default selectreactor) to work around a bug between pyOpenSSL -and the most recent release of Twisted (8.1.0). This bug only affects unit -tests (hang during shutdown), and should not impact regular use. - -The Tahoe source code tarballs now come in two different forms: regular and -"sumo". The regular tarball contains just Tahoe, nothing else. When building -from the regular tarball, the build process will download any unmet -dependencies from the internet (starting with the index at PyPI) so it can -build and install them. The "sumo" tarball contains copies of all the -libraries that Tahoe requires (foolscap, twisted, zfec, etc), so using the -"sumo" tarball should not require any internet access during the build -process. This can be useful if you want to build Tahoe while on an airplane, -a desert island, or other bandwidth-limited environments. - -Similarly, tahoe-lafs.org now hosts a "tahoe-deps" tarball which contains the -latest versions of all these dependencies. This tarball, located at -http://tahoe-lafs.org/source/tahoe/deps/tahoe-deps.tar.gz, can be unpacked in -the tahoe source tree (or in its parent directory), and the build process -should satisfy its downloading needs from it instead of reaching out to PyPI. -This can be useful if you want to build Tahoe from a darcs checkout while on -that airplane or desert island. - -Because of the previous two changes ("sumo" tarballs and the "tahoe-deps" -bundle), most of the files have been removed from misc/dependencies/ . This -brings the regular Tahoe tarball down to 2MB (compressed), and the darcs -checkout (without history) to about 7.6MB. A full darcs checkout will still -be fairly large (because of the historical patches which included the -dependent libraries), but a 'lazy' one should now be small. - -The default "make" target is now an alias for "setup.py build", which itself -is an alias for "setup.py develop --prefix support", with some extra work -before and after (see setup.cfg). Most of the complicated platform-dependent -code in the Makefile was rewritten in Python and moved into setup.py, -simplifying things considerably. - -Likewise, the "make test" target now delegates most of its work to "setup.py -test", which takes care of getting PYTHONPATH configured to access the tahoe -code (and dependencies) that gets put in support/lib/ by the build_tahoe -step. This should allow unit tests to be run even when trial (which is part -of Twisted) wasn't already installed (in this case, trial gets installed to -support/bin because Twisted is a dependency of Tahoe). - -Tahoe is now compatible with the recently-released Python 2.6 , although it -is recommended to use Tahoe on Python 2.5, on which it has received more -thorough testing and deployment. - -Tahoe is now compatible with simplejson-2.0.x . The previous release assumed -that simplejson.loads always returned unicode strings, which is no longer the -case in 2.0.x . - -** Grid Management Tools - -Several tools have been added or updated in the misc/ directory, mostly munin -plugins that can be used to monitor a storage grid. - -The misc/spacetime/ directory contains a "disk watcher" daemon (startable -with 'tahoe start'), which can be configured with a set of HTTP URLs -(pointing at the wapi '/statistics' page of a bunch of storage servers), -and will periodically fetch disk-used/disk-available information from all the -servers. It keeps this information in an Axiom database (a sqlite-based -library available from divmod.org). The daemon computes time-averaged rates -of disk usage, as well as a prediction of how much time is left before the -grid is completely full. - -The misc/munin/ directory contains a new set of munin plugins -(tahoe_diskleft, tahoe_diskusage, tahoe_doomsday) which talk to the -disk-watcher and provide graphs of its calculations. - -To support the disk-watcher, the Tahoe statistics component (visible through -the wapi at the /statistics/ URL) now includes disk-used and disk-available -information. Both are derived through an equivalent of the unix 'df' command -(i.e. they ask the kernel for the number of free blocks on the partition that -encloses the BASEDIR/storage directory). In the future, the disk-available -number will be further influenced by the local storage policy: if that policy -says that the server should refuse new shares when less than 5GB is left on -the partition, then "disk-available" will report zero even though the kernel -sees 5GB remaining. - -The 'tahoe_overhead' munin plugin interacts with an allmydata.com-specific -server which reports the total of the 'deep-size' reports for all active user -accounts, compares this with the disk-watcher data, to report on overhead -percentages. This provides information on how much space could be recovered -once Tahoe implements some form of garbage collection. - -** Configuration Changes: single INI-format tahoe.cfg file - -The Tahoe node is now configured with a single INI-format file, named -"tahoe.cfg", in the node's base directory. Most of the previous -multiple-separate-files are still read for backwards compatibility (the -embedded SSH debug server and the advertised_ip_addresses files are the -exceptions), but new directives will only be added to tahoe.cfg . The "tahoe -create-client" command will create a tahoe.cfg for you, with sample values -commented out. (ticket #518) - -tahoe.cfg now has controls for the foolscap "keepalive" and "disconnect" -timeouts (#521). - -tahoe.cfg now has controls for the encoding parameters: "shares.needed" and -"shares.total" in the "[client]" section. The default parameters are still -3-of-10. - -The inefficient storage 'sizelimit' control (which established an upper bound -on the amount of space that a storage server is allowed to consume) has been -replaced by a lightweight 'reserved_space' control (which establishes a lower -bound on the amount of remaining space). The storage server will reject all -writes that would cause the remaining disk space (as measured by a '/bin/df' -equivalent) to drop below this value. The "[storage]reserved_space=" -tahoe.cfg parameter controls this setting. (note that this only affects -immutable shares: it is an outstanding bug that reserved_space does not -prevent the allocation of new mutable shares, nor does it prevent the growth -of existing mutable shares). - -** Other Changes - -Clients now declare which versions of the protocols they support. This is -part of a new backwards-compatibility system: -http://tahoe-lafs.org/trac/tahoe/wiki/Versioning . - -The version strings for human inspection (as displayed on the Welcome web -page, and included in logs) now includes a platform identifer (frequently -including a linux distribution name, processor architecture, etc). - -Several bugs have been fixed, including one that would cause an exception (in -the logs) if a wapi download operation was cancelled (by closing the TCP -connection, or pushing the "stop" button in a web browser). - -Tahoe now uses Foolscap "Incidents", writing an "incident report" file to -logs/incidents/ each time something weird occurs. These reports are available -to an "incident gatherer" through the flogtool command. For more details, -please see the Foolscap logging documentation. An incident-classifying plugin -function is provided in misc/incident-gatherer/classify_tahoe.py . - -If clients detect corruption in shares, they now automatically report it to -the server holding that share, if it is new enough to accept the report. -These reports are written to files in BASEDIR/storage/corruption-advisories . - -The 'nickname' setting is now defined to be a UTF-8 -encoded string, allowing -non-ascii nicknames. - -The 'tahoe start' command will now accept a --syslog argument and pass it -through to twistd, making it easier to launch non-Tahoe nodes (like the -cpu-watcher) and have them log to syslogd instead of a local file. This is -useful when running a Tahoe node out of a USB flash drive. - -The Mac GUI in src/allmydata/gui/ has been improved. - - -* Release 1.2.0 (2008-07-21) - -** Security - -This release makes the immutable-file "ciphertext hash tree" mandatory. -Previous releases allowed the uploader to decide whether their file would -have an integrity check on the ciphertext or not. A malicious uploader could -use this to create a readcap that would download as one file or a different -one, depending upon which shares the client fetched first, with no errors -raised. There are other integrity checks on the shares themselves, preventing -a storage server or other party from violating the integrity properties of -the read-cap: this failure was only exploitable by the uploader who gives you -a carefully constructed read-cap. If you download the file with Tahoe 1.2.0 -or later, you will not be vulnerable to this problem. #491 - -This change does not introduce a compatibility issue, because all existing -versions of Tahoe will emit the ciphertext hash tree in their shares. - -** Dependencies - -Tahoe now requires Foolscap-0.2.9 . It also requires pycryptopp 0.5 or newer, -since earlier versions had a bug that interacted with specific compiler -versions that could sometimes result in incorrect encryption behavior. Both -packages are included in the Tahoe source tarball in misc/dependencies/ , and -should be built automatically when necessary. - -** Web API - -Web API directory pages should now contain properly-slash-terminated links to -other directories. They have also stopped using absolute links in forms and -pages (which interfered with the use of a front-end load-balancing proxy). - -The behavior of the "Check This File" button changed, in conjunction with -larger internal changes to file checking/verification. The button triggers an -immediate check as before, but the outcome is shown on its own page, and does -not get stored anywhere. As a result, the web directory page no longer shows -historical checker results. - -A new "Deep-Check" button has been added, which allows a user to initiate a -recursive check of the given directory and all files and directories -reachable from it. This can cause quite a bit of work, and has no -intermediate progress information or feedback about the process. In addition, -the results of the deep-check are extremely limited. A later release will -improve this behavior. - -The web server's behavior with respect to non-ASCII (unicode) filenames in -the "GET save=true" operation has been improved. To achieve maximum -compatibility with variously buggy web browsers, the server does not try to -figure out the character set of the inbound filename. It just echoes the same -bytes back to the browser in the Content-Disposition header. This seems to -make both IE7 and Firefox work correctly. - -** Checker/Verifier/Repairer - -Tahoe is slowly acquiring convenient tools to check up on file health, -examine existing shares for errors, and repair files that are not fully -healthy. This release adds a mutable checker/verifier/repairer, although -testing is very limited, and there are no web interfaces to trigger repair -yet. The "Check" button next to each file or directory on the wapi page -will perform a file check, and the "deep check" button on each directory will -recursively check all files and directories reachable from there (which may -take a very long time). - -Future releases will improve access to this functionality. - -** Operations/Packaging - -A "check-grid" script has been added, along with a Makefile target. This is -intended (with the help of a pre-configured node directory) to check upon the -health of a Tahoe grid, uploading and downloading a few files. This can be -used as a monitoring tool for a deployed grid, to be run periodically and to -signal an error if it ever fails. It also helps with compatibility testing, -to verify that the latest Tahoe code is still able to handle files created by -an older version. - -The munin plugins from misc/munin/ are now copied into any generated debian -packages, and are made executable (and uncompressed) so they can be symlinked -directly from /etc/munin/plugins/ . - -Ubuntu "Hardy" was added as a supported debian platform, with a Makefile -target to produce hardy .deb packages. Some notes have been added to -docs/debian.txt about building Tahoe on a debian/ubuntu system. - -Storage servers now measure operation rates and latency-per-operation, and -provides results through the /statistics web page as well as the stats -gatherer. Munin plugins have been added to match. - -** Other - -Tahoe nodes now use Foolscap "incident logging" to record unusual events to -their NODEDIR/logs/incidents/ directory. These incident files can be examined -by Foolscap logging tools, or delivered to an external log-gatherer for -further analysis. Note that Tahoe now requires Foolscap-0.2.9, since 0.2.8 -had a bug that complained about "OSError: File exists" when trying to create -the incidents/ directory for a second time. - -If no servers are available when retrieving a mutable file (like a -directory), the node now reports an error instead of hanging forever. Earlier -releases would not only hang (causing the wapi directory listing to get -stuck half-way through), but the internal dirnode serialization would cause -all subsequent attempts to retrieve or modify the same directory to hang as -well. #463 - -A minor internal exception (reported in logs/twistd.log, in the -"stopProducing" method) was fixed, which complained about "self._paused_at -not defined" whenever a file download was stopped from the web browser end. - - -* Release 1.1.0 (2008-06-11) - -** CLI: new "alias" model - -The new CLI code uses an scp/rsync -like interface, in which directories in -the Tahoe storage grid are referenced by a colon-suffixed alias. The new -commands look like: - tahoe cp local.txt tahoe:virtual.txt - tahoe ls work:subdir - -More functionality is available through the CLI: creating unlinked files and -directories, recursive copy in or out of the storage grid, hardlinks, and -retrieving the raw read- or write- caps through the 'ls' command. Please read -docs/CLI.txt for complete details. - -** wapi: new pages, new commands - -Several new pages were added to the web API: - - /helper_status : to describe what a Helper is doing - /statistics : reports node uptime, CPU usage, other stats - /file : for easy file-download URLs, see #221 - /cap == /uri : future compatibility - -The localdir=/localfile= and t=download operations were removed. These -required special configuration to enable anyways, but this feature was a -security problem, and was mostly obviated by the new "cp -r" command. - -Several new options to the GET command were added: - - t=deep-size : add up the size of all immutable files reachable from the directory - t=deep-stats : return a JSON-encoded description of number of files, size - distribution, total size, etc - -POST is now preferred over PUT for most operations which cause side-effects. - -Most wapi calls now accept overwrite=, and default to overwrite=true . - -"POST /uri/DIRCAP/parent/child?t=mkdir" is now the preferred API to create -multiple directories at once, rather than ...?t=mkdir-p . - -PUT to a mutable file ("PUT /uri/MUTABLEFILECAP", "PUT /uri/DIRCAP/child") -will modify the file in-place. - -** more munin graphs in misc/munin/ - - tahoe-introstats - tahoe-rootdir-space - tahoe_estimate_files - mutable files published/retrieved - tahoe_cpu_watcher - tahoe_spacetime - -** New Dependencies - - zfec 1.1.0 - foolscap 0.2.8 - pycryptopp 0.5 - setuptools (now required at runtime) - -** New Mutable-File Code - -The mutable-file handling code (mostly used for directories) has been -completely rewritten. The new scheme has a better API (with a modify() -method) and is less likely to lose data when several uncoordinated writers -change a file at the same time. - -In addition, a single Tahoe process will coordinate its own writes. If you -make two concurrent directory-modifying wapi calls to a single tahoe node, -it will internally make one of them wait for the other to complete. This -prevents auto-collision (#391). - -The new mutable-file code also detects errors during publish better. Earlier -releases might believe that a mutable file was published when in fact it -failed. - -** other features - -The node now monitors its own CPU usage, as a percentage, measured every 60 -seconds. 1/5/15 minute moving averages are available on the /statistics web -page and via the stats-gathering interface. - -Clients now accelerate reconnection to all servers after being offline -(#374). When a client is offline for a long time, it scales back reconnection -attempts to approximately once per hour, so it may take a while to make the -first attempt, but once any attempt succeeds, the other server connections -will be retried immediately. - -A new "offloaded KeyGenerator" facility can be configured, to move RSA key -generation out from, say, a wapi node, into a separate process. RSA keys -can take several seconds to create, and so a wapi node which is being used -for directory creation will be unavailable for anything else during this -time. The Key Generator process will pre-compute a small pool of keys, to -speed things up further. This also takes better advantage of multi-core CPUs, -or SMP hosts. - -The node will only use a potentially-slow "du -s" command at startup (to -measure how much space has been used) if the "sizelimit" parameter has been -configured (to limit how much space is used). Large storage servers should -turn off sizelimit until a later release improves the space-management code, -since "du -s" on a terabyte filesystem can take hours. - -The Introducer now allows new announcements to replace old ones, to avoid -buildups of obsolete announcements. - -Immutable files are limited to about 12GiB (when using the default 3-of-10 -encoding), because larger files would be corrupted by the four-byte -share-size field on the storage servers (#439). A later release will remove -this limit. Earlier releases would allow >12GiB uploads, but the resulting -file would be unretrievable. - -The docs/ directory has been rearranged, with old docs put in -docs/historical/ and not-yet-implemented ones in docs/proposed/ . - -The Mac OS-X FUSE plugin has a significant bug fix: earlier versions would -corrupt writes that used seek() instead of writing the file in linear order. -The rsync tool is known to perform writes in this order. This has been fixed. diff --git a/NEWS.rst b/NEWS.rst new file mode 100644 index 00000000..26e3f18e --- /dev/null +++ b/NEWS.rst @@ -0,0 +1,1579 @@ +================================== +User-Visible Changes in Tahoe-LAFS +================================== + +Release 1.8.2 (2011-01-30) +-------------------------- + +Compatibility and Dependencies +'''''''''''''''''''''''''''''' + +- Tahoe is now compatible with Twisted-10.2 (released last month), as + well as with earlier versions. The previous Tahoe-1.8.1 release + failed to run against Twisted-10.2, raising an AttributeError on + StreamServerEndpointService (`#1286`_) +- Tahoe now depends upon the "mock" testing library, and the foolscap + dependency was raised to 0.6.1 . It no longer requires pywin32 + (which was used only on windows). Future developers should note that + reactor.spawnProcess and derivatives may no longer be used inside + Tahoe code. + +Other Changes +''''''''''''' + +- the default reserved_space value for new storage nodes is 1 GB + (`#1208`_) +- documentation is now in reStructuredText (.rst) format +- "tahoe cp" should now handle non-ASCII filenames +- the unmaintained Mac/Windows GUI applications have been removed + (`#1282`_) +- tahoe processes should appear in top and ps as "tahoe", not + "python", on some unix platforms. (`#174`_) +- "tahoe debug trial" can be used to run the test suite (`#1296`_) +- the SFTP frontend now reports unknown sizes as "0" instead of "?", + to improve compatibility with clients like FileZilla (`#1337`_) +- "tahoe --version" should now report correct values in situations + where 1.8.1 might have been wrong (`#1287`_) + +.. _`#174`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/174 +.. _`#1208`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1208 +.. _`#1282`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1282 +.. _`#1286`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1286 +.. _`#1287`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1287 +.. _`#1296`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1296 +.. _`#1337`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1337 + + +Release 1.8.1 (2010-10-28) +-------------------------- + +Bugfixes and Improvements +''''''''''''''''''''''''' + +- Allow the repairer to improve the health of a file by uploading some + shares, even if it cannot achieve the configured happiness + threshold. This fixes a regression introduced between v1.7.1 and + v1.8.0. (`#1212`_) +- Fix a memory leak in the ResponseCache which is used during mutable + file/directory operations. (`#1045`_) +- Fix a regression and add a performance improvement in the + downloader. This issue caused repair to fail in some special + cases. (`#1223`_) +- Fix a bug that caused 'tahoe cp' to fail for a grid-to-grid copy + involving a non-ASCII filename. (`#1224`_) +- Fix a rarely-encountered bug involving printing large strings to the + console on Windows. (`#1232`_) +- Perform ~ expansion in the --exclude-from filename argument to + 'tahoe backup'. (`#1241`_) +- The CLI's 'tahoe mv' and 'tahoe ln' commands previously would try to + use an HTTP proxy if the HTTP_PROXY environment variable was set. + These now always connect directly to the WAPI, thus avoiding giving + caps to the HTTP proxy (and also avoiding failures in the case that + the proxy is failing or requires authentication). (`#1253`_) +- The CLI now correctly reports failure in the case that 'tahoe mv' + fails to unlink the file from its old location. (`#1255`_) +- 'tahoe start' now gives a more positive indication that the node has + started. (`#71`_) +- The arguments seen by 'ps' or other tools for node processes are now + more useful (in particular, they include the path of the 'tahoe' + script, rather than an obscure tool named 'twistd'). (`#174`_) + +Removed Features +'''''''''''''''' + +- The tahoe start/stop/restart and node creation commands no longer + accept the -m or --multiple option, for consistency between + platforms. (`#1262`_) + +Packaging +''''''''' + +- We now host binary packages so that users on certain operating + systems can install without having a compiler. + +- Use a newer version of a dependency if needed, even if an older + version is installed. This would previously cause a VersionConflict + error. (`#1190`_) +- Use a precompiled binary of a dependency if one with a sufficiently + high version number is available, instead of attempting to compile + the dependency from source, even if the source version has a higher + version number. (`#1233`_) + +Documentation +''''''''''''' + +- All current documentation in .txt format has been converted to .rst + format. (`#1225`_) +- Added docs/backdoors.rst declaring that we won't add backdoors to + Tahoe-LAFS, or add anything to facilitate government access to data. + (`#1216`_) + +.. _`#71`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/71 +.. _`#174`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/174 +.. _`#1212`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1212 +.. _`#1045`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1045 +.. _`#1190`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1190 +.. _`#1216`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1216 +.. _`#1223`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1223 +.. _`#1224`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1224 +.. _`#1225`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1225 +.. _`#1232`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1232 +.. _`#1233`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1233 +.. _`#1241`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1241 +.. _`#1253`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1253 +.. _`#1255`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1255 +.. _`#1262`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1262 + + +Release 1.8.0 (2010-09-23) +-------------------------- + +New Features +'''''''''''' + +- A completely new downloader which improves performance and + robustness of immutable-file downloads. It uses the fastest K + servers to download the data in K-way parallel. It automatically + fails over to alternate servers if servers fail in mid-download. It + allows seeking to arbitrary locations in the file (the previous + downloader which would only read the entire file sequentially from + beginning to end). It minimizes unnecessary round trips and + unnecessary bytes transferred to improve performance. It sends + requests to fewer servers to reduce the load on servers (the + previous one would send a small request to every server for every + download) (`#287`_, `#288`_, `#448`_, `#798`_, `#800`_, `#990`_, + `#1170`_, `#1191`_) +- Non-ASCII command-line arguments and non-ASCII outputs now work on + Windows. In addition, the command-line tool now works on 64-bit + Windows. (`#1074`_) + +Bugfixes and Improvements +''''''''''''''''''''''''' + +- Document and clean up the command-line options for specifying the + node's base directory. (`#188`_, `#706`_, `#715`_, `#772`_, + `#1108`_) +- The default node directory for Windows is ".tahoe" in the user's + home directory, the same as on other platforms. (`#890`_) +- Fix a case in which full cap URIs could be logged. (`#685`_, + `#1155`_) +- Fix bug in WUI in Python 2.5 when the system clock is set back to + 1969. Now you can use Tahoe-LAFS with Python 2.5 and set your system + clock to 1969 and still use the WUI. (`#1055`_) +- Many improvements in code organization, tests, logging, + documentation, and packaging. (`#983`_, `#1074`_, `#1108`_, + `#1127`_, `#1129`_, `#1131`_, `#1166`_, `#1175`_) + +Dependency Updates +'''''''''''''''''' + +- on x86 and x86-64 platforms, pycryptopp >= 0.5.20 +- pycrypto 2.2 is excluded due to a bug + +.. _`#188`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/188 +.. _`#287`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/287 +.. _`#288`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/288 +.. _`#448`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/448 +.. _`#685`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/685 +.. _`#706`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/706 +.. _`#715`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/715 +.. _`#772`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/772 +.. _`#798`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/798 +.. _`#800`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/800 +.. _`#890`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/890 +.. _`#983`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/983 +.. _`#990`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/990 +.. _`#1055`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1055 +.. _`#1074`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1074 +.. _`#1108`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1108 +.. _`#1155`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1155 +.. _`#1170`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170 +.. _`#1191`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1191 +.. _`#1127`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1127 +.. _`#1129`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 +.. _`#1131`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1131 +.. _`#1166`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1166 +.. _`#1175`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1175 + +Release 1.7.1 (2010-07-18) +-------------------------- + +Bugfixes and Improvements +''''''''''''''''''''''''' + +- Fix bug in which uploader could fail with AssertionFailure or report + that it had achieved servers-of-happiness when it hadn't. (`#1118`_) +- Fix bug in which servers could get into a state where they would + refuse to accept shares of a certain file (`#1117`_) +- Add init scripts for managing the gateway server on Debian/Ubuntu + (`#961`_) +- Fix bug where server version number was always 0 on the welcome page + (`#1067`_) +- Add new command-line command "tahoe unlink" as a synonym for "tahoe + rm" (`#776`_) +- The FTP frontend now encrypts its temporary files, protecting their + contents from an attacker who is able to read the disk. (`#1083`_) +- Fix IP address detection on FreeBSD 7, 8, and 9 (`#1098`_) +- Fix minor layout issue in the Web User Interface with Internet + Explorer (`#1097`_) +- Fix rarely-encountered incompatibility between Twisted logging + utility and the new unicode support added in v1.7.0 (`#1099`_) +- Forward-compatibility improvements for non-ASCII caps (`#1051`_) + +Code improvements +''''''''''''''''' + +- Simplify and tidy-up directories, unicode support, test code + (`#923`_, `#967`_, `#1072`_) + +.. _`#776`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/776 +.. _`#923`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/923 +.. _`#961`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/961 +.. _`#967`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/967 +.. _`#1051`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1051 +.. _`#1067`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1067 +.. _`#1072`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1072 +.. _`#1083`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1083 +.. _`#1097`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1097 +.. _`#1098`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1098 +.. _`#1099`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1099 +.. _`#1117`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1117 +.. _`#1118`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1118 + + +Release 1.7.0 (2010-06-18) +-------------------------- + +New Features +'''''''''''' + +- SFTP support (`#1037`_) + Your Tahoe-LAFS gateway now acts like a full-fledged SFTP server. It + has been tested with sshfs to provide a virtual filesystem in Linux. + Many users have asked for this feature. We hope that it serves them + well! See the `FTP-and-SFTP.rst`_ document to get + started. +- support for non-ASCII character encodings (`#534`_) + Tahoe-LAFS now correctly handles filenames containing non-ASCII + characters on all supported platforms: + + - when reading files in from the local filesystem (such as when you + run "tahoe backup" to back up your local files to a Tahoe-LAFS + grid); + - when writing files out to the local filesystem (such as when you + run "tahoe cp -r" to recursively copy files out of a Tahoe-LAFS + grid); + - when displaying filenames to the terminal (such as when you run + "tahoe ls"), subject to limitations of the terminal and locale; + - when parsing command-line arguments, except on Windows. + +- Servers of Happiness (`#778`_) + Tahoe-LAFS now measures during immutable file upload to see how well + distributed it is across multiple servers. It aborts the upload if + the pieces of the file are not sufficiently well-distributed. + This behavior is controlled by a configuration parameter called + "servers of happiness". With the default settings for its erasure + coding, Tahoe-LAFS generates 10 shares for each file, such that any + 3 of those shares are sufficient to recover the file. The default + value of "servers of happiness" is 7, which means that Tahoe-LAFS + will guarantee that there are at least 7 servers holding some of the + shares, such that any 3 of those servers can completely recover your + file. The new upload code also distributes the shares better than the + previous version in some cases and takes better advantage of + pre-existing shares (when a file has already been previously + uploaded). See the `architecture.rst`_ document [3] for details. + +Bugfixes and Improvements +''''''''''''''''''''''''' + +- Premature abort of upload if some shares were already present and + some servers fail. (`#608`_) +- python ./setup.py install -- can't create or remove files in install + directory. (`#803`_) +- Network failure => internal TypeError. (`#902`_) +- Install of Tahoe on CentOS 5.4. (`#933`_) +- CLI option --node-url now supports https url. (`#1028`_) +- HTML/CSS template files were not correctly installed under + Windows. (`#1033`_) +- MetadataSetter does not enforce restriction on setting "tahoe" + subkeys. (`#1034`_) +- ImportError: No module named + setuptools_darcs.setuptools_darcs. (`#1054`_) +- Renamed Title in xhtml files. (`#1062`_) +- Increase Python version dependency to 2.4.4, to avoid a critical + CPython security bug. (`#1066`_) +- Typo correction for the munin plugin tahoe_storagespace. (`#968`_) +- Fix warnings found by pylint. (`#973`_) +- Changing format of some documentation files. (`#1027`_) +- the misc/ directory was tied up. (`#1068`_) +- The 'ctime' and 'mtime' metadata fields are no longer written except + by "tahoe backup". (`#924`_) +- Unicode filenames in Tahoe-LAFS directories are normalized so that + names that differ only in how accents are encoded are treated as the + same. (`#1076`_) +- Various small improvements to documentation. (`#937`_, `#911`_, + `#1024`_, `#1082`_) + +Removals +'''''''' + +- The 'tahoe debug consolidate' subcommand (for converting old + allmydata Windows client backups to a newer format) has been + removed. + +Dependency Updates +'''''''''''''''''' + +- the Python version dependency is raised to 2.4.4 in some cases + (2.4.3 for Redhat-based Linux distributions, 2.4.2 for UCS-2 builds) + (`#1066`_) +- pycrypto >= 2.0.1 +- pyasn1 >= 0.0.8a +- mock (only required by unit tests) + +.. _`#534`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/534 +.. _`#608`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/608 +.. _`#778`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/778 +.. _`#803`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/803 +.. _`#902`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/902 +.. _`#911`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/911 +.. _`#924`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/924 +.. _`#937`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/937 +.. _`#933`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/933 +.. _`#968`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/968 +.. _`#973`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/973 +.. _`#1024`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1024 +.. _`#1027`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1027 +.. _`#1028`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1028 +.. _`#1033`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1033 +.. _`#1034`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1034 +.. _`#1037`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1037 +.. _`#1054`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1054 +.. _`#1062`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1062 +.. _`#1066`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1066 +.. _`#1068`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1068 +.. _`#1076`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1076 +.. _`#1082`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1082 +.. _architecture.rst: docs/architecture.rst +.. _FTP-and-SFTP.rst: docs/frontends/FTP-and-SFTP.rst + +Release 1.6.1 (2010-02-27) +-------------------------- + +Bugfixes +'''''''' + +- Correct handling of Small Immutable Directories + + Immutable directories can now be deep-checked and listed in the web + UI in all cases. (In v1.6.0, some operations, such as deep-check, on + a directory graph that included very small immutable directories, + would result in an exception causing the whole operation to abort.) + (`#948`_) + +Usability Improvements +'''''''''''''''''''''' + +- Improved user interface messages and error reporting. (`#681`_, + `#837`_, `#939`_) +- The timeouts for operation handles have been greatly increased, so + that you can view the results of an operation up to 4 days after it + has completed. After viewing them for the first time, the results + are retained for a further day. (`#577`_) + +Release 1.6.0 (2010-02-01) +-------------------------- + +New Features +'''''''''''' + +- Immutable Directories + + Tahoe-LAFS can now create and handle immutable + directories. (`#607`_, `#833`_, `#931`_) These are read just like + normal directories, but are "deep-immutable", meaning that all their + children (and everything reachable from those children) must be + immutable objects (i.e. immutable or literal files, and other + immutable directories). + + These directories must be created in a single webapi call that + provides all of the children at once. (Since they cannot be changed + after creation, the usual create/add/add sequence cannot be used.) + They have URIs that start with "URI:DIR2-CHK:" or "URI:DIR2-LIT:", + and are described on the human-facing web interface (aka the "WUI") + with a "DIR-IMM" abbreviation (as opposed to "DIR" for the usual + read-write directories and "DIR-RO" for read-only directories). + + Tahoe-LAFS releases before 1.6.0 cannot read the contents of an + immutable directory. 1.5.0 will tolerate their presence in a + directory listing (and display it as "unknown"). 1.4.1 and earlier + cannot tolerate them: a DIR-IMM child in any directory will prevent + the listing of that directory. + + Immutable directories are repairable, just like normal immutable + files. + + The webapi "POST t=mkdir-immutable" call is used to create immutable + directories. See `webapi.rst`_ for details. + +- "tahoe backup" now creates immutable directories, backupdb has + dircache + + The "tahoe backup" command has been enhanced to create immutable + directories (in previous releases, it created read-only mutable + directories) (`#828`_). This is significantly faster, since it does + not need to create an RSA keypair for each new directory. Also + "DIR-IMM" immutable directories are repairable, unlike "DIR-RO" + read-only mutable directories at present. (A future Tahoe-LAFS + release should also be able to repair DIR-RO.) + + In addition, the backupdb (used by "tahoe backup" to remember what + it has already copied) has been enhanced to store information about + existing immutable directories. This allows it to re-use directories + that have moved but still contain identical contents, or that have + been deleted and later replaced. (The 1.5.0 "tahoe backup" command + could only re-use directories that were in the same place as they + were in the immediately previous backup.) With this change, the + backup process no longer needs to read the previous snapshot out of + the Tahoe-LAFS grid, reducing the network load + considerably. (`#606`_) + + A "null backup" (in which nothing has changed since the previous + backup) will require only two Tahoe-side operations: one to add an + Archives/$TIMESTAMP entry, and a second to update the Latest/ + link. On the local disk side, it will readdir() all your local + directories and stat() all your local files. + + If you've been using "tahoe backup" for a while, you will notice + that your first use of it after upgrading to 1.6.0 may take a long + time: it must create proper immutable versions of all the old + read-only mutable directories. This process won't take as long as + the initial backup (where all the file contents had to be uploaded + too): it will require time proportional to the number and size of + your directories. After this initial pass, all subsequent passes + should take a tiny fraction of the time. + + As noted above, Tahoe-LAFS versions earlier than 1.5.0 cannot list a + directory containing an immutable subdirectory. Tahoe-LAFS versions + earlier than 1.6.0 cannot read the contents of an immutable + directory. + + The "tahoe backup" command has been improved to skip over unreadable + objects (like device files, named pipes, and files with permissions + that prevent the command from reading their contents), instead of + throwing an exception and terminating the backup process. It also + skips over symlinks, because these cannot be represented faithfully + in the Tahoe-side filesystem. A warning message will be emitted each + time something is skipped. (`#729`_, `#850`_, `#641`_) + +- "create-node" command added, "create-client" now implies + --no-storage + + The basic idea behind Tahoe-LAFS's client+server and client-only + processes is that you are creating a general-purpose Tahoe-LAFS + "node" process, which has several components that can be + activated. Storage service is one of these optional components, as + is the Helper, FTP server, and SFTP server. Web gateway + functionality is nominally on this list, but it is always active; a + future release will make it optional. There are three special + purpose servers that can't currently be run as a component in a + node: introducer, key-generator, and stats-gatherer. + + So now "tahoe create-node" will create a Tahoe-LAFS node process, + and after creation you can edit its tahoe.cfg to enable or disable + the desired services. It is a more general-purpose replacement for + "tahoe create-client". The default configuration has storage + service enabled. For convenience, the "--no-storage" argument makes + a tahoe.cfg file that disables storage service. (`#760`_) + + "tahoe create-client" has been changed to create a Tahoe-LAFS node + without a storage service. It is equivalent to "tahoe create-node + --no-storage". This helps to reduce the confusion surrounding the + use of a command with "client" in its name to create a storage + *server*. Use "tahoe create-client" to create a purely client-side + node. If you want to offer storage to the grid, use "tahoe + create-node" instead. + + In the future, other services will be added to the node, and they + will be controlled through options in tahoe.cfg . The most important + of these services may get additional --enable-XYZ or --disable-XYZ + arguments to "tahoe create-node". + +- Performance Improvements + + Download of immutable files begins as soon as the downloader has + located the K necessary shares (`#928`_, `#287`_). In both the + previous and current releases, a downloader will first issue queries + to all storage servers on the grid to locate shares before it begins + downloading the shares. In previous releases of Tahoe-LAFS, download + would not begin until all storage servers on the grid had replied to + the query, at which point K shares would be chosen for download from + among the shares that were located. In this release, download begins + as soon as any K shares are located. This means that downloads start + sooner, which is particularly important if there is a server on the + grid that is extremely slow or even hung in such a way that it will + never respond. In previous releases such a server would have a + negative impact on all downloads from that grid. In this release, + such a server will have no impact on downloads, as long as K shares + can be found on other, quicker, servers. This also means that + downloads now use the "best-alacrity" servers that they talk to, as + measured by how quickly the servers reply to the initial query. This + might cause downloads to go faster, especially on grids with + heterogeneous servers or geographical dispersion. + +Minor Changes +''''''''''''' + +- The webapi acquired a new "t=mkdir-with-children" command, to create + and populate a directory in a single call. This is significantly + faster than using separate "t=mkdir" and "t=set-children" operations + (it uses one gateway-to-grid roundtrip, instead of three or + four). (`#533`_) + +- The t=set-children (note the hyphen) operation is now documented in + webapi.rst, and is the new preferred spelling of the + old t=set_children (with an underscore). The underscore version + remains for backwards compatibility. (`#381`_, `#927`_) + +- The tracebacks produced by errors in CLI tools should now be in + plain text, instead of HTML (which is unreadable outside of a + browser). (`#646`_) + +- The [storage]reserved_space configuration knob (which causes the + storage server to refuse shares when available disk space drops + below a threshold) should work on Windows now, not just + UNIX. (`#637`_) + +- "tahoe cp" should now exit with status "1" if it cannot figure out a + suitable target filename, such as when you copy from a bare + filecap. (`#761`_) + +- "tahoe get" no longer creates a zero-length file upon + error. (`#121`_) + +- "tahoe ls" can now list single files. (`#457`_) + +- "tahoe deep-check --repair" should tolerate repair failures now, + instead of halting traversal. (`#874`_, `#786`_) + +- "tahoe create-alias" no longer corrupts the aliases file if it had + previously been edited to have no trailing newline. (`#741`_) + +- Many small packaging improvements were made to facilitate the + "tahoe-lafs" package being included in Ubuntu. Several mac/win32 + binary libraries were removed, some figleaf code-coverage files were + removed, a bundled copy of darcsver-1.2.1 was removed, and + additional licensing text was added. + +- Several DeprecationWarnings for python2.6 were silenced. (`#859`_) + +- The checker --add-lease option would sometimes fail for shares + stored on old (Tahoe v1.2.0) servers. (`#875`_) + +- The documentation for installing on Windows (docs/quickstart.rst) + has been improved. (`#773`_) + +For other changes not mentioned here, see +. +To include the tickets mentioned above, go to +. + +.. _`#121`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/121 +.. _`#287`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/287 +.. _`#381`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/381 +.. _`#457`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/457 +.. _`#533`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/533 +.. _`#577`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/577 +.. _`#606`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/606 +.. _`#607`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/607 +.. _`#637`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/637 +.. _`#641`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/641 +.. _`#646`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/646 +.. _`#681`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/681 +.. _`#729`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/729 +.. _`#741`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/741 +.. _`#760`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/760 +.. _`#761`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/761 +.. _`#768`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/768 +.. _`#773`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/773 +.. _`#786`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/786 +.. _`#828`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/828 +.. _`#833`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/833 +.. _`#859`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/859 +.. _`#874`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/874 +.. _`#875`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/875 +.. _`#931`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/931 +.. _`#837`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/837 +.. _`#850`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/850 +.. _`#927`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/927 +.. _`#928`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/928 +.. _`#939`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/939 +.. _`#948`: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/948 +.. _webapi.rst: docs/frontends/webapi.rst + +Release 1.5.0 (2009-08-01) +-------------------------- + +Improvements +'''''''''''' + +- Uploads of immutable files now use pipelined writes, improving + upload speed slightly (10%) over high-latency connections. (`#392`_) + +- Processing large directories has been sped up, by removing a O(N^2) + algorithm from the dirnode decoding path and retaining unmodified + encrypted entries. (`#750`_, `#752`_) + +- The human-facing web interface (aka the "WUI") received a + significant CSS makeover by Kevin Reid, making it much prettier and + easier to read. The WUI "check" and "deep-check" forms now include a + "Renew Lease" checkbox, mirroring the CLI --add-lease option, so + leases can be added or renewed from the web interface. + +- The CLI "tahoe mv" command now refuses to overwrite + directories. (`#705`_) + +- The CLI "tahoe webopen" command, when run without arguments, will + now bring up the "Welcome Page" (node status and mkdir/upload + forms). + +- The 3.5MB limit on mutable files was removed, so it should be + possible to upload arbitrarily-sized mutable files. Note, however, + that the data format and algorithm remains the same, so using + mutable files still requires bandwidth, computation, and RAM in + proportion to the size of the mutable file. (`#694`_) + +- This version of Tahoe-LAFS will tolerate directory entries that + contain filecap formats which it does not recognize: files and + directories from the future. This should improve the user + experience (for 1.5.0 users) when we add new cap formats in the + future. Previous versions would fail badly, preventing the user from + seeing or editing anything else in those directories. These + unrecognized objects can be renamed and deleted, but obviously not + read or written. Also they cannot generally be copied. (`#683`_) + +Bugfixes +'''''''' + +- deep-check-and-repair now tolerates read-only directories, such as + the ones produced by the "tahoe backup" CLI command. Read-only + directories and mutable files are checked, but not + repaired. Previous versions threw an exception when attempting the + repair and failed to process the remaining contents. We cannot yet + repair these read-only objects, but at least this version allows the + rest of the check+repair to proceed. (`#625`_) + +- A bug in 1.4.1 which caused a server to be listed multiple times + (and frequently broke all connections to that server) was + fixed. (`#653`_) + +- The plaintext-hashing code was removed from the Helper interface, + removing the Helper's ability to mount a + partial-information-guessing attack. (`#722`_) + +Platform/packaging changes +'''''''''''''''''''''''''' + +- Tahoe-LAFS now runs on NetBSD, OpenBSD, ArchLinux, and NixOS, and on + an embedded system based on an ARM CPU running at 266 MHz. + +- Unit test timeouts have been raised to allow the tests to complete + on extremely slow platforms like embedded ARM-based NAS boxes, which + may take several hours to run the test suite. An ARM-specific + data-corrupting bug in an older version of Crypto++ (5.5.2) was + identified: ARM-users are encouraged to use recent + Crypto++/pycryptopp which avoids this problem. + +- Tahoe-LAFS now requires a SQLite library, either the sqlite3 that + comes built-in with python2.5/2.6, or the add-on pysqlite2 if you're + using python2.4. In the previous release, this was only needed for + the "tahoe backup" command: now it is mandatory. + +- Several minor documentation updates were made. + +- To help get Tahoe-LAFS into Linux distributions like Fedora and + Debian, packaging improvements are being made in both Tahoe-LAFS and + related libraries like pycryptopp and zfec. + +- The Crypto++ library included in the pycryptopp package has been + upgraded to version 5.6.0 of Crypto++, which includes a more + efficient implementation of SHA-256 in assembly for x86 or amd64 + architectures. + +dependency updates +'''''''''''''''''' + +- foolscap-0.4.1 +- no python-2.4.0 or 2.4.1 (2.4.2 is good) (they contained a bug in base64.b32decode) +- avoid python-2.6 on windows with mingw: compiler issues +- python2.4 requires pysqlite2 (2.5,2.6 does not) +- no python-3.x +- pycryptopp-0.5.15 + +.. _#392: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/392 +.. _#625: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/625 +.. _#653: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/653 +.. _#683: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/683 +.. _#694: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/694 +.. _#705: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/705 +.. _#722: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/722 +.. _#750: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/750 +.. _#752: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/752 + +Release 1.4.1 (2009-04-13) +-------------------------- + +Garbage Collection +'''''''''''''''''' + +- The big feature for this release is the implementation of garbage + collection, allowing Tahoe storage servers to delete shares for old + deleted files. When enabled, this uses a "mark and sweep" process: + clients are responsible for updating the leases on their shares + (generally by running "tahoe deep-check --add-lease"), and servers + are allowed to delete any share which does not have an up-to-date + lease. The process is described in detail in + `garbage-collection.rst`_. + + The server must be configured to enable garbage-collection, by + adding directives to the [storage] section that define an age limit + for shares. The default configuration will not delete any shares. + + Both servers and clients should be upgraded to this release to make + the garbage-collection as pleasant as possible. 1.2.0 servers have + code to perform the update-lease operation but it suffers from a + fatal bug, while 1.3.0 servers have update-lease but will return an + exception for unknown storage indices, causing clients to emit an + Incident for each exception, slowing the add-lease process down to a + crawl. 1.1.0 servers did not have the add-lease operation at all. + +Security/Usability Problems Fixed +''''''''''''''''''''''''''''''''' + +- A super-linear algorithm in the Merkle Tree code was fixed, which + previously caused e.g. download of a 10GB file to take several hours + before the first byte of plaintext could be produced. The new + "alacrity" is about 2 minutes. A future release should reduce this + to a few seconds by fixing ticket `#442`_. + +- The previous version permitted a small timing attack (due to our use + of strcmp) against the write-enabler and lease-renewal/cancel + secrets. An attacker who could measure response-time variations of + approximatly 3ns against a very noisy background time of about 15ms + might be able to guess these secrets. We do not believe this attack + was actually feasible. This release closes the attack by first + hashing the two strings to be compared with a random secret. + +webapi changes +'''''''''''''' + +- In most cases, HTML tracebacks will only be sent if an "Accept: + text/html" header was provided with the HTTP request. This will + generally cause browsers to get an HTMLized traceback but send + regular text/plain tracebacks to non-browsers (like the CLI + clients). More errors have been mapped to useful HTTP error codes. + +- The streaming webapi operations (deep-check and manifest) now have a + way to indicate errors (an output line that starts with "ERROR" + instead of being legal JSON). See `webapi.rst`_ for + details. + +- The storage server now has its own status page (at /storage), linked + from the Welcome page. This page shows progress and results of the + two new share-crawlers: one which merely counts shares (to give an + estimate of how many files/directories are being stored in the + grid), the other examines leases and reports how much space would be + freed if GC were enabled. The page also shows how much disk space is + present, used, reserved, and available for the Tahoe server, and + whether the server is currently running in "read-write" mode or + "read-only" mode. + +- When a directory node cannot be read (perhaps because of insufficent + shares), a minimal webapi page is created so that the "more-info" + links (including a Check/Repair operation) will still be accessible. + +- A new "reliability" page was added, with the beginnings of work on a + statistical loss model. You can tell this page how many servers you + are using and their independent failure probabilities, and it will + tell you the likelihood that an arbitrary file will survive each + repair period. The "numpy" package must be installed to access this + page. A partial paper, written by Shawn Willden, has been added to + docs/proposed/lossmodel.lyx . + +CLI changes +''''''''''' + +- "tahoe check" and "tahoe deep-check" now accept an "--add-lease" + argument, to update a lease on all shares. This is the "mark" side + of garbage collection. + +- In many cases, CLI error messages have been improved: the ugly + HTMLized traceback has been replaced by a normal python traceback. + +- "tahoe deep-check" and "tahoe manifest" now have better error + reporting. "tahoe cp" is now non-verbose by default. + +- "tahoe backup" now accepts several "--exclude" arguments, to ignore + certain files (like editor temporary files and version-control + metadata) during backup. + +- On windows, the CLI now accepts local paths like "c:\dir\file.txt", + which previously was interpreted as a Tahoe path using a "c:" alias. + +- The "tahoe restart" command now uses "--force" by default (meaning + it will start a node even if it didn't look like there was one + already running). + +- The "tahoe debug consolidate" command was added. This takes a series + of independent timestamped snapshot directories (such as those + created by the allmydata.com windows backup program, or a series of + "tahoe cp -r" commands) and creates new snapshots that used shared + read-only directories whenever possible (like the output of "tahoe + backup"). In the most common case (when the snapshots are fairly + similar), the result will use significantly fewer directories than + the original, allowing "deep-check" and similar tools to run much + faster. In some cases, the speedup can be an order of magnitude or + more. This tool is still somewhat experimental, and only needs to + be run on large backups produced by something other than "tahoe + backup", so it was placed under the "debug" category. + +- "tahoe cp -r --caps-only tahoe:dir localdir" is a diagnostic tool + which, instead of copying the full contents of files into the local + directory, merely copies their filecaps. This can be used to verify + the results of a "consolidation" operation. + +other fixes +''''''''''' + +- The codebase no longer rauses RuntimeError as a kind of + assert(). Specific exception classes were created for each previous + instance of RuntimeError. + + -Many unit tests were changed to use a non-network test harness, + speeding them up considerably. + +- Deep-traversal operations (manifest and deep-check) now walk + individual directories in alphabetical order. Occasional turn breaks + are inserted to prevent a stack overflow when traversing directories + with hundreds of entries. + +- The experimental SFTP server had its path-handling logic changed + slightly, to accomodate more SFTP clients, although there are still + issues (`#645`_). + +.. _#442: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/442 +.. _#645: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/645 +.. _garbage-collection.rst: docs/garbage-collection.rst + +Release 1.3.0 (2009-02-13) +-------------------------- + +Checker/Verifier/Repairer +''''''''''''''''''''''''' + +- The primary focus of this release has been writing a checker / + verifier / repairer for files and directories. "Checking" is the + act of asking storage servers whether they have a share for the + given file or directory: if there are not enough shares available, + the file or directory will be unrecoverable. "Verifying" is the act + of downloading and cryptographically asserting that the server's + share is undamaged: it requires more work (bandwidth and CPU) than + checking, but can catch problems that simple checking + cannot. "Repair" is the act of replacing missing or damaged shares + with new ones. + +- This release includes a full checker, a partial verifier, and a + partial repairer. The repairer is able to handle missing shares: new + shares are generated and uploaded to make up for the missing + ones. This is currently the best application of the repairer: to + replace shares that were lost because of server departure or + permanent drive failure. + +- The repairer in this release is somewhat able to handle corrupted + shares. The limitations are: + + - Immutable verifier is incomplete: not all shares are used, and not + all fields of those shares are verified. Therefore the immutable + verifier has only a moderate chance of detecting corrupted shares. + - The mutable verifier is mostly complete: all shares are examined, + and most fields of the shares are validated. + - The storage server protocol offers no way for the repairer to + replace or delete immutable shares. If corruption is detected, the + repairer will upload replacement shares to other servers, but the + corrupted shares will be left in place. + - read-only directories and read-only mutable files must be repaired + by someone who holds the write-cap: the read-cap is + insufficient. Moreover, the deep-check-and-repair operation will + halt with an error if it attempts to repair one of these read-only + objects. + - Some forms of corruption can cause both download and repair + operations to fail. A future release will fix this, since download + should be tolerant of any corruption as long as there are at least + 'k' valid shares, and repair should be able to fix any file that is + downloadable. + +- If the downloader, verifier, or repairer detects share corruption, + the servers which provided the bad shares will be notified (via a + file placed in the BASEDIR/storage/corruption-advisories directory) + so their operators can manually delete the corrupted shares and + investigate the problem. In addition, the "incident gatherer" + mechanism will automatically report share corruption to an incident + gatherer service, if one is configured. Note that corrupted shares + indicate hardware failures, serious software bugs, or malice on the + part of the storage server operator, so a corrupted share should be + considered highly unusual. + +- By periodically checking/repairing all files and directories, + objects in the Tahoe filesystem remain resistant to recoverability + failures due to missing and/or broken servers. + +- This release includes a wapi mechanism to initiate checks on + individual files and directories (with or without verification, and + with or without automatic repair). A related mechanism is used to + initiate a "deep-check" on a directory: recursively traversing the + directory and its children, checking (and/or verifying/repairing) + everything underneath. Both mechanisms can be run with an + "output=JSON" argument, to obtain machine-readable check/repair + status results. These results include a copy of the filesystem + statistics from the "deep-stats" operation (including total number + of files, size histogram, etc). If repair is possible, a "Repair" + button will appear on the results page. + +- The client web interface now features some extra buttons to initiate + check and deep-check operations. When these operations finish, they + display a results page that summarizes any problems that were + encountered. All long-running deep-traversal operations, including + deep-check, use a start-and-poll mechanism, to avoid depending upon + a single long-lived HTTP connection. `webapi.rst`_ has + details. + +Efficient Backup +'''''''''''''''' + +- The "tahoe backup" command is new in this release, which creates + efficient versioned backups of a local directory. Given a local + pathname and a target Tahoe directory, this will create a read-only + snapshot of the local directory in $target/Archives/$timestamp. It + will also create $target/Latest, which is a reference to the latest + such snapshot. Each time you run "tahoe backup" with the same source + and target, a new $timestamp snapshot will be added. These snapshots + will share directories that have not changed since the last backup, + to speed up the process and minimize storage requirements. In + addition, a small database is used to keep track of which local + files have been uploaded already, to avoid uploading them a second + time. This drastically reduces the work needed to do a "null backup" + (when nothing has changed locally), making "tahoe backup' suitable + to run from a daily cronjob. + + Note that the "tahoe backup" CLI command must be used in conjunction + with a 1.3.0-or-newer Tahoe client node; there was a bug in the + 1.2.0 webapi implementation that would prevent the last step (create + $target/Latest) from working. + +Large Files +''''''''''' + +- The 12GiB (approximate) immutable-file-size limitation is + lifted. This release knows how to handle so-called "v2 immutable + shares", which permit immutable files of up to about 18 EiB (about + 3*10^14). These v2 shares are created if the file to be uploaded is + too large to fit into v1 shares. v1 shares are created if the file + is small enough to fit into them, so that files created with + tahoe-1.3.0 can still be read by earlier versions if they are not + too large. Note that storage servers also had to be changed to + support larger files, and this release is the first release in which + they are able to do that. Clients will detect which servers are + capable of supporting large files on upload and will not attempt to + upload shares of a large file to a server which doesn't support it. + +FTP/SFTP Server +''''''''''''''' + +- Tahoe now includes experimental FTP and SFTP servers. When + configured with a suitable method to translate username+password + into a root directory cap, it provides simple access to the virtual + filesystem. Remember that FTP is completely unencrypted: passwords, + filenames, and file contents are all sent over the wire in + cleartext, so FTP should only be used on a local (127.0.0.1) + connection. This feature is still in development: there are no unit + tests yet, and behavior with respect to Unicode filenames is + uncertain. Please see `FTP-and-SFTP.rst`_ for + configuration details. (`#512`_, `#531`_) + +CLI Changes +''''''''''' + +- This release adds the 'tahoe create-alias' command, which is a + combination of 'tahoe mkdir' and 'tahoe add-alias'. This also allows + you to start using a new tahoe directory without exposing its URI in + the argv list, which is publicly visible (through the process table) + on most unix systems. Thanks to Kevin Reid for bringing this issue + to our attention. + +- The single-argument form of "tahoe put" was changed to create an + unlinked file. I.e. "tahoe put bar.txt" will take the contents of a + local "bar.txt" file, upload them to the grid, and print the + resulting read-cap; the file will not be attached to any + directories. This seemed a bit more useful than the previous + behavior (copy stdin, upload to the grid, attach the resulting file + into your default tahoe: alias in a child named 'bar.txt'). + +- "tahoe put" was also fixed to handle mutable files correctly: "tahoe + put bar.txt URI:SSK:..." will read the contents of the local bar.txt + and use them to replace the contents of the given mutable file. + +- The "tahoe webopen" command was modified to accept aliases. This + means "tahoe webopen tahoe:" will cause your web browser to open to + a "wui" page that gives access to the directory associated with the + default "tahoe:" alias. It should also accept leading slashes, like + "tahoe webopen tahoe:/stuff". + +- Many esoteric debugging commands were moved down into a "debug" + subcommand: + + - tahoe debug dump-cap + - tahoe debug dump-share + - tahoe debug find-shares + - tahoe debug catalog-shares + - tahoe debug corrupt-share + + The last command ("tahoe debug corrupt-share") flips a random bit + of the given local sharefile. This is used to test the file + verifying/repairing code, and obviously should not be used on user + data. + +The cli might not correctly handle arguments which contain non-ascii +characters in Tahoe v1.3 (although depending on your platform it +might, especially if your platform can be configured to pass such +characters on the command-line in utf-8 encoding). See +http://tahoe-lafs.org/trac/tahoe/ticket/565 for details. + +Web changes +''''''''''' + +- The "default webapi port", used when creating a new client node (and + in the getting-started documentation), was changed from 8123 to + 3456, to reduce confusion when Tahoe accessed through a Firefox + browser on which the "Torbutton" extension has been installed. Port + 8123 is occasionally used as a Tor control port, so Torbutton adds + 8123 to Firefox's list of "banned ports" to avoid CSRF attacks + against Tor. Once 8123 is banned, it is difficult to diagnose why + you can no longer reach a Tahoe node, so the Tahoe default was + changed. Note that 3456 is reserved by IANA for the "vat" protocol, + but there are argueably more Torbutton+Tahoe users than vat users + these days. Note that this will only affect newly-created client + nodes. Pre-existing client nodes, created by earlier versions of + tahoe, may still be listening on 8123. + +- All deep-traversal operations (start-manifest, start-deep-size, + start-deep-stats, start-deep-check) now use a start-and-poll + approach, instead of using a single (fragile) long-running + synchronous HTTP connection. All these "start-" operations use POST + instead of GET. The old "GET manifest", "GET deep-size", and "POST + deep-check" operations have been removed. + +- The new "POST start-manifest" operation, when it finally completes, + results in a table of (path,cap), instead of the list of verifycaps + produced by the old "GET manifest". The table is available in + several formats: use output=html, output=text, or output=json to + choose one. The JSON output also includes stats, and a list of + verifycaps and storage-index strings. The "return_to=" and + "when_done=" arguments have been removed from the t=check and + deep-check operations. + +- The top-level status page (/status) now has a machine-readable form, + via "/status/?t=json". This includes information about the + currently-active uploads and downloads, which may be useful for + frontends that wish to display progress information. There is no + easy way to correlate the activities displayed here with recent wapi + requests, however. + +- Any files in BASEDIR/public_html/ (configurable) will be served in + response to requests in the /static/ portion of the URL space. This + will simplify the deployment of javascript-based frontends that can + still access wapi calls by conforming to the (regrettable) + "same-origin policy". + +- The welcome page now has a "Report Incident" button, which is tied + into the "Incident Gatherer" machinery. If the node is attached to + an incident gatherer (via log_gatherer.furl), then pushing this + button will cause an Incident to be signalled: this means recent log + events are aggregated and sent in a bundle to the gatherer. The user + can push this button after something strange takes place (and they + can provide a short message to go along with it), and the relevant + data will be delivered to a centralized incident-gatherer for later + processing by operations staff. + +- The "HEAD" method should now work correctly, in addition to the + usual "GET", "PUT", and "POST" methods. "HEAD" is supposed to return + exactly the same headers as "GET" would, but without any of the + actual response body data. For mutable files, this now does a brief + mapupdate (to figure out the size of the file that would be + returned), without actually retrieving the file's contents. + +- The "GET" operation on files can now support the HTTP "Range:" + header, allowing requests for partial content. This allows certain + media players to correctly stream audio and movies out of a Tahoe + grid. The current implementation uses a disk-based cache in + BASEDIR/private/cache/download , which holds the plaintext of the + files being downloaded. Future implementations might not use this + cache. GET for immutable files now returns an ETag header. + +- Each file and directory now has a "Show More Info" web page, which + contains much of the information that was crammed into the directory + page before. This includes readonly URIs, storage index strings, + object type, buttons to control checking/verifying/repairing, and + deep-check/deep-stats buttons (for directories). For mutable files, + the "replace contents" upload form has been moved here too. As a + result, the directory page is now much simpler and cleaner, and + several potentially-misleading links (like t=uri) are now gone. + +- Slashes are discouraged in Tahoe file/directory names, since they + cause problems when accessing the filesystem through the + wapi. However, there are a couple of accidental ways to generate + such names. This release tries to make it easier to correct such + mistakes by escaping slashes in several places, allowing slashes in + the t=info and t=delete commands, and in the source (but not the + target) of a t=rename command. + +Packaging +''''''''' + +- Tahoe's dependencies have been extended to require the + "[secure_connections]" feature from Foolscap, which will cause + pyOpenSSL to be required and/or installed. If OpenSSL and its + development headers are already installed on your system, this can + occur automatically. Tahoe now uses pollreactor (instead of the + default selectreactor) to work around a bug between pyOpenSSL and + the most recent release of Twisted (8.1.0). This bug only affects + unit tests (hang during shutdown), and should not impact regular + use. + +- The Tahoe source code tarballs now come in two different forms: + regular and "sumo". The regular tarball contains just Tahoe, nothing + else. When building from the regular tarball, the build process will + download any unmet dependencies from the internet (starting with the + index at PyPI) so it can build and install them. The "sumo" tarball + contains copies of all the libraries that Tahoe requires (foolscap, + twisted, zfec, etc), so using the "sumo" tarball should not require + any internet access during the build process. This can be useful if + you want to build Tahoe while on an airplane, a desert island, or + other bandwidth-limited environments. + +- Similarly, tahoe-lafs.org now hosts a "tahoe-deps" tarball which + contains the latest versions of all these dependencies. This + tarball, located at + http://tahoe-lafs.org/source/tahoe/deps/tahoe-deps.tar.gz, can be + unpacked in the tahoe source tree (or in its parent directory), and + the build process should satisfy its downloading needs from it + instead of reaching out to PyPI. This can be useful if you want to + build Tahoe from a darcs checkout while on that airplane or desert + island. + +- Because of the previous two changes ("sumo" tarballs and the + "tahoe-deps" bundle), most of the files have been removed from + misc/dependencies/ . This brings the regular Tahoe tarball down to + 2MB (compressed), and the darcs checkout (without history) to about + 7.6MB. A full darcs checkout will still be fairly large (because of + the historical patches which included the dependent libraries), but + a 'lazy' one should now be small. + +- The default "make" target is now an alias for "setup.py build", + which itself is an alias for "setup.py develop --prefix support", + with some extra work before and after (see setup.cfg). Most of the + complicated platform-dependent code in the Makefile was rewritten in + Python and moved into setup.py, simplifying things considerably. + +- Likewise, the "make test" target now delegates most of its work to + "setup.py test", which takes care of getting PYTHONPATH configured + to access the tahoe code (and dependencies) that gets put in + support/lib/ by the build_tahoe step. This should allow unit tests + to be run even when trial (which is part of Twisted) wasn't already + installed (in this case, trial gets installed to support/bin because + Twisted is a dependency of Tahoe). + +- Tahoe is now compatible with the recently-released Python 2.6 , + although it is recommended to use Tahoe on Python 2.5, on which it + has received more thorough testing and deployment. + +- Tahoe is now compatible with simplejson-2.0.x . The previous release + assumed that simplejson.loads always returned unicode strings, which + is no longer the case in 2.0.x . + +Grid Management Tools +''''''''''''''''''''' + +- Several tools have been added or updated in the misc/ directory, + mostly munin plugins that can be used to monitor a storage grid. + + - The misc/spacetime/ directory contains a "disk watcher" daemon + (startable with 'tahoe start'), which can be configured with a set + of HTTP URLs (pointing at the wapi '/statistics' page of a bunch of + storage servers), and will periodically fetch + disk-used/disk-available information from all the servers. It keeps + this information in an Axiom database (a sqlite-based library + available from divmod.org). The daemon computes time-averaged rates + of disk usage, as well as a prediction of how much time is left + before the grid is completely full. + + - The misc/munin/ directory contains a new set of munin plugins + (tahoe_diskleft, tahoe_diskusage, tahoe_doomsday) which talk to the + disk-watcher and provide graphs of its calculations. + + - To support the disk-watcher, the Tahoe statistics component + (visible through the wapi at the /statistics/ URL) now includes + disk-used and disk-available information. Both are derived through + an equivalent of the unix 'df' command (i.e. they ask the kernel + for the number of free blocks on the partition that encloses the + BASEDIR/storage directory). In the future, the disk-available + number will be further influenced by the local storage policy: if + that policy says that the server should refuse new shares when less + than 5GB is left on the partition, then "disk-available" will + report zero even though the kernel sees 5GB remaining. + + - The 'tahoe_overhead' munin plugin interacts with an + allmydata.com-specific server which reports the total of the + 'deep-size' reports for all active user accounts, compares this + with the disk-watcher data, to report on overhead percentages. This + provides information on how much space could be recovered once + Tahoe implements some form of garbage collection. + +Configuration Changes: single INI-format tahoe.cfg file +''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +- The Tahoe node is now configured with a single INI-format file, + named "tahoe.cfg", in the node's base directory. Most of the + previous multiple-separate-files are still read for backwards + compatibility (the embedded SSH debug server and the + advertised_ip_addresses files are the exceptions), but new + directives will only be added to tahoe.cfg . The "tahoe + create-client" command will create a tahoe.cfg for you, with sample + values commented out. (ticket `#518`_) + +- tahoe.cfg now has controls for the foolscap "keepalive" and + "disconnect" timeouts (`#521`_). + +- tahoe.cfg now has controls for the encoding parameters: + "shares.needed" and "shares.total" in the "[client]" section. The + default parameters are still 3-of-10. + +- The inefficient storage 'sizelimit' control (which established an + upper bound on the amount of space that a storage server is allowed + to consume) has been replaced by a lightweight 'reserved_space' + control (which establishes a lower bound on the amount of remaining + space). The storage server will reject all writes that would cause + the remaining disk space (as measured by a '/bin/df' equivalent) to + drop below this value. The "[storage]reserved_space=" tahoe.cfg + parameter controls this setting. (note that this only affects + immutable shares: it is an outstanding bug that reserved_space does + not prevent the allocation of new mutable shares, nor does it + prevent the growth of existing mutable shares). + +Other Changes +''''''''''''' + +- Clients now declare which versions of the protocols they + support. This is part of a new backwards-compatibility system: + http://tahoe-lafs.org/trac/tahoe/wiki/Versioning . + +- The version strings for human inspection (as displayed on the + Welcome web page, and included in logs) now includes a platform + identifer (frequently including a linux distribution name, processor + architecture, etc). + +- Several bugs have been fixed, including one that would cause an + exception (in the logs) if a wapi download operation was cancelled + (by closing the TCP connection, or pushing the "stop" button in a + web browser). + +- Tahoe now uses Foolscap "Incidents", writing an "incident report" + file to logs/incidents/ each time something weird occurs. These + reports are available to an "incident gatherer" through the flogtool + command. For more details, please see the Foolscap logging + documentation. An incident-classifying plugin function is provided + in misc/incident-gatherer/classify_tahoe.py . + +- If clients detect corruption in shares, they now automatically + report it to the server holding that share, if it is new enough to + accept the report. These reports are written to files in + BASEDIR/storage/corruption-advisories . + +- The 'nickname' setting is now defined to be a UTF-8 -encoded string, + allowing non-ascii nicknames. + +- The 'tahoe start' command will now accept a --syslog argument and + pass it through to twistd, making it easier to launch non-Tahoe + nodes (like the cpu-watcher) and have them log to syslogd instead of + a local file. This is useful when running a Tahoe node out of a USB + flash drive. + +- The Mac GUI in src/allmydata/gui/ has been improved. + +.. _#512: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/512 +.. _#518: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/518 +.. _#521: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/521 +.. _#531: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/531 + +Release 1.2.0 (2008-07-21) +-------------------------- + +Security +'''''''' + +- This release makes the immutable-file "ciphertext hash tree" + mandatory. Previous releases allowed the uploader to decide whether + their file would have an integrity check on the ciphertext or not. A + malicious uploader could use this to create a readcap that would + download as one file or a different one, depending upon which shares + the client fetched first, with no errors raised. There are other + integrity checks on the shares themselves, preventing a storage + server or other party from violating the integrity properties of the + read-cap: this failure was only exploitable by the uploader who + gives you a carefully constructed read-cap. If you download the file + with Tahoe 1.2.0 or later, you will not be vulnerable to this + problem. `#491`_ + + This change does not introduce a compatibility issue, because all + existing versions of Tahoe will emit the ciphertext hash tree in + their shares. + +Dependencies +'''''''''''' + +- Tahoe now requires Foolscap-0.2.9 . It also requires pycryptopp 0.5 + or newer, since earlier versions had a bug that interacted with + specific compiler versions that could sometimes result in incorrect + encryption behavior. Both packages are included in the Tahoe source + tarball in misc/dependencies/ , and should be built automatically + when necessary. + +Web API +''''''' + +- Web API directory pages should now contain properly-slash-terminated + links to other directories. They have also stopped using absolute + links in forms and pages (which interfered with the use of a + front-end load-balancing proxy). + +- The behavior of the "Check This File" button changed, in conjunction + with larger internal changes to file checking/verification. The + button triggers an immediate check as before, but the outcome is + shown on its own page, and does not get stored anywhere. As a + result, the web directory page no longer shows historical checker + results. + +- A new "Deep-Check" button has been added, which allows a user to + initiate a recursive check of the given directory and all files and + directories reachable from it. This can cause quite a bit of work, + and has no intermediate progress information or feedback about the + process. In addition, the results of the deep-check are extremely + limited. A later release will improve this behavior. + +- The web server's behavior with respect to non-ASCII (unicode) + filenames in the "GET save=true" operation has been improved. To + achieve maximum compatibility with variously buggy web browsers, the + server does not try to figure out the character set of the inbound + filename. It just echoes the same bytes back to the browser in the + Content-Disposition header. This seems to make both IE7 and Firefox + work correctly. + +Checker/Verifier/Repairer +''''''''''''''''''''''''' + +- Tahoe is slowly acquiring convenient tools to check up on file + health, examine existing shares for errors, and repair files that + are not fully healthy. This release adds a mutable + checker/verifier/repairer, although testing is very limited, and + there are no web interfaces to trigger repair yet. The "Check" + button next to each file or directory on the wapi page will perform + a file check, and the "deep check" button on each directory will + recursively check all files and directories reachable from there + (which may take a very long time). + + Future releases will improve access to this functionality. + +Operations/Packaging +'''''''''''''''''''' + +- A "check-grid" script has been added, along with a Makefile + target. This is intended (with the help of a pre-configured node + directory) to check upon the health of a Tahoe grid, uploading and + downloading a few files. This can be used as a monitoring tool for a + deployed grid, to be run periodically and to signal an error if it + ever fails. It also helps with compatibility testing, to verify that + the latest Tahoe code is still able to handle files created by an + older version. + +- The munin plugins from misc/munin/ are now copied into any generated + debian packages, and are made executable (and uncompressed) so they + can be symlinked directly from /etc/munin/plugins/ . + +- Ubuntu "Hardy" was added as a supported debian platform, with a + Makefile target to produce hardy .deb packages. Some notes have been + added to `debian.rst`_ about building Tahoe on a debian/ubuntu + system. + +- Storage servers now measure operation rates and + latency-per-operation, and provides results through the /statistics + web page as well as the stats gatherer. Munin plugins have been + added to match. + +Other +''''' + +- Tahoe nodes now use Foolscap "incident logging" to record unusual + events to their NODEDIR/logs/incidents/ directory. These incident + files can be examined by Foolscap logging tools, or delivered to an + external log-gatherer for further analysis. Note that Tahoe now + requires Foolscap-0.2.9, since 0.2.8 had a bug that complained about + "OSError: File exists" when trying to create the incidents/ + directory for a second time. + +- If no servers are available when retrieving a mutable file (like a + directory), the node now reports an error instead of hanging + forever. Earlier releases would not only hang (causing the wapi + directory listing to get stuck half-way through), but the internal + dirnode serialization would cause all subsequent attempts to + retrieve or modify the same directory to hang as well. `#463`_ + +- A minor internal exception (reported in logs/twistd.log, in the + "stopProducing" method) was fixed, which complained about + "self._paused_at not defined" whenever a file download was stopped + from the web browser end. + +.. _#463: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/463 +.. _#491: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/491 +.. _debian.rst: docs/debian.rst + +Release 1.1.0 (2008-06-11) +-------------------------- + +CLI: new "alias" model +'''''''''''''''''''''' + +- The new CLI code uses an scp/rsync -like interface, in which + directories in the Tahoe storage grid are referenced by a + colon-suffixed alias. The new commands look like: + + - tahoe cp local.txt tahoe:virtual.txt + - tahoe ls work:subdir + +- More functionality is available through the CLI: creating unlinked + files and directories, recursive copy in or out of the storage grid, + hardlinks, and retrieving the raw read- or write- caps through the + 'ls' command. Please read `CLI.rst`_ for complete details. + +wapi: new pages, new commands +''''''''''''''''''''''''''''' + +- Several new pages were added to the web API: + + - /helper_status : to describe what a Helper is doing + - /statistics : reports node uptime, CPU usage, other stats + - /file : for easy file-download URLs, see `#221`_ + - /cap == /uri : future compatibility + +- The localdir=/localfile= and t=download operations were + removed. These required special configuration to enable anyways, but + this feature was a security problem, and was mostly obviated by the + new "cp -r" command. + +- Several new options to the GET command were added: + + - t=deep-size : add up the size of all immutable files reachable from the directory + - t=deep-stats : return a JSON-encoded description of number of files, size distribution, total size, etc + +- POST is now preferred over PUT for most operations which cause + side-effects. + +- Most wapi calls now accept overwrite=, and default to overwrite=true + +- "POST /uri/DIRCAP/parent/child?t=mkdir" is now the preferred API to + create multiple directories at once, rather than ...?t=mkdir-p . + +- PUT to a mutable file ("PUT /uri/MUTABLEFILECAP", "PUT + /uri/DIRCAP/child") will modify the file in-place. + +- more munin graphs in misc/munin/ + + - tahoe-introstats + - tahoe-rootdir-space + - tahoe_estimate_files + - mutable files published/retrieved + - tahoe_cpu_watcher + - tahoe_spacetime + +New Dependencies +'''''''''''''''' +- zfec 1.1.0 +- foolscap 0.2.8 +- pycryptopp 0.5 +- setuptools (now required at runtime) + +New Mutable-File Code +''''''''''''''''''''' + +- The mutable-file handling code (mostly used for directories) has + been completely rewritten. The new scheme has a better API (with a + modify() method) and is less likely to lose data when several + uncoordinated writers change a file at the same time. + +- In addition, a single Tahoe process will coordinate its own + writes. If you make two concurrent directory-modifying wapi calls to + a single tahoe node, it will internally make one of them wait for + the other to complete. This prevents auto-collision (`#391`_). + +- The new mutable-file code also detects errors during publish + better. Earlier releases might believe that a mutable file was + published when in fact it failed. + +other features +'''''''''''''' + +- The node now monitors its own CPU usage, as a percentage, measured + every 60 seconds. 1/5/15 minute moving averages are available on the + /statistics web page and via the stats-gathering interface. + +- Clients now accelerate reconnection to all servers after being + offline (`#374`_). When a client is offline for a long time, it + scales back reconnection attempts to approximately once per hour, so + it may take a while to make the first attempt, but once any attempt + succeeds, the other server connections will be retried immediately. + +- A new "offloaded KeyGenerator" facility can be configured, to move + RSA key generation out from, say, a wapi node, into a separate + process. RSA keys can take several seconds to create, and so a wapi + node which is being used for directory creation will be unavailable + for anything else during this time. The Key Generator process will + pre-compute a small pool of keys, to speed things up further. This + also takes better advantage of multi-core CPUs, or SMP hosts. + +- The node will only use a potentially-slow "du -s" command at startup + (to measure how much space has been used) if the "sizelimit" + parameter has been configured (to limit how much space is + used). Large storage servers should turn off sizelimit until a later + release improves the space-management code, since "du -s" on a + terabyte filesystem can take hours. + +- The Introducer now allows new announcements to replace old ones, to + avoid buildups of obsolete announcements. + +- Immutable files are limited to about 12GiB (when using the default + 3-of-10 encoding), because larger files would be corrupted by the + four-byte share-size field on the storage servers (`#439`_). A later + release will remove this limit. Earlier releases would allow >12GiB + uploads, but the resulting file would be unretrievable. + +- The docs/ directory has been rearranged, with old docs put in + docs/historical/ and not-yet-implemented ones in docs/proposed/ . + +- The Mac OS-X FUSE plugin has a significant bug fix: earlier versions + would corrupt writes that used seek() instead of writing the file in + linear order. The rsync tool is known to perform writes in this + order. This has been fixed. + +.. _#221: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/221 +.. _#374: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/374 +.. _#391: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/391 +.. _#439: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/439 +.. _CLI.rst: docs/CLI.rst diff --git a/docs/how_to_make_a_tahoe-lafs_release.rst b/docs/how_to_make_a_tahoe-lafs_release.rst index c8f54b25..a77ef356 100644 --- a/docs/how_to_make_a_tahoe-lafs_release.rst +++ b/docs/how_to_make_a_tahoe-lafs_release.rst @@ -1,4 +1,4 @@ -[ ] 1 update doc files: `<../relnotes.txt>`_, `<../CREDITS>`_, ``_, `<../NEWS>`_. Add release name and date to top-most item in NEWS. +[ ] 1 update doc files: `<../relnotes.txt>`_, `<../CREDITS>`_, ``_, `<../NEWS.rst>`_. Add release name and date to top-most item in NEWS. [ ] 2 change ``_ to point to just the current allmydata-tahoe-X.Y.Z.zip source code file, or else to point to a directory which contains only allmydata-tahoe-X.Y.Z.* source code files diff --git a/misc/debian/rules.sid b/misc/debian/rules.sid index 6c65b746..53afdd95 100644 --- a/misc/debian/rules.sid +++ b/misc/debian/rules.sid @@ -14,7 +14,7 @@ DEBNAME := $(firstword $(DEB_PACKAGES)) STAGING_DIR := $(CURDIR)/debian/$(DEBNAME) -DEB_INSTALL_DOCS_ALL := COPYING.TGPPL.html CREDITS NEWS README.txt relnotes.txt \ +DEB_INSTALL_DOCS_ALL := COPYING.TGPPL.html CREDITS NEWS.rst README.txt relnotes.txt \ docs misc/operations_helpers/spacetime misc/operations_helpers/cpu-watcher.tac DEB_COMPRESS_EXCLUDE := .tac diff --git a/misc/debian_helpers/etch/debian/rules b/misc/debian_helpers/etch/debian/rules index 9c07c564..f07d54b9 100644 --- a/misc/debian_helpers/etch/debian/rules +++ b/misc/debian_helpers/etch/debian/rules @@ -15,7 +15,7 @@ DEBNAME := $(firstword $(DEB_PACKAGES)) STAGING_DIR := $(CURDIR)/debian/$(DEBNAME) DEB_INSTALL_DOCS_ALL := COPYING.GPL COPYING.TGPPL.html CREDITS \ - NEWS README.txt relnotes.txt docs misc/operations_helpers/spacetime misc/operations_helpers/cpu-watcher.tac + NEWS.rst README.txt relnotes.txt docs misc/operations_helpers/spacetime misc/operations_helpers/cpu-watcher.tac DEB_COMPRESS_EXCLUDE := .tac diff --git a/misc/debian_helpers/lenny/debian/rules b/misc/debian_helpers/lenny/debian/rules index 9c07c564..f07d54b9 100644 --- a/misc/debian_helpers/lenny/debian/rules +++ b/misc/debian_helpers/lenny/debian/rules @@ -15,7 +15,7 @@ DEBNAME := $(firstword $(DEB_PACKAGES)) STAGING_DIR := $(CURDIR)/debian/$(DEBNAME) DEB_INSTALL_DOCS_ALL := COPYING.GPL COPYING.TGPPL.html CREDITS \ - NEWS README.txt relnotes.txt docs misc/operations_helpers/spacetime misc/operations_helpers/cpu-watcher.tac + NEWS.rst README.txt relnotes.txt docs misc/operations_helpers/spacetime misc/operations_helpers/cpu-watcher.tac DEB_COMPRESS_EXCLUDE := .tac diff --git a/misc/debian_helpers/sid/debian/rules b/misc/debian_helpers/sid/debian/rules index 9c07c564..f07d54b9 100644 --- a/misc/debian_helpers/sid/debian/rules +++ b/misc/debian_helpers/sid/debian/rules @@ -15,7 +15,7 @@ DEBNAME := $(firstword $(DEB_PACKAGES)) STAGING_DIR := $(CURDIR)/debian/$(DEBNAME) DEB_INSTALL_DOCS_ALL := COPYING.GPL COPYING.TGPPL.html CREDITS \ - NEWS README.txt relnotes.txt docs misc/operations_helpers/spacetime misc/operations_helpers/cpu-watcher.tac + NEWS.rst README.txt relnotes.txt docs misc/operations_helpers/spacetime misc/operations_helpers/cpu-watcher.tac DEB_COMPRESS_EXCLUDE := .tac diff --git a/relnotes.txt b/relnotes.txt index 670496c9..2f94f154 100644 --- a/relnotes.txt +++ b/relnotes.txt @@ -19,7 +19,7 @@ released October 28, 2010 [1]. v1.8.2 is a stable bugfix release, adding compatibility with the recently-released Twisted-10.2, and correcting a number of -minor issues. See the NEWS file [2] for details. +minor issues. See the NEWS.rst file [2] for details. WHAT IS IT GOOD FOR? @@ -145,7 +145,7 @@ San Francisco, California, USA [1] http://tahoe-lafs.org/trac/tahoe/browser/relnotes.txt?rev=4865 -[2] http://tahoe-lafs.org/trac/tahoe/browser/NEWS?rev=5000 +[2] http://tahoe-lafs.org/trac/tahoe/browser/NEWS.rst?rev=5000 [3] http://tahoe-lafs.org/trac/tahoe/wiki/RelatedProjects [4] http://tahoe-lafs.org/trac/tahoe/browser/docs/known_issues.rst [5] http://tahoe-lafs.org/trac/tahoe/browser/COPYING.GPL diff --git a/src/allmydata/test/test_repairer.py b/src/allmydata/test/test_repairer.py index 942d3272..7efb4a82 100644 --- a/src/allmydata/test/test_repairer.py +++ b/src/allmydata/test/test_repairer.py @@ -553,7 +553,7 @@ class Repairer(GridTestMixin, unittest.TestCase, RepairTestMixin, # why is test_repair_from_corruption_of_1 disabled? Read on: # - # As recently documented in NEWS for the 1.3.0 release, the current + # As recently documented in NEWS.rst for the 1.3.0 release, the current # immutable repairer suffers from several limitations: # # * minimalistic verifier: it's just download without decryption, so we -- 2.37.2