-This work also comes with the added permission that you may combine it with a
+.. -*- coding: utf-8-with-signature -*-
+
+This work also comes with the added permission that you may combine it with a
work licensed under the OpenSSL license (any version) and distribute the
resulting combined work, as long as you follow the requirements of the
licences of this work in regard to all of the resulting combined work
-==================================
+.. -*- coding: utf-8-with-signature -*-
+
+==================================
User-Visible Changes in Tahoe-LAFS
==================================
-
-
-.. -*- coding: utf-8 -*-
+.. -*- coding: utf-8-with-signature -*-
Welcome to Tahoe-LAFS!
======================
+.. -*- coding: utf-8-with-signature -*-
+
=======================
Tahoe-LAFS Architecture
=======================
Overview
========
-(See the `docs/specifications directory <specifications>`_ for more details.)
+(See the `docs/specifications directory`_ for more details.)
There are three layers: the key-value store, the filesystem, and the
application.
copies files from the local disk onto the decentralized filesystem. We later
provide read-only access to those files, allowing users to recover them.
There are several other applications built on top of the Tahoe-LAFS
-filesystem (see the `RelatedProjects
-<https://tahoe-lafs.org/trac/tahoe-lafs/wiki/RelatedProjects>`_ page of the
-wiki for a list).
+filesystem (see the RelatedProjects_ page of the wiki for a list).
+.. _docs/specifications directory: specifications
+.. _RelatedProjects: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/RelatedProjects
The Key-Value Store
===================
facility with high interconnect bandwidth. In this case, the helper is placed
in the same facility, so the helper-to-storage-server bandwidth is huge.
-See `<helper.rst>`_ for details about the upload helper.
+See helper.rst_ for details about the upload helper.
+
+.. _helper.rst: helper.rst
The Filesystem Layer
least frequently enough to prevent any of the leases from expiring before the
next renewal pass.
-See `<garbage-collection.rst>`_ for further information, and for how to
-configure garbage collection.
+See garbage-collection.rst_ for further information, and for how to configure
+garbage collection.
+
+.. _garbage-collection.rst: garbage-collection.rst
File Repairer
+.. -*- coding: utf-8-with-signature -*-
+
Statement on Backdoors
======================
+.. -*- coding: utf-8-with-signature -*-
+
==================
The Tahoe BackupDB
==================
-
+.. -*- coding: utf-8-with-signature -*-
+
=======================================================
Things To Be Careful About As We Venture Boldly Forth
=======================================================
+.. -*- coding: utf-8-with-signature -*-
+
=============================
Configuring a Tahoe-LAFS node
=============================
a Twisted listening-port specification string, like "``tcp:80``" or
"``tcp:3456:interface=127.0.0.1``". For a full description of the format,
- see `the Twisted strports documentation
- <https://twistedmatrix.com/documents/current/api/twisted.application.strports.html>`_.
- Please note, if interface= is not specified, Tahoe-LAFS will attempt to
- bind the port specified on all interfaces.
+ see `the Twisted strports documentation`_. Please note, if interface= is
+ not specified, Tahoe-LAFS will attempt to bind the port specified on all
+ interfaces.
``FURL string``
a Foolscap endpoint identifier, like
``pb://soklj4y7eok5c3xkmjeqpw@192.168.69.247:44801/eqpwqtzm``
+.. _the Twisted strports documentation: https://twistedmatrix.com/documents/current/api/twisted.application.strports.html
+
Node Types
==========
This controls where the node's web server should listen, providing node
status and, if the node is a client/server, providing web-API service as
- defined in `webapi.rst <frontends/webapi.rst>_`.
+ defined in webapi.rst_.
This file contains a Twisted "strports" specification such as "``3456``"
or "``tcp:3456:interface=127.0.0.1``". The "``tahoe create-node``" or
used for files that usually (on a Unix system) go into ``/tmp``. The
string will be interpreted relative to the node's base directory.
+.. _webapi.rst: frontends/webapi.rst
+
Client Configuration
====================
``helper.furl = (FURL string, optional)``
If provided, the node will attempt to connect to and use the given helper
- for uploads. See `<helper.rst>`_ for details.
+ for uploads. See helper.rst_ for details.
``key_generator.furl = (FURL string, optional)``
ratios are more reliable, and small ``N``/``k`` ratios use less disk
space. ``N`` cannot be larger than 256, because of the 8-bit
erasure-coding algorithm that Tahoe-LAFS uses. ``k`` can not be greater
- than ``N``. See `<performance.rst>`_ for more details.
+ than ``N``. See performance.rst_ for more details.
``shares.happy`` allows you control over how well to "spread out" the
shares of an immutable file. For a successful upload, shares are
controlled by this parameter and will always use SDMF. We may revisit
this decision in future versions of Tahoe-LAFS.
- See `<specifications/mutable.rst>`_ for details about mutable file
- formats.
+ See mutable.rst_ for details about mutable file formats.
+
+.. _helper.rst: helper.rst
+.. _performance.rst: performance.rst
+.. _mutable.rst: specifications/mutable.rst
Frontend Configuration
======================
directories and files, as well as a number of pages to check on the
status of your Tahoe node. It also provides a machine-oriented "WAPI",
with a REST-ful HTTP interface that can be used by other programs
- (including the CLI tools). Please see `<frontends/webapi.rst>`_ for full
- details, and the ``web.port`` and ``web.static`` config variables above.
- The `<frontends/download-status.rst>`_ document also describes a few WUI
- status pages.
+ (including the CLI tools). Please see webapi.rst_ for full details, and
+ the ``web.port`` and ``web.static`` config variables above. The
+ `download-status.rst`_ document also describes a few WUI status pages.
CLI
The main "bin/tahoe" executable includes subcommands for manipulating the
filesystem, uploading/downloading files, and creating/running Tahoe
- nodes. See `<frontends/CLI.rst>`_ for details.
+ nodes. See CLI.rst_ for details.
SFTP, FTP
Tahoe can also run both SFTP and FTP servers, and map a username/password
- pair to a top-level Tahoe directory. See `<frontends/FTP-and-SFTP.rst>`_
- for instructions on configuring these services, and the ``[sftpd]`` and
+ pair to a top-level Tahoe directory. See FTP-and-SFTP.rst_ for
+ instructions on configuring these services, and the ``[sftpd]`` and
``[ftpd]`` sections of ``tahoe.cfg``.
Drop-Upload
As of Tahoe-LAFS v1.9.0, a node running on Linux can be configured to
automatically upload files that are created or changed in a specified
- local directory. See `<frontends/drop-upload.rst>`_ for details.
+ local directory. See drop-upload.rst_ for details.
+.. _download-status.rst: frontends/download-status.rst
+.. _CLI.rst: frontends/CLI.rst
+.. _FTP-and-SFTP.rst: frontends/FTP-and-SFTP.rst
+.. _drop-upload.rst: frontends/drop-upload.rst
Storage Server Configuration
that are being decommissioned: the ``storage/`` directory could be
mounted read-only, while shares are moved to other servers. Note that
this currently only affects immutable shares. Mutable shares (used for
- directories) will be written and modified anyway. See ticket `#390
- <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/390>`_ for the current
- status of this bug. The default value is ``False``.
+ directories) will be written and modified anyway. See ticket `#390`_ for
+ the current status of this bug. The default value is ``False``.
``reserved_space = (str, optional)``
These settings control garbage collection, in which the server will
delete shares that no longer have an up-to-date lease on them. Please see
- `<garbage-collection.rst>`_ for full details.
+ garbage-collection.rst_ for full details.
+
+.. _#390: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/390
+.. _garbage-collection.rst: garbage-collection.rst
Running A Helper
``enabled = (boolean, optional)``
- If ``True``, the node will run a helper (see `<helper.rst>`_ for
- details). The helper's contact FURL will be placed in
- ``private/helper.furl``, from which it can be copied to any clients that
- wish to use it. Clearly nodes should not both run a helper and attempt to
- use one: do not create ``helper.furl`` and also define
- ``[helper]enabled`` in the same node. The default is ``False``.
+ If ``True``, the node will run a helper (see helper.rst_ for details).
+ The helper's contact FURL will be placed in ``private/helper.furl``, from
+ which it can be copied to any clients that wish to use it. Clearly nodes
+ should not both run a helper and attempt to use one: do not create
+ ``helper.furl`` and also define ``[helper]enabled`` in the same node.
+ The default is ``False``.
Running An Introducer
``private/helper.furl``
If the node is running a helper (for use by other clients), its contact
- FURL will be placed here. See `<helper.rst>`_ for more details.
+ FURL will be placed here. See helper.rst_ for more details.
``private/root_dir.cap`` (optional)
files. The web-API has a facility to block access to filecaps by their
storage index, returning a 403 "Forbidden" error instead of the original
file. For more details, see the "Access Blacklist" section of
- `<frontends/webapi.rst>`_.
+ webapi.rst_.
Example
Tahoe-LAFS releases before v1.3.0 had no ``tahoe.cfg`` file, and used
distinct files for each item. This is no longer supported and if you have
configuration in the old format you must manually convert it to the new
-format for Tahoe-LAFS to detect it. See `<historical/configuration.rst>`_.
+format for Tahoe-LAFS to detect it. See `historical/configuration.rst`_.
+
+.. _historical/configuration.rst: historical/configuration.rst
-
+.. -*- coding: utf-8-with-signature -*-
What Is It?
-----------
+.. -*- coding: utf-8-with-signature -*-
+
=========================
Debian and Ubuntu Support
=========================
+.. -*- coding: utf-8-with-signature -*-
+
=========================
Filesystem-specific notes
=========================
If "dir_index" is present in the "features:" line, then you're all set. If
not, you'll need to use tune2fs and e2fsck to enable and build the index. See
-`<http://wiki.dovecot.org/MailboxFormat/Maildir>`_ for some hints.
+`http://wiki2.dovecot.org/MailboxFormat/Maildir`_ for some hints.
+
+.. _`http://wiki2.dovecot.org/MailboxFormat/Maildir`: http://wiki2.dovecot.org/MailboxFormat/Maildir
+.. -*- coding: utf-8-with-signature -*-
+
===========================
The Tahoe-LAFS CLI commands
===========================
+.. -*- coding: utf-8-with-signature -*-
+
=================================
Tahoe-LAFS SFTP and FTP Frontends
=================================
+.. -*- coding: utf-8-with-signature -*-
+
===============
Download status
===============
+.. -*- coding: utf-8-with-signature -*-
+
===============================
Tahoe-LAFS Drop-Upload Frontend
===============================
with an immutable file. (`#1712`_)
If a file in the upload directory is changed (actually relinked to a new
-file), then the old file is still present on the grid, and any other caps
-to it will remain valid. See `docs/garbage-collection.rst
-<../garbage-collection.rst>`_ for how to reclaim the space used by files
-that are no longer needed.
+file), then the old file is still present on the grid, and any other caps to
+it will remain valid. See `docs/garbage-collection.rst`_ for how to reclaim
+the space used by files that are no longer needed.
Unicode names are supported, but the local name of a file must be encoded
correctly in order for it to be uploaded. The expected encoding is that
.. _`#1710`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1710
.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711
.. _`#1712`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1712
+
+.. _docs/garbage-collection.rst: ../garbage-collection.rst
+
+.. -*- coding: utf-8-with-signature -*-
+
==========================
The Tahoe REST-ful Web API
==========================
This string is actually a Twisted "strports" specification, meaning you can
get more control over the interface to which the server binds by supplying
additional arguments. For more details, see the documentation on
-`twisted.application.strports
-<https://twistedmatrix.com/documents/current/api/twisted.application.strports.html>`_.
+`twisted.application.strports`_.
Writing "tcp:3456:interface=127.0.0.1" into the web.port line does the same
but binds to the loopback interface, ensuring that only the programs on the
option to the 'tahoe create-node' command. By default, the node listens on
port 3456, on the loopback (127.0.0.1) interface.
+.. _twisted.application.strports: https://twistedmatrix.com/documents/current/api/twisted.application.strports.html
+
Basic Concepts: GET, PUT, DELETE, POST
======================================
-As described in `docs/architecture.rst <../architecture.rst>`_, each file
-and directory in a Tahoe virtual filesystem is referenced by an identifier
-that combines the designation of the object with the authority to do something
-with it (such as read or modify the contents). This identifier is called a
-"read-cap" or "write-cap", depending upon whether it enables read-only or
-read-write access. These "caps" are also referred to as URIs (which may be
-confusing because they are not currently `RFC3986
-<https://tools.ietf.org/html/rfc3986>`_-compliant URIs).
+As described in `docs/architecture.rst`_, each file and directory in a Tahoe
+virtual filesystem is referenced by an identifier that combines the
+designation of the object with the authority to do something with it (such as
+read or modify the contents). This identifier is called a "read-cap" or
+"write-cap", depending upon whether it enables read-only or read-write
+access. These "caps" are also referred to as URIs (which may be confusing
+because they are not currently RFC3986_-compliant URIs).
The Tahoe web-based API is "REST-ful", meaning it implements the concepts of
"REpresentational State Transfer": the original scheme by which the World
``text/*``, or text/html (or if there is no Accept header), HTML tracebacks will
be generated.
+.. _RFC3986: https://tools.ietf.org/html/rfc3986
+.. _docs/architecture.rst: ../architecture.rst
+
URLs
====
The metadata may have a "no-write" field. If this is set to true in the
metadata of a link, it will not be possible to open that link for writing
- via the SFTP frontend; see `<FTP-and-SFTP.rst>`_ for details.
- Also, if the "no-write" field is set to true in the metadata of a link to
- a mutable child, it will cause the link to be diminished to read-only.
+ via the SFTP frontend; see FTP-and-SFTP.rst_ for details. Also, if the
+ "no-write" field is set to true in the metadata of a link to a mutable
+ child, it will cause the link to be diminished to read-only.
Note that the web-API-using client application must not provide the
"Content-Type: multipart/form-data" header that usually accompanies HTML
This operation will return an error if the parent directory is immutable,
or already has a child named NAME.
+.. _FTP-and-SFTP.rst: FTP-and-SFTP.rst
+
Getting Information About a File Or Directory (as JSON)
-------------------------------------------------------
web requests themselves).
For more details, please see the "Consistency vs Availability" and "The Prime
-Coordination Directive" sections of `mutable.rst <../specifications/mutable.rst>`_.
+Coordination Directive" sections of mutable.rst_.
+
+.. _mutable.rst: ../specifications/mutable.rst
Access Blacklist
.. [1] URLs and HTTP and UTF-8, Oh My
HTTP does not provide a mechanism to specify the character set used to
- encode non-ASCII names in URLs
- (`RFC3986#2.1 <https://tools.ietf.org/html/rfc3986#section-2.1>`_).
- We prefer the convention that the ``filename=`` argument shall be a
- URL-escaped UTF-8 encoded Unicode string.
- For example, suppose we want to provoke the server into using a filename of
- "f i a n c e-acute e" (i.e. f i a n c U+00E9 e). The UTF-8 encoding of this
- is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\\xC3\\xA9e", as python's
- ``repr()`` function would show). To encode this into a URL, the non-printable
- characters must be escaped with the urlencode ``%XX`` mechanism, giving
- us "fianc%C3%A9e". Thus, the first line of the HTTP request will be
- "``GET /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1``". Not all
- browsers provide this: IE7 by default uses the Latin-1 encoding, which is
- "fianc%E9e" (although it has a configuration option to send URLs as UTF-8).
+ encode non-ASCII names in URLs (`RFC3986#2.1`_). We prefer the convention
+ that the ``filename=`` argument shall be a URL-escaped UTF-8 encoded Unicode
+ string. For example, suppose we want to provoke the server into using a
+ filename of "f i a n c e-acute e" (i.e. f i a n c U+00E9 e). The UTF-8
+ encoding of this is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or
+ "fianc\\xC3\\xA9e", as python's ``repr()`` function would show). To encode
+ this into a URL, the non-printable characters must be escaped with the
+ urlencode ``%XX`` mechanism, giving us "fianc%C3%A9e". Thus, the first line
+ of the HTTP request will be "``GET
+ /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1``". Not all browsers
+ provide this: IE7 by default uses the Latin-1 encoding, which is "fianc%E9e"
+ (although it has a configuration option to send URLs as UTF-8).
The response header will need to indicate a non-ASCII filename. The actual
mechanism to do this is not clear. For ASCII filenames, the response header
(note, the last four bytes of that line, not including the newline, are
0xC3 0xA9 0x65 0x22)
- `RFC2231#4 <https://tools.ietf.org/html/rfc2231#section-4>`_
- (dated 1997): suggests that the following might work, and
- `some developers have reported <http://markmail.org/message/dsjyokgl7hv64ig3>`_
- that it is supported by Firefox (but not IE7)::
+ `RFC2231#4`_ (dated 1997): suggests that the following might work, and `some
+ developers have reported`_ that it is supported by Firefox (but not IE7)::
#2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
- My reading of `RFC2616#19.5.1 <https://tools.ietf.org/html/rfc2616#section-19.5.1>`_
- (which defines Content-Disposition) says that the filename= parameter is
- defined to be wrapped in quotes (presumably to allow spaces without breaking
- the parsing of subsequent parameters), which would give us::
+ My reading of `RFC2616#19.5.1`_ (which defines Content-Disposition) says
+ that the filename= parameter is defined to be wrapped in quotes (presumably
+ to allow spaces without breaking the parsing of subsequent parameters),
+ which would give us::
#3: Content-Disposition: attachment; filename*=utf-8''"fianc%C3%A9e"
into the response header, rather than enforcing the UTF-8 convention. This
means it does not try to decode the filename from the URL argument, nor does
it encode the filename into the response header.
+
+.. _RFC3986#2.1: https://tools.ietf.org/html/rfc3986#section-2.1
+.. _RFC2231#4: https://tools.ietf.org/html/rfc2231#section-4
+.. _some developers have reported: http://markmail.org/message/dsjyokgl7hv64ig3
+.. _RFC2616#19.5.1: https://tools.ietf.org/html/rfc2616#section-19.5.1
+
+.. -*- coding: utf-8-with-signature -*-
+
===========================
Garbage Collection in Tahoe
===========================
There are several tradeoffs to be considered when choosing the renewal timer
and the lease duration, and there is no single optimal pair of values. See
-the `<lease-tradeoffs.svg>`_ diagram to get an idea for the tradeoffs involved.
+the lease-tradeoffs.svg_ diagram to get an idea for the tradeoffs involved.
If lease renewal occurs quickly and with 100% reliability, than any renewal
time that is shorter than the lease duration will suffice, but a larger ratio
of duration-over-renewal-time will be more robust in the face of occasional
Renewing leases can be expected to take about one second per file/directory,
depending upon the number of servers and the network speeds involved.
+.. _lease-tradeoffs.svg: lease-tradeoffs.svg
+
+
Client-side Renewal
===================
older objects have up-to-date leases on them.
A separate "rebalancing manager/service" is also planned -- see ticket
-`#543 <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/543>`_. The exact
-details of what this service will do are not settled, but it is likely to
-work by acquiring manifests from rootcaps on a periodic basis, keeping track
-of checker results, managing lease-addition, and prioritizing repair and
-rebalancing of shares. Eventually it may use multiple worker nodes to perform
-these jobs in parallel.
+`#543`_. The exact details of what this service will do are not settled, but
+it is likely to work by acquiring manifests from rootcaps on a periodic
+basis, keeping track of checker results, managing lease-addition, and
+prioritizing repair and rebalancing of shares. Eventually it may use multiple
+worker nodes to perform these jobs in parallel.
+
+.. _#543: http://tahoe-lafs.org/trac/tahoe-lafs/ticket/543
+
Server Side Expiration
======================
+.. -*- coding: utf-8-with-signature -*-
+
=======================
The Tahoe Upload Helper
=======================
+.. -*- coding: utf-8-with-signature -*-
+
=======================
Old Configuration Files
=======================
-
+.. -*- coding: utf-8-with-signature -*-
+
See also `cautions.rst`_.
.. _cautions.rst: cautions.rst
+.. -*- coding: utf-8-with-signature -*-
+
=============
Tahoe Logging
=============
+.. -*- coding: utf-8-with-signature -*-
+
=======================
Node Keys in Tahoe-LAFS
=======================
+.. -*- coding: utf-8-with-signature -*-
+
============================================
Performance costs for some common operations
============================================
-
+.. -*- coding: utf-8-with-signature -*-
+
=====================
Lease database design
=====================
-
+.. -*- coding: utf-8-with-signature -*-
+
==================
Getting Tahoe-LAFS
==================
+.. -*- coding: utf-8-with-signature -*-
+
=====================
How To Run Tahoe-LAFS
=====================
This is how to run a Tahoe-LAFS client or a complete Tahoe-LAFS grid.
First you have to install the Tahoe-LAFS software, as documented in
-`quickstart.rst <quickstart.rst>`_.
+quickstart.rst_.
The ``tahoe`` program in the ``bin`` directory is used to create,
start, and stop nodes. Each node lives in a separate base directory, in
the Tahoe-LAFS code. There is also an *introducer node* that is
responsible for getting the other nodes talking to each other.
-If you're getting started we recommend you try connecting to
-the `public test grid
-<https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TestGrid>`_ as you only
-need to create a client node. When you want to create your own grid
-you'll need to create the introducer and several initial storage nodes
-(see the note about small grids below).
+If you're getting started we recommend you try connecting to the `public test
+grid`_ as you only need to create a client node. When you want to create your
+own grid you'll need to create the introducer and several initial storage
+nodes (see the note about small grids below).
If the Tahoe-LAFS ``bin`` directory is not on your PATH, then in all
the command lines below, specify the full path to ``bin/tahoe``.
-To construct a client node, run "``tahoe create-client``", which will
-create ``~/.tahoe`` to be the node's base directory. Acquire the
-``introducer.furl`` (see below if you are running your own introducer,
-or use the one from the `TestGrid page
-<https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TestGrid>`_), and paste it
-after ``introducer.furl =`` in the ``[client]`` section of
-``~/.tahoe/tahoe.cfg``. Then use "``tahoe run ~/.tahoe``". After that,
-the node should be off and running. The first thing it will do is
-connect to the introducer and get itself connected to all other nodes
-on the grid.
+To construct a client node, run "``tahoe create-client``", which will create
+``~/.tahoe`` to be the node's base directory. Acquire the ``introducer.furl``
+(see below if you are running your own introducer, or use the one from the
+`TestGrid page`_), and paste it after ``introducer.furl =`` in the
+``[client]`` section of ``~/.tahoe/tahoe.cfg``. Then use "``tahoe run
+~/.tahoe``". After that, the node should be off and running. The first thing
+it will do is connect to the introducer and get itself connected to all other
+nodes on the grid.
By default, "``tahoe create-client``" creates a client-only node, that
does not offer its disk space to other nodes. To configure other behavior,
-use "``tahoe create-node``" or see `configuration.rst <configuration.rst>`_.
+use "``tahoe create-node``" or see configuration.rst_.
To construct an introducer, create a new base directory for it (the
name of the directory is up to you), ``cd`` into it, and run
``introducer.furl`` into the ``private/`` subdirectory of that base
directory. This file contains the URL the other nodes must use in order
to connect to this introducer. (Note that "``tahoe run .``" doesn't
-work for introducers, this is a known issue: `#937
-<http://allmydata.org/trac/tahoe-lafs/ticket/937>`_.)
+work for introducers, this is a known issue: `#937`_.)
The "``tahoe run``" command above will run the node in the foreground.
On Unix, you can run it in the background instead by using the
"``tahoe start``" command. To stop a node started in this way, use
"``tahoe stop``". ``tahoe --help`` gives a summary of all commands.
-See `configuration.rst <configuration.rst>`_ for more details about how
-to configure Tahoe-LAFS, including how to get other clients to connect
-to your node if it is behind a firewall or NAT device.
+See configuration.rst_ for more details about how to configure Tahoe-LAFS,
+including how to get other clients to connect to your node if it is behind a
+firewall or NAT device.
+
+.. _quickstart.rst: quickstart.rst
+.. _public test grid: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TestGrid
+.. _TestGrid page: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TestGrid
+.. _configuration.rst: configuration.rst
+.. _#937: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/937
+
A note about small grids
------------------------
By default, Tahoe-LAFS ships with the configuration parameter
-``shares.happy`` set to 7. If you are using Tahoe-LAFS on a
-grid with fewer than 7 storage nodes, this won't work well for
-you -- none of your uploads will succeed. To fix this, see
-`configuration.rst <configuration.rst>`_ to learn how to set
+``shares.happy`` set to 7. If you are using Tahoe-LAFS on a grid with fewer
+than 7 storage nodes, this won't work well for you -- none of your uploads
+will succeed. To fix this, see configuration.rst_ to learn how to set
``shares.happy`` to a more suitable value for your grid.
+
Do Stuff With It
================
The WUI
-------
-Point your web browser to `http://127.0.0.1:3456
-<http://127.0.0.1:3456>`_ -- which is the URL of the gateway running on
-your own local computer -- to use your newly created node.
+Point your web browser to `http://127.0.0.1:3456`_ -- which is the URL of the
+gateway running on your own local computer -- to use your newly created node.
Create a new directory (with the button labelled "create a directory").
Your web browser will load the new directory. Now if you want to be
You can do more or less everything you want to do with a decentralized
filesystem through the WUI.
+.. _http://127.0.0.1:3456: http://127.0.0.1:3456
+
+
The CLI
-------
-Prefer the command-line? Run "``tahoe --help``" (the same command-line
-tool that is used to start and stop nodes serves to navigate and use
-the decentralized filesystem). To get started, create a new directory
-and mark it as the 'tahoe:' alias by running
-"``tahoe create-alias tahoe``". Once you've done that, you can do
-"``tahoe ls tahoe:``" and "``tahoe cp LOCALFILE tahoe:foo.txt``" to
-work with your filesystem. The Tahoe-LAFS CLI uses similar syntax to
-the well-known scp and rsync tools. See `CLI.rst <frontends/CLI.rst>`_
-for more details.
+Prefer the command-line? Run "``tahoe --help``" (the same command-line tool
+that is used to start and stop nodes serves to navigate and use the
+decentralized filesystem). To get started, create a new directory and mark it
+as the 'tahoe:' alias by running "``tahoe create-alias tahoe``". Once you've
+done that, you can do "``tahoe ls tahoe:``" and "``tahoe cp LOCALFILE
+tahoe:foo.txt``" to work with your filesystem. The Tahoe-LAFS CLI uses
+similar syntax to the well-known scp and rsync tools. See CLI.rst_ for more
+details.
As with the WUI (and with all current interfaces to Tahoe-LAFS), you
are responsible for remembering directory capabilities yourself. If you
create a new directory and lose the capability to it, then you cannot
access that directory ever again.
+.. _CLI.rst: frontends/CLI.rst
+
+
The SFTP and FTP frontends
--------------------------
-You can access your Tahoe-LAFS grid via any `SFTP
-<http://en.wikipedia.org/wiki/SSH_file_transfer_protocol>`_ or `FTP
-<http://en.wikipedia.org/wiki/File_Transfer_Protocol>`_ client.
-See `FTP-and-SFTP.rst <frontends/FTP-and-SFTP.rst>`_ for how to set
+You can access your Tahoe-LAFS grid via any SFTP_ or FTP_ client.
+See `FTP-and-SFTP.rst`_ for how to set
this up. On most Unix platforms, you can also use SFTP to plug
Tahoe-LAFS into your computer's local filesystem via ``sshfs``.
-The `SftpFrontend
-<https://tahoe-lafs.org/trac/tahoe-lafs/wiki/SftpFrontend>`_ page on the
-wiki has more information about using SFTP with Tahoe-LAFS.
+The SftpFrontend_ page on the wiki has more information about using SFTP with
+Tahoe-LAFS.
+
+.. _SFTP: https://en.wikipedia.org/wiki/SSH_file_transfer_protocol
+.. _FTP: http://en.wikipedia.org/wiki/File_Transfer_Protocol
+.. _FTP-and-SFTP.rst: frontends/FTP-and-SFTP.rst
+.. _SftpFrontend: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/SftpFrontend
+
The WAPI
--------
Want to program your Tahoe-LAFS node to do your bidding? Easy! See
-`webapi.rst <frontends/webapi.rst>`_.
+webapi.rst_.
+
+.. _webapi.rst: frontends/webapi.rst
+
Socialize
=========
You can chat with other users of and hackers of this software on the
-#tahoe-lafs IRC channel at ``irc.freenode.net``, or on the `tahoe-dev
-mailing list
-<https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev>`_.
+#tahoe-lafs IRC channel at ``irc.freenode.net``, or on the `tahoe-dev mailing
+list`_.
+
+.. _tahoe-dev mailing list: https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
+.. -*- coding: utf-8-with-signature -*-
+
===================
URI Extension Block
===================
-
+.. -*- coding: utf-8-with-signature -*-
+
=============================================================
Redundant Array of Independent Clouds: Share To Cloud Mapping
=============================================================
+.. -*- coding: utf-8-with-signature -*-
+
==========================
Tahoe-LAFS Directory Nodes
==========================
Earlier releases used a "vdrive server": this server was abolished in the
v0.7.0 release.
-For details of how mutable files work, please see `<mutable.rst>`_ in this
+For details of how mutable files work, please see mutable.rst_ in this
directory.
For releases since v0.7.0, we achieve most of our desired properties. The
version 1 and other shares of version 2). In extreme cases of simultaneous
update, mutable files might suffer from non-monotonicity.
+.. _mutable.rst: mutable.rst
+
Dirnode secret values
=====================
As mentioned before, dirnodes are simply a special way to interpret the
contents of a mutable file, so the secret keys and capability strings
-described in `<mutable.rst>`_ are all the same. Each dirnode contains an RSA
+described in mutable.rst_ are all the same. Each dirnode contains an RSA
public/private keypair, and the holder of the "write capability" will be able
to retrieve the private key (as well as the AES encryption key used for the
data itself). The holder of the "read capability" will be able to obtain the
+.. -*- coding: utf-8-with-signature -*-
+
=============
File Encoding
=============
.. image:: file-encoding6.svg
+
Hashes
======
attacks. Using the tag protects our Merkle trees against attacks in which the
hash of a leaf is confused with a hash of two children (allowing an attacker
to generate corrupted data that nevertheless appears to be valid), and is
-simply good "cryptograhic hygiene". The `"Chosen Protocol Attack" by Kelsey,
-Schneier, and Wagner <http://www.schneier.com/paper-chosen-protocol.html>`_ is
-relevant. Putting the tag in a netstring guards against attacks that seek to
-confuse the end of the tag with the beginning of the subsequent value.
+simply good "cryptograhic hygiene". The `“Chosen Protocol Attack” by Kelsey,
+Schneier, and Wagner`_ is relevant. Putting the tag in a netstring guards
+against attacks that seek to confuse the end of the tag with the beginning of
+the subsequent value.
+.. _“Chosen Protocol Attack” by Kelsey, Schneier, and Wagner: http://www.schneier.com/paper-chosen-protocol.html
+.. -*- coding: utf-8-with-signature -*-
+
=============
Mutable Files
=============
+.. -*- coding: utf-8-with-signature -*-
+
==============================
Specification Document Outline
==============================
+.. -*- coding: utf-8-with-signature -*-
+
====================
Servers of Happiness
====================
+.. -*- coding: utf-8-with-signature -*-
+
==========
Tahoe URIs
==========
encodings are expressed in lower-case, with the trailing '=' signs removed.
For example, the following is a CHK URI, generated from a previous version of
-the contents of `<../architecture.rst>`_::
+the contents of architecture.rst_::
URI:CHK:ihrbeov7lbvoduupd4qblysj7a:bg5agsdt62jb34hxvxmdsbza6do64f4fg5anxxod2buttbo6udzq:3:10:28733
URI:CHK: prefix really indicates that an immutable file is in use, without
saying anything about how the key was derived.
+.. _architecture.rst: ../architecture.rst
+
+
LIT URIs
--------
Where (writekey) is the base32 encoding of the 16-byte AES encryption key
that is used to encrypt the RSA private key, and (fingerprint) is the base32
encoded 32-byte SHA-256 hash of the RSA public key. For more details about
-the way these keys are used, please see `<mutable.rst>`_.
+the way these keys are used, please see mutable.rst_.
The format for mutable read-caps is::
"Sub-Space Keys" from the Freenet project, which uses a vaguely similar
structure to provide mutable file access.
+.. _mutable.rst: mutable.rst
+
+
Directory URIs
==============
The grid layer provides a mapping from URI to data. To turn this into a graph
of directories and files, the "vdrive" layer (which sits on top of the grid
layer) needs to keep track of "directory nodes", or "dirnodes" for short.
-`<dirnodes.rst>`_ describes how these work.
+dirnodes.rst_ describes how these work.
Dirnodes are contained inside mutable files, and are thus simply a particular
way to interpret the contents of these files. As a result, a directory
Historical note: the "DIR2" prefix is used because the non-distributed
dirnodes in earlier Tahoe releases had already claimed the "DIR" prefix.
+.. _dirnodes.rst: dirnodes.rst
+
+
Internal Usage of URIs
======================
-================
+.. -*- coding: utf-8-with-signature -*-
+
+================
Tahoe Statistics
================
the desired portnumber into a file named "portnum" (i.e. $BASEDIR/portnum),
and the next time the gatherer is started, it will start listening on the
given port. The portnum file is actually a "strports specification string",
-as described in `docs/configuration.rst <configuration.rst>`_.
+as described in configuration.rst_.
Once running, the stats gatherer will create a standard python "pickle" file
in $BASEDIR/stats.pickle . Once a minute, the gatherer will pull stats
total-disk-available number for the entire grid (however, the "disk watcher"
daemon, in misc/operations_helpers/spacetime/, is better suited for this specific task).
+.. _configuration.rst: configuration.rst
+
Using Munin To Graph Stats Values
=================================
graphs of various things over multiple time scales (last hour, last month,
last year).
-.. _Munin: http://munin-monitoring.org/
-
Most of the plugins are designed to pull stats from a single Tahoe node, and
are configured with the e.g. http://localhost:3456/statistics?t=json URL. The
"tahoe_stats" plugin is designed to read from the pickle file created by the
Please see the docstrings at the beginning of each plugin for details, and
the "tahoe-conf" file for notes about configuration and installing these
plugins into a Munin environment.
+
+.. _Munin: http://munin-monitoring.org/
+.. -*- coding: utf-8-with-signature -*-
+
==================================
Avoiding Write Collisions in Tahoe
==================================
projects / tahoe-lafs / tahoe-lafs.git / commitdiff