1. `Reading a file`_
2. `Writing/Uploading a File`_
3. `Creating a New Directory`_
- 4. `Get Information About A File Or Directory (as JSON)`_
- 5. `Attaching an existing File or Directory by its read- or write-cap`_
- 6. `Adding multiple files or directories to a parent directory at once`_
- 7. `Deleting a File or Directory`_
+ 4. `Getting Information About A File Or Directory (as JSON)`_
+ 5. `Attaching an Existing File or Directory by its read- or write-cap`_
+ 6. `Adding Multiple Files or Directories to a Parent Directory at Once`_
+ 7. `Unlinking a File or Directory`_
6. `Browser Operations: Human-Oriented Interfaces`_
1. `Viewing A Directory (as HTML)`_
2. `Viewing/Downloading a File`_
- 3. `Get Information About A File Or Directory (as HTML)`_
+ 3. `Getting Information About A File Or Directory (as HTML)`_
4. `Creating a Directory`_
5. `Uploading a File`_
6. `Attaching An Existing File Or Directory (by URI)`_
- 7. `Deleting A Child`_
+ 7. `Unlinking A Child`_
8. `Renaming A Child`_
9. `Other Utilities`_
10. `Debugging and Testing Features`_
7. `Other Useful Pages`_
8. `Static Files in /public_html`_
-9. `Safety and security issues -- names vs. URIs`_
+9. `Safety and Security Issues -- Names vs. URIs`_
10. `Concurrency Issues`_
+11. `Access Blacklist`_
+
Enabling the web-API port
=========================
option to the 'tahoe create-node' command. By default, the node listens on
port 3456, on the loopback (127.0.0.1) interface.
+
Basic Concepts: GET, PUT, DELETE, POST
======================================
operations are required to have no side-effects.
PUT is used to upload new objects into the filesystem, or to replace an
-existing object. DELETE it used to delete objects from the filesystem. Both
-PUT and DELETE are required to be idempotent: performing the same operation
-multiple times must have the same side-effects as only performing it once.
+existing link or the contents of a mutable file. DELETE is used to unlink
+objects from directories. Both PUT and DELETE are required to be idempotent:
+performing the same operation multiple times must have the same side-effects
+as only performing it once.
POST is used for more complicated actions that cannot be expressed as a GET,
PUT, or DELETE. POST operations can be thought of as a method call: sending
some message to the object referenced by the URL. In Tahoe, POST is also used
for operations that must be triggered by an HTML form (including upload and
-delete), because otherwise a regular web browser has no way to accomplish
+unlinking), because otherwise a regular web browser has no way to accomplish
these tasks. In general, everything that can be done with a PUT or DELETE can
also be done with a POST.
expected to use the RESTful interface described above. The second is a human
using a standard web browser to work with the filesystem. This user is given
a series of HTML pages with links to download files, and forms that use POST
-actions to upload, rename, and delete files.
+actions to upload, rename, and unlink files.
When an error occurs, the HTTP response code will be set to an appropriate
400-series code (like 404 Not Found for an unknown childname, or 400 Bad Request
-when the parameters to a webapi operation are invalid), and the HTTP response
+when the parameters to a web-API operation are invalid), and the HTTP response
body will usually contain a few lines of explanation as to the cause of the
error and possible responses. Unusual exceptions may result in a 500 Internal
Server Error as a catch-all, with a default response body containing
``text/*``, or text/html (or if there is no Accept header), HTML tracebacks will
be generated.
+
URLs
====
Also note that the filenames inside upload POST forms are interpreted using
whatever character set was provided in the conventional '_charset' field, and
defaults to UTF-8 if not otherwise specified. The JSON representation of each
-directory contains native unicode strings. Tahoe directories are specified to
-contain unicode filenames, and cannot contain binary strings that are not
+directory contains native Unicode strings. Tahoe directories are specified to
+contain Unicode filenames, and cannot contain binary strings that are not
representable as such.
All Tahoe operations that refer to existing files or directories must include
-a suitable read- or write- cap in the URL: the webapi server won't add one
+a suitable read- or write- cap in the URL: the web-API server won't add one
for you. If you don't know the cap, you can't access the file. This allows
-the security properties of Tahoe caps to be extended across the webapi
+the security properties of Tahoe caps to be extended across the web-API
interface.
+
Slow Operations, Progress, and Cancelling
=========================================
handles. Instead, they emit line-oriented status results immediately. Client
code can cancel the operation by simply closing the HTTP connection.
+
Programmatic Operations
=======================
that use HTTP to communicate with a Tahoe node. A later section describes
operations that are intended for web browsers.
+
Reading A File
--------------
"Browser Operations", for details on how to modify these URLs for that
purpose.
+
Writing/Uploading A File
------------------------
To use the /uri/$FILECAP form, $FILECAP must be a write-cap for a mutable file.
In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a
- writeable mutable file, that file's contents will be overwritten in-place. If
- it is a read-cap for a mutable file, an error will occur. If it is an
- immutable file, the old file will be discarded, and a new one will be put in
- its place.
+ writeable mutable file, that file's contents will be overwritten
+ in-place. If it is a read-cap for a mutable file, an error will occur.
+ If it is an immutable file, the old file will be discarded, and a new
+ one will be put in its place. If the target file is a writable mutable
+ file, you may also specify an "offset" parameter -- a byte offset that
+ determines where in the mutable file the data from the HTTP request
+ body is placed. This operation is relatively efficient for MDMF mutable
+ files, and is relatively inefficient (but still supported) for SDMF
+ mutable files. If no offset parameter is specified, then the entire
+ file is replaced with the data from the HTTP request body. For an
+ immutable file, the "offset" parameter is not valid.
When creating a new file, if "mutable=true" is in the query arguments, the
operation will create a mutable file instead of an immutable one.
If "mutable=true" is in the query arguments, the operation will create a
mutable file, and return its write-cap in the HTTP respose. The default is
- to create an immutable file, returning the read-cap as a response.
+ to create an immutable file, returning the read-cap as a response. If
+ you create a mutable file, you can also use the "mutable-type" query
+ parameter. If "mutable-type=sdmf", then the mutable file will be created
+ in the old SDMF mutable file format. This is desirable for files that
+ need to be read by old clients. If "mutable-type=mdmf", then the file
+ will be created in the new MDMF mutable file format. MDMF mutable files
+ can be downloaded more efficiently, and modified in-place efficiently,
+ but are not compatible with older versions of Tahoe-LAFS. If no
+ "mutable-type" argument is given, the file is created in whatever
+ format was configured in tahoe.cfg.
+
Creating A New Directory
------------------------
}
For forward-compatibility, a mutable directory can also contain caps in
- a format that is unknown to the webapi server. When such caps are retrieved
+ a format that is unknown to the web-API server. When such caps are retrieved
from a mutable directory in a "ro_uri" field, they will be prefixed with
the string "ro.", indicating that they must not be decoded without
checking that they are read-only. The "ro." prefix must not be stripped
- off without performing this check. (Future versions of the webapi server
+ off without performing this check. (Future versions of the web-API server
will perform it where necessary.)
If both the "rw_uri" and "ro_uri" fields are present in a given PROPDICT,
- and the webapi server recognizes the rw_uri as a write cap, then it will
+ and the web-API server recognizes the rw_uri as a write cap, then it will
reset the ro_uri to the corresponding read cap and discard the original
contents of ro_uri (in order to ensure that the two caps correspond to the
same object and that the ro_uri is in fact read-only). However this may not
- happen for caps in a format unknown to the webapi server. Therefore, when
- writing a directory the webapi client should ensure that the contents
+ happen for caps in a format unknown to the web-API server. Therefore, when
+ writing a directory the web-API client should ensure that the contents
of "rw_uri" and "ro_uri" for a given PROPDICT are a consistent
- (write cap, read cap) pair if possible. If the webapi client only has
+ (write cap, read cap) pair if possible. If the web-API client only has
one cap and does not know whether it is a write cap or read cap, then
it is acceptable to set "rw_uri" to that cap and omit "ro_uri". The
client must not put a write cap into a "ro_uri" field.
Also, if the "no-write" field is set to true in the metadata of a link to
a mutable child, it will cause the link to be diminished to read-only.
- Note that the webapi-using client application must not provide the
+ Note that the web-API-using client application must not provide the
"Content-Type: multipart/form-data" header that usually accompanies HTML
form submissions, since the body is not formatted this way. Doing so will
cause a server error as the lower-level code misparses the request body.
- Child file names should each be expressed as a unicode string, then used as
+ Child file names should each be expressed as a Unicode string, then used as
keys of the dictionary. The dictionary should then be converted into JSON,
and the resulting string encoded into UTF-8. This UTF-8 bytestring should
then be used as the POST body.
immutable files, literal files, and deep-immutable directories.
For forward-compatibility, a deep-immutable directory can also contain caps
- in a format that is unknown to the webapi server. When such caps are retrieved
+ in a format that is unknown to the web-API server. When such caps are retrieved
from a deep-immutable directory in a "ro_uri" field, they will be prefixed
with the string "imm.", indicating that they must not be decoded without
checking that they are immutable. The "imm." prefix must not be stripped
- off without performing this check. (Future versions of the webapi server
+ off without performing this check. (Future versions of the web-API server
will perform it where necessary.)
The cap for each child may be given either in the "rw_uri" or "ro_uri"
field of the PROPDICT (not both). If a cap is given in the "rw_uri" field,
- then the webapi server will check that it is an immutable read-cap of a
+ then the web-API server will check that it is an immutable read-cap of a
*known* format, and give an error if it is not. If a cap is given in the
- "ro_uri" field, then the webapi server will still check whether known
+ "ro_uri" field, then the web-API server will still check whether known
caps are immutable, but for unknown caps it will simply assume that the
cap can be stored, as described above. Note that an attacker would be
able to store any cap in an immutable directory, so this check when
This operation will return an error if the parent directory is immutable,
or already has a child named NAME.
-Get Information About A File Or Directory (as JSON)
----------------------------------------------------
+
+Getting Information About A File Or Directory (as JSON)
+-------------------------------------------------------
``GET /uri/$FILECAP?t=json``
instead of the 'tahoe':'linkmotime' and 'tahoe':'linkcrtime' keys. Starting
in Tahoe v1.4.0, the 'linkmotime'/'linkcrtime' keys in the 'tahoe' sub-dict
are populated. However, prior to Tahoe v1.7beta, a bug caused the 'tahoe'
-sub-dict to be deleted by webapi requests in which new metadata is
+sub-dict to be deleted by web-API requests in which new metadata is
specified, and not to be added to existing child links that lack it.
From Tahoe v1.7.0 onward, the 'mtime' and 'ctime' fields are no longer
time.
-Attaching an existing File or Directory by its read- or write-cap
+Attaching an Existing File or Directory by its read- or write-cap
-----------------------------------------------------------------
``PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri``
Note that this operation does not take its child cap in the form of
separate "rw_uri" and "ro_uri" fields. Therefore, it cannot accept a
- child cap in a format unknown to the webapi server, unless its URI
+ child cap in a format unknown to the web-API server, unless its URI
starts with "ro." or "imm.". This restriction is necessary because the
server is not able to attenuate an unknown write cap to a read cap.
Unknown URIs starting with "ro." or "imm.", on the other hand, are
would result in granting the cap's write authority to holders of the
directory read cap.
-Adding multiple files or directories to a parent directory at once
+
+Adding Multiple Files or Directories to a Parent Directory at Once
------------------------------------------------------------------
``POST /uri/$DIRCAP/[SUBDIRS..]?t=set_children``
backward compatibility should continue to use "set_children".
-Deleting a File or Directory
-----------------------------
+Unlinking a File or Directory
+-----------------------------
``DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME``
be modified.
Note that this does not actually delete the file or directory that the name
- points to from the tahoe grid -- it only removes the named reference from
+ points to from the tahoe grid -- it only unlinks the named reference from
this directory. If there are other names in this directory or in other
directories that point to the resource, then it will remain accessible
through those paths. Even if all names pointing to this object are removed
This method returns the file- or directory- cap of the object that was just
removed.
+
Browser Operations: Human-oriented interfaces
=============================================
specified by using <input type="hidden"> elements. For clarity, the
descriptions below display the most significant arguments as URL query args.
+
Viewing A Directory (as HTML)
-----------------------------
browser, which contains HREF links to all files and directories reachable
from this directory. These HREF links do not have a t= argument, meaning
that a human who follows them will get pages also meant for a human. It also
- contains forms to upload new files, and to delete files and directories.
- Those forms use POST methods to do their job.
+ contains forms to upload new files, and to unlink files and directories
+ from their parent directory. Those forms use POST methods to do their job.
+
Viewing/Downloading a File
--------------------------
this form can *only* be used with file caps; it is an error to use a
directory cap after the /named/ prefix.
-Get Information About A File Or Directory (as HTML)
----------------------------------------------------
+
+Getting Information About A File Or Directory (as HTML)
+-------------------------------------------------------
``GET /uri/$FILECAP?t=info``
* deep-check/deep-size/deep-stats/manifest (for directories)
* replace-conents form (for mutable files)
+
Creating a Directory
--------------------
If a "mutable=true" argument is provided, the operation will create a
mutable file, and the response body will contain the write-cap instead of
the upload results page. The default is to create an immutable file,
- returning the upload results page as a response.
+ returning the upload results page as a response. If you create a
+ mutable file, you may choose to specify the format of that mutable file
+ with the "mutable-type" parameter. If "mutable-type=mdmf", then the
+ file will be created as an MDMF mutable file. If "mutable-type=sdmf",
+ then the file will be created as an SDMF mutable file. If no value is
+ specified, the file will be created in whatever format is specified in
+ tahoe.cfg.
``POST /uri/$DIRCAP/[SUBDIRS../]?t=upload``
the "PUT /uri/$FILECAP" form, but uses a POST for the benefit of HTML forms
in a web browser.
+
Attaching An Existing File Or Directory (by URI)
------------------------------------------------
directory, with a specified child name. This behaves much like the PUT t=uri
operation, and is a lot like a UNIX hardlink. It is subject to the same
restrictions as that operation on the use of cap formats unknown to the
- webapi server.
+ web-API server.
This will create additional intermediate directories as necessary, although
since it is expected to be triggered by a form that was retrieved by "GET
This accepts the same replace= argument as POST t=upload.
-Deleting A Child
-----------------
+
+Unlinking A Child
+-----------------
``POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME``
+``POST /uri/$DIRCAP/[SUBDIRS../]?t=unlink&name=CHILDNAME``
+
This instructs the node to remove a child object (file or subdirectory) from
the given directory, which must be mutable. Note that the entire subtree is
unlinked from the parent. Unlike deleting a subdirectory in a UNIX local
into the subtree will see that the child subdirectories are not modified by
this operation. Only the link from the given directory to its child is severed.
+ In Tahoe-LAFS v1.9.0 and later, t=unlink can be used as a synonym for t=delete.
+ If interoperability with older web-API servers is required, t=delete should
+ be used.
+
+
Renaming A Child
----------------
If the object is an immutable file, this will return the same value as
t=uri.
+
Debugging and Testing Features
------------------------------
was untraversable, since the manifest entry is emitted to the HTTP response
body before the child is traversed.
+
Other Useful Pages
==================
Static Files in /public_html
============================
-The webapi server will take any request for a URL that starts with /static
+The web-API server will take any request for a URL that starts with /static
and serve it from a configurable directory which defaults to
$BASEDIR/public_html . This is configured by setting the "[node]web.static"
value in $BASEDIR/tahoe.cfg . If this is left at the default value of
served with the contents of the file $BASEDIR/public_html/subdir/foo.html .
This can be useful to serve a javascript application which provides a
-prettier front-end to the rest of the Tahoe webapi.
+prettier front-end to the rest of the Tahoe web-API.
-Safety and security issues -- names vs. URIs
+Safety and Security Issues -- Names vs. URIs
============================================
Summary: use explicit file- and dir- caps whenever possible, to reduce the
The read and write caps in a given directory node are separate URIs, and
can't be assumed to point to the same object even if they were retrieved in
-the same operation (although the webapi server attempts to ensure this
+the same operation (although the web-API server attempts to ensure this
in most cases). If you need to rely on that property, you should explicitly
verify it. More generally, you should not make assumptions about the
internal consistency of the contents of mutable directories. As a result
directory) is found by following this name (or sequence of names) when my
request reaches the server". Use URIs if you want "this particular object".
+
Concurrency Issues
==================
Tahoe nodes implement internal serialization to make sure that a single Tahoe
node cannot conflict with itself. For example, it is safe to issue two
-directory modification requests to a single tahoe node's webapi server at the
+directory modification requests to a single tahoe node's web-API server at the
same time, because the Tahoe node will internally delay one of them until
after the other has finished being applied. (This feature was introduced in
Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing
Coordination Directive" sections of `mutable.rst <../specifications/mutable.rst>`_.
+Access Blacklist
+================
+
+Gateway nodes may find it necessary to prohibit access to certain files. The
+web-API has a facility to block access to filecaps by their storage index,
+returning a 403 "Forbidden" error instead of the original file.
+
+This blacklist is recorded in $NODEDIR/access.blacklist, and contains one
+blocked file per line. Comment lines (starting with ``#``) are ignored. Each
+line consists of the storage-index (in the usual base32 format as displayed
+by the "More Info" page, or by the "tahoe debug dump-cap" command), followed
+by whitespace, followed by a reason string, which will be included in the 403
+error message. This could hold a URL to a page that explains why the file is
+blocked, for example.
+
+So for example, if you found a need to block access to a file with filecap
+``URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861``,
+you could do the following::
+
+ tahoe debug dump-cap URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861
+ -> storage index: whpepioyrnff7orecjolvbudeu
+ echo "whpepioyrnff7orecjolvbudeu my puppy told me to" >>$NODEDIR/access.blacklist
+ tahoe restart $NODEDIR
+ tahoe get URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861
+ -> error, 403 Access Prohibited: my puppy told me to
+
+The ``access.blacklist`` file will be checked each time a file or directory
+is accessed: the file's ``mtime`` is used to decide whether it need to be
+reloaded. Therefore no node restart is necessary when creating the initial
+blacklist, nor when adding second, third, or additional entries to the list.
+When modifying the file, be careful to update it atomically, otherwise a
+request may arrive while the file is only halfway written, and the partial
+file may be incorrectly parsed.
+
+The blacklist is applied to all access paths (including FTP, SFTP, and CLI
+operations), not just the web-API. The blacklist also applies to directories.
+If a directory is blacklisted, the gateway will refuse access to both that
+directory and any child files/directories underneath it, when accessed via
+"DIRCAP/SUBDIR/FILENAME" -style URLs. Users who go directly to the child
+file/dir will bypass the blacklist.
+
+The node will log the SI of the file being blocked, and the reason code, into
+the ``logs/twistd.log`` file.
+
+
.. [1] URLs and HTTP and UTF-8, Oh My
HTTP does not provide a mechanism to specify the character set used to
- encode non-ascii names in URLs (rfc2396#2.1). We prefer the convention that
- the filename= argument shall be a URL-encoded UTF-8 encoded unicode object.
+ encode non-ASCII names in URLs
+ (`RFC3986#2.1 <http://tools.ietf.org/html/rfc3986#section-2.1>`_).
+ We prefer the convention that the ``filename=`` argument shall be a
+ URL-escaped UTF-8 encoded Unicode string.
For example, suppose we want to provoke the server into using a filename of
- "f i a n c e-acute e" (i.e. F I A N C U+00E9 E). The UTF-8 encoding of this
- is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\xC3\xA9e", as python's
- repr() function would show). To encode this into a URL, the non-printable
- characters must be escaped with the urlencode '%XX' mechansim, giving us
- "fianc%C3%A9e". Thus, the first line of the HTTP request will be "GET
- /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1". Not all browsers
- provide this: IE7 uses the Latin-1 encoding, which is fianc%E9e.
+ "f i a n c e-acute e" (i.e. f i a n c U+00E9 e). The UTF-8 encoding of this
+ is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\\xC3\\xA9e", as python's
+ ``repr()`` function would show). To encode this into a URL, the non-printable
+ characters must be escaped with the urlencode ``%XX`` mechanism, giving
+ us "fianc%C3%A9e". Thus, the first line of the HTTP request will be
+ "``GET /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1``". Not all
+ browsers provide this: IE7 by default uses the Latin-1 encoding, which is
+ "fianc%E9e" (although it has a configuration option to send URLs as UTF-8).
The response header will need to indicate a non-ASCII filename. The actual
mechanism to do this is not clear. For ASCII filenames, the response header
Content-Disposition: attachment; filename="english.txt"
- If Tahoe were to enforce the utf-8 convention, it would need to decode the
- URL argument into a unicode string, and then encode it back into a sequence
+ If Tahoe were to enforce the UTF-8 convention, it would need to decode the
+ URL argument into a Unicode string, and then encode it back into a sequence
of bytes when creating the response header. One possibility would be to use
- unencoded utf-8. Developers suggest that IE7 might accept this::
+ unencoded UTF-8. Developers suggest that IE7 might accept this::
#1: Content-Disposition: attachment; filename="fianc\xC3\xA9e"
(note, the last four bytes of that line, not including the newline, are
`RFC2231#4 <http://tools.ietf.org/html/rfc2231#section-4>`_
(dated 1997): suggests that the following might work, and
`some developers have reported <http://markmail.org/message/dsjyokgl7hv64ig3>`_
- that it is supported by firefox (but not IE7)::
+ that it is supported by Firefox (but not IE7)::
#2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
However this is contrary to the examples in the email thread listed above.
Developers report that IE7 (when it is configured for UTF-8 URL encoding,
- which is not the default in asian countries), will accept::
+ which is not the default in Asian countries), will accept::
#4: Content-Disposition: attachment; filename=fianc%C3%A9e
However, for maximum compatibility, Tahoe simply copies bytes from the URL
- into the response header, rather than enforcing the utf-8 convention. This
+ into the response header, rather than enforcing the UTF-8 convention. This
means it does not try to decode the filename from the URL argument, nor does
it encode the filename into the response header.
projects / tahoe-lafs / tahoe-lafs.git / blobdiff