1 ==========================
2 The Tahoe REST-ful Web API
3 ==========================
5 1. `Enabling the web-API port`_
6 2. `Basic Concepts: GET, PUT, DELETE, POST`_
11 4. `Slow Operations, Progress, and Cancelling`_
12 5. `Programmatic Operations`_
15 2. `Writing/Uploading a File`_
16 3. `Creating a New Directory`_
17 4. `Getting Information About A File Or Directory (as JSON)`_
18 5. `Attaching an Existing File or Directory by its read- or write-cap`_
19 6. `Adding Multiple Files or Directories to a Parent Directory at Once`_
20 7. `Unlinking a File or Directory`_
22 6. `Browser Operations: Human-Oriented Interfaces`_
24 1. `Viewing A Directory (as HTML)`_
25 2. `Viewing/Downloading a File`_
26 3. `Getting Information About A File Or Directory (as HTML)`_
27 4. `Creating a Directory`_
28 5. `Uploading a File`_
29 6. `Attaching An Existing File Or Directory (by URI)`_
30 7. `Unlinking A Child`_
31 8. `Renaming A Child`_
33 10. `Debugging and Testing Features`_
35 7. `Other Useful Pages`_
36 8. `Static Files in /public_html`_
37 9. `Safety and Security Issues -- Names vs. URIs`_
38 10. `Concurrency Issues`_
39 11. `Access Blacklist`_
42 Enabling the web-API port
43 =========================
45 Every Tahoe node is capable of running a built-in HTTP server. To enable
46 this, just write a port number into the "[node]web.port" line of your node's
47 tahoe.cfg file. For example, writing "web.port = 3456" into the "[node]"
48 section of $NODEDIR/tahoe.cfg will cause the node to run a webserver on port
51 This string is actually a Twisted "strports" specification, meaning you can
52 get more control over the interface to which the server binds by supplying
53 additional arguments. For more details, see the documentation on
54 `twisted.application.strports
55 <http://twistedmatrix.com/documents/current/api/twisted.application.strports.html>`_.
57 Writing "tcp:3456:interface=127.0.0.1" into the web.port line does the same
58 but binds to the loopback interface, ensuring that only the programs on the
59 local host can connect. Using "ssl:3456:privateKey=mykey.pem:certKey=cert.pem"
62 This webport can be set when the node is created by passing a --webport
63 option to the 'tahoe create-node' command. By default, the node listens on
64 port 3456, on the loopback (127.0.0.1) interface.
67 Basic Concepts: GET, PUT, DELETE, POST
68 ======================================
70 As described in `docs/architecture.rst <../architecture.rst>`_, each file
71 and directory in a Tahoe virtual filesystem is referenced by an identifier
72 that combines the designation of the object with the authority to do something
73 with it (such as read or modify the contents). This identifier is called a
74 "read-cap" or "write-cap", depending upon whether it enables read-only or
75 read-write access. These "caps" are also referred to as URIs (which may be
76 confusing because they are not currently `RFC3986
77 <http://tools.ietf.org/html/rfc3986>`_-compliant URIs).
79 The Tahoe web-based API is "REST-ful", meaning it implements the concepts of
80 "REpresentational State Transfer": the original scheme by which the World
81 Wide Web was intended to work. Each object (file or directory) is referenced
82 by a URL that includes the read- or write- cap. HTTP methods (GET, PUT, and
83 DELETE) are used to manipulate these objects. You can think of the URL as a
84 noun, and the method as a verb.
86 In REST, the GET method is used to retrieve information about an object, or
87 to retrieve some representation of the object itself. When the object is a
88 file, the basic GET method will simply return the contents of that file.
89 Other variations (generally implemented by adding query parameters to the
90 URL) will return information about the object, such as metadata. GET
91 operations are required to have no side-effects.
93 PUT is used to upload new objects into the filesystem, or to replace an
94 existing link or the contents of a mutable file. DELETE is used to unlink
95 objects from directories. Both PUT and DELETE are required to be idempotent:
96 performing the same operation multiple times must have the same side-effects
97 as only performing it once.
99 POST is used for more complicated actions that cannot be expressed as a GET,
100 PUT, or DELETE. POST operations can be thought of as a method call: sending
101 some message to the object referenced by the URL. In Tahoe, POST is also used
102 for operations that must be triggered by an HTML form (including upload and
103 unlinking), because otherwise a regular web browser has no way to accomplish
104 these tasks. In general, everything that can be done with a PUT or DELETE can
105 also be done with a POST.
107 Tahoe's web API is designed for two different kinds of consumer. The first is
108 a program that needs to manipulate the virtual file system. Such programs are
109 expected to use the RESTful interface described above. The second is a human
110 using a standard web browser to work with the filesystem. This user is given
111 a series of HTML pages with links to download files, and forms that use POST
112 actions to upload, rename, and unlink files.
114 When an error occurs, the HTTP response code will be set to an appropriate
115 400-series code (like 404 Not Found for an unknown childname, or 400 Bad Request
116 when the parameters to a web-API operation are invalid), and the HTTP response
117 body will usually contain a few lines of explanation as to the cause of the
118 error and possible responses. Unusual exceptions may result in a 500 Internal
119 Server Error as a catch-all, with a default response body containing
120 a Nevow-generated HTML-ized representation of the Python exception stack trace
121 that caused the problem. CLI programs which want to copy the response body to
122 stderr should provide an "Accept: text/plain" header to their requests to get
123 a plain text stack trace instead. If the Accept header contains ``*/*``, or
124 ``text/*``, or text/html (or if there is no Accept header), HTML tracebacks will
131 Tahoe uses a variety of read- and write- caps to identify files and
132 directories. The most common of these is the "immutable file read-cap", which
133 is used for most uploaded files. These read-caps look like the following::
135 URI:CHK:ime6pvkaxuetdfah2p2f35pe54:4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a:3:10:202
137 The next most common is a "directory write-cap", which provides both read and
138 write access to a directory, and look like this::
140 URI:DIR2:djrdkfawoqihigoett4g6auz6a:jx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq
142 There are also "directory read-caps", which start with "URI:DIR2-RO:", and
143 give read-only access to a directory. Finally there are also mutable file
144 read- and write- caps, which start with "URI:SSK", and give access to mutable
147 (Later versions of Tahoe will make these strings shorter, and will remove the
148 unfortunate colons, which must be escaped when these caps are embedded in
151 To refer to any Tahoe object through the web API, you simply need to combine
152 a prefix (which indicates the HTTP server to use) with the cap (which
153 indicates which object inside that server to access). Since the default Tahoe
154 webport is 3456, the most common prefix is one that will use a local node
155 listening on this port::
157 http://127.0.0.1:3456/uri/ + $CAP
159 So, to access the directory named above (which happens to be the
160 publically-writeable sample directory on the Tahoe test grid, described at
161 http://allmydata.org/trac/tahoe/wiki/TestGrid), the URL would be::
163 http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/
165 (note that the colons in the directory-cap are url-encoded into "%3A"
168 Likewise, to access the file named above, use::
170 http://127.0.0.1:3456/uri/URI%3ACHK%3Aime6pvkaxuetdfah2p2f35pe54%3A4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a%3A3%3A10%3A202
172 In the rest of this document, we'll use "$DIRCAP" as shorthand for a read-cap
173 or write-cap that refers to a directory, and "$FILECAP" to abbreviate a cap
174 that refers to a file (whether mutable or immutable). So those URLs above can
177 http://127.0.0.1:3456/uri/$DIRCAP/
178 http://127.0.0.1:3456/uri/$FILECAP
180 The operation summaries below will abbreviate these further, by eliding the
181 server prefix. They will be displayed like this::
190 Tahoe directories contain named child entries, just like directories in a regular
191 local filesystem. These child entries, called "dirnodes", consist of a name,
192 metadata, a write slot, and a read slot. The write and read slots normally contain
193 a write-cap and read-cap referring to the same object, which can be either a file
194 or a subdirectory. The write slot may be empty (actually, both may be empty,
195 but that is unusual).
197 If you have a Tahoe URL that refers to a directory, and want to reference a
198 named child inside it, just append the child name to the URL. For example, if
199 our sample directory contains a file named "welcome.txt", we can refer to
202 http://127.0.0.1:3456/uri/$DIRCAP/welcome.txt
204 (or http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/welcome.txt)
206 Multiple levels of subdirectories can be handled this way::
208 http://127.0.0.1:3456/uri/$DIRCAP/tahoe-source/docs/architecture.rst
210 In this document, when we need to refer to a URL that references a file using
211 this child-of-some-directory format, we'll use the following string::
213 /uri/$DIRCAP/[SUBDIRS../]FILENAME
215 The "[SUBDIRS../]" part means that there are zero or more (optional)
216 subdirectory names in the middle of the URL. The "FILENAME" at the end means
217 that this whole URL refers to a file of some sort, rather than to a
220 When we need to refer specifically to a directory in this way, we'll write::
222 /uri/$DIRCAP/[SUBDIRS../]SUBDIR
225 Note that all components of pathnames in URLs are required to be UTF-8
226 encoded, so "resume.doc" (with an acute accent on both E's) would be accessed
229 http://127.0.0.1:3456/uri/$DIRCAP/r%C3%A9sum%C3%A9.doc
231 Also note that the filenames inside upload POST forms are interpreted using
232 whatever character set was provided in the conventional '_charset' field, and
233 defaults to UTF-8 if not otherwise specified. The JSON representation of each
234 directory contains native Unicode strings. Tahoe directories are specified to
235 contain Unicode filenames, and cannot contain binary strings that are not
236 representable as such.
238 All Tahoe operations that refer to existing files or directories must include
239 a suitable read- or write- cap in the URL: the web-API server won't add one
240 for you. If you don't know the cap, you can't access the file. This allows
241 the security properties of Tahoe caps to be extended across the web-API
245 Slow Operations, Progress, and Cancelling
246 =========================================
248 Certain operations can be expected to take a long time. The "t=deep-check",
249 described below, will recursively visit every file and directory reachable
250 from a given starting point, which can take minutes or even hours for
251 extremely large directory structures. A single long-running HTTP request is a
252 fragile thing: proxies, NAT boxes, browsers, and users may all grow impatient
253 with waiting and give up on the connection.
255 For this reason, long-running operations have an "operation handle", which
256 can be used to poll for status/progress messages while the operation
257 proceeds. This handle can also be used to cancel the operation. These handles
258 are created by the client, and passed in as a an "ophandle=" query argument
259 to the POST or PUT request which starts the operation. The following
260 operations can then be used to retrieve status:
262 ``GET /operations/$HANDLE?output=HTML (with or without t=status)``
264 ``GET /operations/$HANDLE?output=JSON (same)``
266 These two retrieve the current status of the given operation. Each operation
267 presents a different sort of information, but in general the page retrieved
270 * whether the operation is complete, or if it is still running
271 * how much of the operation is complete, and how much is left, if possible
273 Note that the final status output can be quite large: a deep-manifest of a
274 directory structure with 300k directories and 200k unique files is about
275 275MB of JSON, and might take two minutes to generate. For this reason, the
276 full status is not provided until the operation has completed.
278 The HTML form will include a meta-refresh tag, which will cause a regular
279 web browser to reload the status page about 60 seconds later. This tag will
280 be removed once the operation has completed.
282 There may be more status information available under
283 /operations/$HANDLE/$ETC : i.e., the handle forms the root of a URL space.
285 ``POST /operations/$HANDLE?t=cancel``
287 This terminates the operation, and returns an HTML page explaining what was
288 cancelled. If the operation handle has already expired (see below), this
289 POST will return a 404, which indicates that the operation is no longer
290 running (either it was completed or terminated). The response body will be
291 the same as a GET /operations/$HANDLE on this operation handle, and the
292 handle will be expired immediately afterwards.
294 The operation handle will eventually expire, to avoid consuming an unbounded
295 amount of memory. The handle's time-to-live can be reset at any time, by
296 passing a retain-for= argument (with a count of seconds) to either the
297 initial POST that starts the operation, or the subsequent GET request which
298 asks about the operation. For example, if a 'GET
299 /operations/$HANDLE?output=JSON&retain-for=600' query is performed, the
300 handle will remain active for 600 seconds (10 minutes) after the GET was
303 In addition, if the GET includes a release-after-complete=True argument, and
304 the operation has completed, the operation handle will be released
307 If a retain-for= argument is not used, the default handle lifetimes are:
309 * handles will remain valid at least until their operation finishes
310 * uncollected handles for finished operations (i.e. handles for
311 operations that have finished but for which the GET page has not been
312 accessed since completion) will remain valid for four days, or for
313 the total time consumed by the operation, whichever is greater.
314 * collected handles (i.e. the GET page has been retrieved at least once
315 since the operation completed) will remain valid for one day.
317 Many "slow" operations can begin to use unacceptable amounts of memory when
318 operating on large directory structures. The memory usage increases when the
319 ophandle is polled, as the results must be copied into a JSON string, sent
320 over the wire, then parsed by a client. So, as an alternative, many "slow"
321 operations have streaming equivalents. These equivalents do not use operation
322 handles. Instead, they emit line-oriented status results immediately. Client
323 code can cancel the operation by simply closing the HTTP connection.
326 Programmatic Operations
327 =======================
329 Now that we know how to build URLs that refer to files and directories in a
330 Tahoe virtual filesystem, what sorts of operations can we do with those URLs?
331 This section contains a catalog of GET, PUT, DELETE, and POST operations that
332 can be performed on these URLs. This set of operations are aimed at programs
333 that use HTTP to communicate with a Tahoe node. A later section describes
334 operations that are intended for web browsers.
340 ``GET /uri/$FILECAP``
342 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME``
344 This will retrieve the contents of the given file. The HTTP response body
345 will contain the sequence of bytes that make up the file.
347 To view files in a web browser, you may want more control over the
348 Content-Type and Content-Disposition headers. Please see the next section
349 "Browser Operations", for details on how to modify these URLs for that
353 Writing/Uploading A File
354 ------------------------
356 ``PUT /uri/$FILECAP``
358 ``PUT /uri/$DIRCAP/[SUBDIRS../]FILENAME``
360 Upload a file, using the data from the HTTP request body, and add whatever
361 child links and subdirectories are necessary to make the file available at
362 the given location. Once this operation succeeds, a GET on the same URL will
363 retrieve the same contents that were just uploaded. This will create any
364 necessary intermediate subdirectories.
366 To use the /uri/$FILECAP form, $FILECAP must be a write-cap for a mutable file.
368 In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a
369 writeable mutable file, that file's contents will be overwritten
370 in-place. If it is a read-cap for a mutable file, an error will occur.
371 If it is an immutable file, the old file will be discarded, and a new
372 one will be put in its place. If the target file is a writable mutable
373 file, you may also specify an "offset" parameter -- a byte offset that
374 determines where in the mutable file the data from the HTTP request
375 body is placed. This operation is relatively efficient for MDMF mutable
376 files, and is relatively inefficient (but still supported) for SDMF
377 mutable files. If no offset parameter is specified, then the entire
378 file is replaced with the data from the HTTP request body. For an
379 immutable file, the "offset" parameter is not valid.
381 When creating a new file, you can control the type of file created by
382 specifying a format= argument in the query string. format=mdmf creates an MDMF
383 mutable file. format=sdmf creates an SDMF mutable file. format=chk creates an
384 immutable file. The value of the format argument is case-insensitive. For
385 compatibility with previous versions of Tahoe-LAFS, the webapi will also
386 accept a mutable=true argument in the query string. If mutable=true is given,
387 then the new file will be mutable, and its format will be the default mutable
388 file format, as configured on the Tahoe-LAFS node hosting the webapi server.
389 Use of mutable=true is discouraged; new code should use format= instead of
390 mutable=true. If neither format= nor mutable=true are given, the
391 newly-created file will be immutable.
393 This returns the file-cap of the resulting file. If a new file was created
394 by this method, the HTTP response code (as dictated by rfc2616) will be set
395 to 201 CREATED. If an existing file was replaced or modified, the response
398 Note that the 'curl -T localfile http://127.0.0.1:3456/uri/$DIRCAP/foo.txt'
399 command can be used to invoke this operation.
403 This uploads a file, and produces a file-cap for the contents, but does not
404 attach the file into the filesystem. No directories will be modified by
405 this operation. The file-cap is returned as the body of the HTTP response.
407 This method accepts format= and mutable=true as query string arguments, and
408 interprets those arguments in the same way as the linked forms of PUT
409 described immediately above.
411 Creating A New Directory
412 ------------------------
414 ``POST /uri?t=mkdir``
418 Create a new empty directory and return its write-cap as the HTTP response
419 body. This does not make the newly created directory visible from the
420 filesystem. The "PUT" operation is provided for backwards compatibility:
421 new code should use POST.
423 This supports a format= argument in the query string. The format=
424 argument, if specified, controls the format of the directory. format=mdmf
425 indicates that the directory should be stored as an MDMF file; format=sdmf
426 indicates that the directory should be stored as an SDMF file. The value of
427 the format= argument is case-insensitive. If no format= argument is
428 given, the directory's format is determined by the default mutable file
429 format, as configured on the Tahoe-LAFS node responding to the request.
431 ``POST /uri?t=mkdir-with-children``
433 Create a new directory, populated with a set of child nodes, and return its
434 write-cap as the HTTP response body. The new directory is not attached to
435 any other directory: the returned write-cap is the only reference to it.
437 The format of the directory can be controlled with the format= argument in
438 the query string, as described above.
440 Initial children are provided as the body of the POST form (this is more
441 efficient than doing separate mkdir and set_children operations). If the
442 body is empty, the new directory will be empty. If not empty, the body will
443 be interpreted as a UTF-8 JSON-encoded dictionary of children with which the
444 new directory should be populated, using the same format as would be
445 returned in the 'children' value of the t=json GET request, described below.
446 Each dictionary key should be a child name, and each value should be a list
447 of [TYPE, PROPDICT], where PROPDICT contains "rw_uri", "ro_uri", and
448 "metadata" keys (all others are ignored). For example, the PUT request body
452 "Fran\u00e7ais": [ "filenode", {
453 "ro_uri": "URI:CHK:...",
456 "ctime": 1202777696.7564139,
457 "mtime": 1202777696.7564139,
459 "linkcrtime": 1202777696.7564139,
460 "linkmotime": 1202777696.7564139
462 "subdir": [ "dirnode", {
463 "rw_uri": "URI:DIR2:...",
464 "ro_uri": "URI:DIR2-RO:...",
466 "ctime": 1202778102.7589991,
467 "mtime": 1202778111.2160511,
469 "linkcrtime": 1202777696.7564139,
470 "linkmotime": 1202777696.7564139
474 For forward-compatibility, a mutable directory can also contain caps in
475 a format that is unknown to the web-API server. When such caps are retrieved
476 from a mutable directory in a "ro_uri" field, they will be prefixed with
477 the string "ro.", indicating that they must not be decoded without
478 checking that they are read-only. The "ro." prefix must not be stripped
479 off without performing this check. (Future versions of the web-API server
480 will perform it where necessary.)
482 If both the "rw_uri" and "ro_uri" fields are present in a given PROPDICT,
483 and the web-API server recognizes the rw_uri as a write cap, then it will
484 reset the ro_uri to the corresponding read cap and discard the original
485 contents of ro_uri (in order to ensure that the two caps correspond to the
486 same object and that the ro_uri is in fact read-only). However this may not
487 happen for caps in a format unknown to the web-API server. Therefore, when
488 writing a directory the web-API client should ensure that the contents
489 of "rw_uri" and "ro_uri" for a given PROPDICT are a consistent
490 (write cap, read cap) pair if possible. If the web-API client only has
491 one cap and does not know whether it is a write cap or read cap, then
492 it is acceptable to set "rw_uri" to that cap and omit "ro_uri". The
493 client must not put a write cap into a "ro_uri" field.
495 The metadata may have a "no-write" field. If this is set to true in the
496 metadata of a link, it will not be possible to open that link for writing
497 via the SFTP frontend; see `<FTP-and-SFTP.rst>`_ for details.
498 Also, if the "no-write" field is set to true in the metadata of a link to
499 a mutable child, it will cause the link to be diminished to read-only.
501 Note that the web-API-using client application must not provide the
502 "Content-Type: multipart/form-data" header that usually accompanies HTML
503 form submissions, since the body is not formatted this way. Doing so will
504 cause a server error as the lower-level code misparses the request body.
506 Child file names should each be expressed as a Unicode string, then used as
507 keys of the dictionary. The dictionary should then be converted into JSON,
508 and the resulting string encoded into UTF-8. This UTF-8 bytestring should
509 then be used as the POST body.
511 ``POST /uri?t=mkdir-immutable``
513 Like t=mkdir-with-children above, but the new directory will be
514 deep-immutable. This means that the directory itself is immutable, and that
515 it can only contain objects that are treated as being deep-immutable, like
516 immutable files, literal files, and deep-immutable directories.
518 For forward-compatibility, a deep-immutable directory can also contain caps
519 in a format that is unknown to the web-API server. When such caps are retrieved
520 from a deep-immutable directory in a "ro_uri" field, they will be prefixed
521 with the string "imm.", indicating that they must not be decoded without
522 checking that they are immutable. The "imm." prefix must not be stripped
523 off without performing this check. (Future versions of the web-API server
524 will perform it where necessary.)
526 The cap for each child may be given either in the "rw_uri" or "ro_uri"
527 field of the PROPDICT (not both). If a cap is given in the "rw_uri" field,
528 then the web-API server will check that it is an immutable read-cap of a
529 *known* format, and give an error if it is not. If a cap is given in the
530 "ro_uri" field, then the web-API server will still check whether known
531 caps are immutable, but for unknown caps it will simply assume that the
532 cap can be stored, as described above. Note that an attacker would be
533 able to store any cap in an immutable directory, so this check when
534 creating the directory is only to help non-malicious clients to avoid
535 accidentally giving away more authority than intended.
537 A non-empty request body is mandatory, since after the directory is created,
538 it will not be possible to add more children to it.
540 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir``
542 ``PUT /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir``
544 Create new directories as necessary to make sure that the named target
545 ($DIRCAP/SUBDIRS../SUBDIR) is a directory. This will create additional
546 intermediate mutable directories as necessary. If the named target directory
547 already exists, this will make no changes to it.
549 If the final directory is created, it will be empty.
551 This accepts a format= argument in the query string, which controls the
552 format of the named target directory, if it does not already exist. format=
553 is interpreted in the same way as in the POST /uri?t=mkdir form. Note that
554 format= only controls the format of the named target directory;
555 intermediate directories, if created, are created based on the default
556 mutable type, as configured on the Tahoe-LAFS server responding to the
559 This operation will return an error if a blocking file is present at any of
560 the parent names, preventing the server from creating the necessary parent
561 directory; or if it would require changing an immutable directory.
563 The write-cap of the new directory will be returned as the HTTP response
566 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-with-children``
568 Like /uri?t=mkdir-with-children, but the final directory is created as a
569 child of an existing mutable directory. This will create additional
570 intermediate mutable directories as necessary. If the final directory is
571 created, it will be populated with initial children from the POST request
572 body, as described above.
574 This accepts a format= argument in the query string, which controls the
575 format of the target directory, if the target directory is created as part
576 of the operation. format= is interpreted in the same way as in the POST/
577 uri?t=mkdir-with-children operation. Note that format= only controls the
578 format of the named target directory; intermediate directories, if created,
579 are created using the default mutable type setting, as configured on the
580 Tahoe-LAFS server responding to the request.
582 This operation will return an error if a blocking file is present at any of
583 the parent names, preventing the server from creating the necessary parent
584 directory; or if it would require changing an immutable directory; or if
585 the immediate parent directory already has a a child named SUBDIR.
587 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-immutable``
589 Like /uri?t=mkdir-immutable, but the final directory is created as a child
590 of an existing mutable directory. The final directory will be deep-immutable,
591 and will be populated with the children specified as a JSON dictionary in
592 the POST request body.
594 In Tahoe 1.6 this operation creates intermediate mutable directories if
595 necessary, but that behaviour should not be relied on; see ticket #920.
597 This operation will return an error if the parent directory is immutable,
598 or already has a child named SUBDIR.
600 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME``
602 Create a new empty mutable directory and attach it to the given existing
603 directory. This will create additional intermediate directories as necessary.
605 This accepts a format= argument in the query string, which controls the
606 format of the named target directory, if it does not already exist. format=
607 is interpreted in the same way as in the POST /uri?t=mkdir form. Note that
608 format= only controls the format of the named target directory;
609 intermediate directories, if created, are created based on the default
610 mutable type, as configured on the Tahoe-LAFS server responding to the
613 This operation will return an error if a blocking file is present at any of
614 the parent names, preventing the server from creating the necessary parent
615 directory, or if it would require changing any immutable directory.
617 The URL of this operation points to the parent of the bottommost new directory,
618 whereas the /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir operation above has a URL
619 that points directly to the bottommost new directory.
621 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME``
623 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME, but the new directory will
624 be populated with initial children via the POST request body. This command
625 will create additional intermediate mutable directories as necessary.
627 This accepts a format= argument in the query string, which controls the
628 format of the target directory, if the target directory is created as part
629 of the operation. format= is interpreted in the same way as in the POST/
630 uri?t=mkdir-with-children operation. Note that format= only controls the
631 format of the named target directory; intermediate directories, if created,
632 are created using the default mutable type setting, as configured on the
633 Tahoe-LAFS server responding to the request.
635 This operation will return an error if a blocking file is present at any of
636 the parent names, preventing the server from creating the necessary parent
637 directory; or if it would require changing an immutable directory; or if
638 the immediate parent directory already has a a child named NAME.
640 Note that the name= argument must be passed as a queryarg, because the POST
641 request body is used for the initial children JSON.
643 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-immutable&name=NAME``
645 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME, but the
646 final directory will be deep-immutable. The children are specified as a
647 JSON dictionary in the POST request body. Again, the name= argument must be
648 passed as a queryarg.
650 In Tahoe 1.6 this operation creates intermediate mutable directories if
651 necessary, but that behaviour should not be relied on; see ticket #920.
653 This operation will return an error if the parent directory is immutable,
654 or already has a child named NAME.
657 Getting Information About A File Or Directory (as JSON)
658 -------------------------------------------------------
660 ``GET /uri/$FILECAP?t=json``
662 ``GET /uri/$DIRCAP?t=json``
664 ``GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json``
666 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json``
668 This returns a machine-parseable JSON-encoded description of the given
669 object. The JSON always contains a list, and the first element of the list is
670 always a flag that indicates whether the referenced object is a file or a
671 directory. If it is a capability to a file, then the information includes
672 file size and URI, like this::
674 GET /uri/$FILECAP?t=json :
678 "verify_uri": verify_uri,
684 If it is a capability to a directory followed by a path from that directory
685 to a file, then the information also includes metadata from the link to the
686 file in the parent directory, like this::
688 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json
692 "verify_uri": verify_uri,
697 "ctime": 1202777696.7564139,
698 "mtime": 1202777696.7564139,
700 "linkcrtime": 1202777696.7564139,
701 "linkmotime": 1202777696.7564139
704 If it is a directory, then it includes information about the children of
705 this directory, as a mapping from child name to a set of data about the
706 child (the same data that would appear in a corresponding GET?t=json of the
707 child itself). The child entries also include metadata about each child,
708 including link-creation- and link-change- timestamps. The output looks like
711 GET /uri/$DIRCAP?t=json :
712 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json :
715 "rw_uri": read_write_uri,
716 "ro_uri": read_only_uri,
717 "verify_uri": verify_uri,
721 "foo.txt": [ "filenode",
726 "ctime": 1202777696.7564139,
727 "mtime": 1202777696.7564139,
729 "linkcrtime": 1202777696.7564139,
730 "linkmotime": 1202777696.7564139
732 "subdir": [ "dirnode",
737 "ctime": 1202778102.7589991,
738 "mtime": 1202778111.2160511,
740 "linkcrtime": 1202777696.7564139,
741 "linkmotime": 1202777696.7564139
745 In the above example, note how 'children' is a dictionary in which the keys
746 are child names and the values depend upon whether the child is a file or a
747 directory. The value is mostly the same as the JSON representation of the
748 child object (except that directories do not recurse -- the "children"
749 entry of the child is omitted, and the directory view includes the metadata
750 that is stored on the directory edge).
752 The rw_uri field will be present in the information about a directory
753 if and only if you have read-write access to that directory. The verify_uri
754 field will be present if and only if the object has a verify-cap
755 (non-distributed LIT files do not have verify-caps).
757 If the cap is of an unknown format, then the file size and verify_uri will
760 GET /uri/$UNKNOWNCAP?t=json :
763 "ro_uri": unknown_read_uri
766 GET /uri/$DIRCAP/[SUBDIRS../]UNKNOWNCHILDNAME?t=json :
769 "rw_uri": unknown_write_uri,
770 "ro_uri": unknown_read_uri,
773 "ctime": 1202777696.7564139,
774 "mtime": 1202777696.7564139,
776 "linkcrtime": 1202777696.7564139,
777 "linkmotime": 1202777696.7564139
780 As in the case of file nodes, the metadata will only be present when the
781 capability is to a directory followed by a path. The "mutable" field is also
782 not always present; when it is absent, the mutability of the object is not
788 The value of the 'tahoe':'linkmotime' key is updated whenever a link to a
789 child is set. The value of the 'tahoe':'linkcrtime' key is updated whenever
790 a link to a child is created -- i.e. when there was not previously a link
793 Note however, that if the edge in the Tahoe filesystem points to a mutable
794 file and the contents of that mutable file is changed, then the
795 'tahoe':'linkmotime' value on that edge will *not* be updated, since the
796 edge itself wasn't updated -- only the mutable file was.
798 The timestamps are represented as a number of seconds since the UNIX epoch
799 (1970-01-01 00:00:00 UTC), with leap seconds not being counted in the long
802 In Tahoe earlier than v1.4.0, 'mtime' and 'ctime' keys were populated
803 instead of the 'tahoe':'linkmotime' and 'tahoe':'linkcrtime' keys. Starting
804 in Tahoe v1.4.0, the 'linkmotime'/'linkcrtime' keys in the 'tahoe' sub-dict
805 are populated. However, prior to Tahoe v1.7beta, a bug caused the 'tahoe'
806 sub-dict to be deleted by web-API requests in which new metadata is
807 specified, and not to be added to existing child links that lack it.
809 From Tahoe v1.7.0 onward, the 'mtime' and 'ctime' fields are no longer
810 populated or updated (see ticket #924), except by "tahoe backup" as
811 explained below. For backward compatibility, when an existing link is
812 updated and 'tahoe':'linkcrtime' is not present in the previous metadata
813 but 'ctime' is, the old value of 'ctime' is used as the new value of
814 'tahoe':'linkcrtime'.
816 The reason we added the new fields in Tahoe v1.4.0 is that there is a
817 "set_children" API (described below) which you can use to overwrite the
818 values of the 'mtime'/'ctime' pair, and this API is used by the
819 "tahoe backup" command (in Tahoe v1.3.0 and later) to set the 'mtime' and
820 'ctime' values when backing up files from a local filesystem into the
821 Tahoe filesystem. As of Tahoe v1.4.0, the set_children API cannot be used
822 to set anything under the 'tahoe' key of the metadata dict -- if you
823 include 'tahoe' keys in your 'metadata' arguments then it will silently
826 Therefore, if the 'tahoe' sub-dict is present, you can rely on the
827 'linkcrtime' and 'linkmotime' values therein to have the semantics described
828 above. (This is assuming that only official Tahoe clients have been used to
829 write those links, and that their system clocks were set to what you expected
830 -- there is nothing preventing someone from editing their Tahoe client or
831 writing their own Tahoe client which would overwrite those values however
832 they like, and there is nothing to constrain their system clock from taking
835 When an edge is created or updated by "tahoe backup", the 'mtime' and
836 'ctime' keys on that edge are set as follows:
838 * 'mtime' is set to the timestamp read from the local filesystem for the
839 "mtime" of the local file in question, which means the last time the
840 contents of that file were changed.
842 * On Windows, 'ctime' is set to the creation timestamp for the file
843 read from the local filesystem. On other platforms, 'ctime' is set to
844 the UNIX "ctime" of the local file, which means the last time that
845 either the contents or the metadata of the local file was changed.
847 There are several ways that the 'ctime' field could be confusing:
849 1. You might be confused about whether it reflects the time of the creation
850 of a link in the Tahoe filesystem (by a version of Tahoe < v1.7.0) or a
851 timestamp copied in by "tahoe backup" from a local filesystem.
853 2. You might be confused about whether it is a copy of the file creation
854 time (if "tahoe backup" was run on a Windows system) or of the last
855 contents-or-metadata change (if "tahoe backup" was run on a different
858 3. You might be confused by the fact that changing the contents of a
859 mutable file in Tahoe doesn't have any effect on any links pointing at
860 that file in any directories, although "tahoe backup" sets the link
861 'ctime'/'mtime' to reflect timestamps about the local file corresponding
862 to the Tahoe file to which the link points.
864 4. Also, quite apart from Tahoe, you might be confused about the meaning
865 of the "ctime" in UNIX local filesystems, which people sometimes think
866 means file creation time, but which actually means, in UNIX local
867 filesystems, the most recent time that the file contents or the file
868 metadata (such as owner, permission bits, extended attributes, etc.)
869 has changed. Note that although "ctime" does not mean file creation time
870 in UNIX, links created by a version of Tahoe prior to v1.7.0, and never
871 written by "tahoe backup", will have 'ctime' set to the link creation
875 Attaching an Existing File or Directory by its read- or write-cap
876 -----------------------------------------------------------------
878 ``PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri``
880 This attaches a child object (either a file or directory) to a specified
881 location in the virtual filesystem. The child object is referenced by its
882 read- or write- cap, as provided in the HTTP request body. This will create
883 intermediate directories as necessary.
885 This is similar to a UNIX hardlink: by referencing a previously-uploaded file
886 (or previously-created directory) instead of uploading/creating a new one,
887 you can create two references to the same object.
889 The read- or write- cap of the child is provided in the body of the HTTP
890 request, and this same cap is returned in the response body.
892 The default behavior is to overwrite any existing object at the same
893 location. To prevent this (and make the operation return an error instead
894 of overwriting), add a "replace=false" argument, as "?t=uri&replace=false".
895 With replace=false, this operation will return an HTTP 409 "Conflict" error
896 if there is already an object at the given location, rather than
897 overwriting the existing object. To allow the operation to overwrite a
898 file, but return an error when trying to overwrite a directory, use
899 "replace=only-files" (this behavior is closer to the traditional UNIX "mv"
900 command). Note that "true", "t", and "1" are all synonyms for "True", and
901 "false", "f", and "0" are synonyms for "False", and the parameter is
904 Note that this operation does not take its child cap in the form of
905 separate "rw_uri" and "ro_uri" fields. Therefore, it cannot accept a
906 child cap in a format unknown to the web-API server, unless its URI
907 starts with "ro." or "imm.". This restriction is necessary because the
908 server is not able to attenuate an unknown write cap to a read cap.
909 Unknown URIs starting with "ro." or "imm.", on the other hand, are
910 assumed to represent read caps. The client should not prefix a write
911 cap with "ro." or "imm." and pass it to this operation, since that
912 would result in granting the cap's write authority to holders of the
916 Adding Multiple Files or Directories to a Parent Directory at Once
917 ------------------------------------------------------------------
919 ``POST /uri/$DIRCAP/[SUBDIRS..]?t=set_children``
921 ``POST /uri/$DIRCAP/[SUBDIRS..]?t=set-children`` (Tahoe >= v1.6)
923 This command adds multiple children to a directory in a single operation.
924 It reads the request body and interprets it as a JSON-encoded description
925 of the child names and read/write-caps that should be added.
927 The body should be a JSON-encoded dictionary, in the same format as the
928 "children" value returned by the "GET /uri/$DIRCAP?t=json" operation
929 described above. In this format, each key is a child names, and the
930 corresponding value is a tuple of (type, childinfo). "type" is ignored, and
931 "childinfo" is a dictionary that contains "rw_uri", "ro_uri", and
932 "metadata" keys. You can take the output of "GET /uri/$DIRCAP1?t=json" and
933 use it as the input to "POST /uri/$DIRCAP2?t=set_children" to make DIR2
934 look very much like DIR1 (except for any existing children of DIR2 that
935 were not overwritten, and any existing "tahoe" metadata keys as described
938 When the set_children request contains a child name that already exists in
939 the target directory, this command defaults to overwriting that child with
940 the new value (both child cap and metadata, but if the JSON data does not
941 contain a "metadata" key, the old child's metadata is preserved). The
942 command takes a boolean "overwrite=" query argument to control this
943 behavior. If you use "?t=set_children&overwrite=false", then an attempt to
944 replace an existing child will instead cause an error.
946 Any "tahoe" key in the new child's "metadata" value is ignored. Any
947 existing "tahoe" metadata is preserved. The metadata["tahoe"] value is
948 reserved for metadata generated by the tahoe node itself. The only two keys
949 currently placed here are "linkcrtime" and "linkmotime". For details, see
950 the section above entitled "Get Information About A File Or Directory (as
951 JSON)", in the "About the metadata" subsection.
953 Note that this command was introduced with the name "set_children", which
954 uses an underscore rather than a hyphen as other multi-word command names
955 do. The variant with a hyphen is now accepted, but clients that desire
956 backward compatibility should continue to use "set_children".
959 Unlinking a File or Directory
960 -----------------------------
962 ``DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME``
964 This removes the given name from its parent directory. CHILDNAME is the
965 name to be removed, and $DIRCAP/SUBDIRS.. indicates the directory that will
968 Note that this does not actually delete the file or directory that the name
969 points to from the tahoe grid -- it only unlinks the named reference from
970 this directory. If there are other names in this directory or in other
971 directories that point to the resource, then it will remain accessible
972 through those paths. Even if all names pointing to this object are removed
973 from their parent directories, then someone with possession of its read-cap
974 can continue to access the object through that cap.
976 The object will only become completely unreachable once 1: there are no
977 reachable directories that reference it, and 2: nobody is holding a read-
978 or write- cap to the object. (This behavior is very similar to the way
979 hardlinks and anonymous files work in traditional UNIX filesystems).
981 This operation will not modify more than a single directory. Intermediate
982 directories which were implicitly created by PUT or POST methods will *not*
983 be automatically removed by DELETE.
985 This method returns the file- or directory- cap of the object that was just
989 Browser Operations: Human-oriented interfaces
990 =============================================
992 This section describes the HTTP operations that provide support for humans
993 running a web browser. Most of these operations use HTML forms that use POST
994 to drive the Tahoe node. This section is intended for HTML authors who want
995 to write web pages that contain forms and buttons which manipulate the Tahoe
998 Note that for all POST operations, the arguments listed can be provided
999 either as URL query arguments or as form body fields. URL query arguments are
1000 separated from the main URL by "?", and from each other by "&". For example,
1001 "POST /uri/$DIRCAP?t=upload&mutable=true". Form body fields are usually
1002 specified by using <input type="hidden"> elements. For clarity, the
1003 descriptions below display the most significant arguments as URL query args.
1006 Viewing A Directory (as HTML)
1007 -----------------------------
1009 ``GET /uri/$DIRCAP/[SUBDIRS../]``
1011 This returns an HTML page, intended to be displayed to a human by a web
1012 browser, which contains HREF links to all files and directories reachable
1013 from this directory. These HREF links do not have a t= argument, meaning
1014 that a human who follows them will get pages also meant for a human. It also
1015 contains forms to upload new files, and to unlink files and directories
1016 from their parent directory. Those forms use POST methods to do their job.
1019 Viewing/Downloading a File
1020 --------------------------
1022 ``GET /uri/$FILECAP``
1024 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME``
1026 This will retrieve the contents of the given file. The HTTP response body
1027 will contain the sequence of bytes that make up the file.
1029 If you want the HTTP response to include a useful Content-Type header,
1030 either use the second form (which starts with a $DIRCAP), or add a
1031 "filename=foo" query argument, like "GET /uri/$FILECAP?filename=foo.jpg".
1032 The bare "GET /uri/$FILECAP" does not give the Tahoe node enough information
1033 to determine a Content-Type (since Tahoe immutable files are merely
1034 sequences of bytes, not typed+named file objects).
1036 If the URL has both filename= and "save=true" in the query arguments, then
1037 the server to add a "Content-Disposition: attachment" header, along with a
1038 filename= parameter. When a user clicks on such a link, most browsers will
1039 offer to let the user save the file instead of displaying it inline (indeed,
1040 most browsers will refuse to display it inline). "true", "t", "1", and other
1041 case-insensitive equivalents are all treated the same.
1043 Character-set handling in URLs and HTTP headers is a dubious art [1]_. For
1044 maximum compatibility, Tahoe simply copies the bytes from the filename=
1045 argument into the Content-Disposition header's filename= parameter, without
1046 trying to interpret them in any particular way.
1049 ``GET /named/$FILECAP/FILENAME``
1051 This is an alternate download form which makes it easier to get the correct
1052 filename. The Tahoe server will provide the contents of the given file, with
1053 a Content-Type header derived from the given filename. This form is used to
1054 get browsers to use the "Save Link As" feature correctly, and also helps
1055 command-line tools like "wget" and "curl" use the right filename. Note that
1056 this form can *only* be used with file caps; it is an error to use a
1057 directory cap after the /named/ prefix.
1060 Getting Information About A File Or Directory (as HTML)
1061 -------------------------------------------------------
1063 ``GET /uri/$FILECAP?t=info``
1065 ``GET /uri/$DIRCAP/?t=info``
1067 ``GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR/?t=info``
1069 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=info``
1071 This returns a human-oriented HTML page with more detail about the selected
1072 file or directory object. This page contains the following items:
1076 * JSON representation
1077 * raw contents (text/plain)
1078 * access caps (URIs): verify-cap, read-cap, write-cap (for mutable objects)
1079 * check/verify/repair form
1080 * deep-check/deep-size/deep-stats/manifest (for directories)
1081 * replace-contents form (for mutable files)
1084 Creating a Directory
1085 --------------------
1087 ``POST /uri?t=mkdir``
1089 This creates a new empty directory, but does not attach it to the virtual
1092 If a "redirect_to_result=true" argument is provided, then the HTTP response
1093 will cause the web browser to be redirected to a /uri/$DIRCAP page that
1094 gives access to the newly-created directory. If you bookmark this page,
1095 you'll be able to get back to the directory again in the future. This is the
1096 recommended way to start working with a Tahoe server: create a new unlinked
1097 directory (using redirect_to_result=true), then bookmark the resulting
1098 /uri/$DIRCAP page. There is a "create directory" button on the Welcome page
1099 to invoke this action.
1101 This accepts a format= argument in the query string. Refer to the
1102 documentation of the PUT /uri?t=mkdir operation in `Creating A
1103 New Directory`_ for information on the behavior of the format= argument.
1105 If "redirect_to_result=true" is not provided (or is given a value of
1106 "false"), then the HTTP response body will simply be the write-cap of the
1109 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME``
1111 This creates a new empty directory as a child of the designated SUBDIR. This
1112 will create additional intermediate directories as necessary.
1114 This accepts a format= argument in the query string. Refer to the
1115 documentation of POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME in
1116 `Creating A New Directory`_ for information on the behavior of the format=
1119 If a "when_done=URL" argument is provided, the HTTP response will cause the
1120 web browser to redirect to the given URL. This provides a convenient way to
1121 return the browser to the directory that was just modified. Without a
1122 when_done= argument, the HTTP response will simply contain the write-cap of
1123 the directory that was just created.
1129 ``POST /uri?t=upload``
1131 This uploads a file, and produces a file-cap for the contents, but does not
1132 attach the file into the filesystem. No directories will be modified by
1135 The file must be provided as the "file" field of an HTML encoded form body,
1136 produced in response to an HTML form like this::
1138 <form action="/uri" method="POST" enctype="multipart/form-data">
1139 <input type="hidden" name="t" value="upload" />
1140 <input type="file" name="file" />
1141 <input type="submit" value="Upload Unlinked" />
1144 If a "when_done=URL" argument is provided, the response body will cause the
1145 browser to redirect to the given URL. If the when_done= URL has the string
1146 "%(uri)s" in it, that string will be replaced by a URL-escaped form of the
1147 newly created file-cap. (Note that without this substitution, there is no
1148 way to access the file that was just uploaded).
1150 The default (in the absence of when_done=) is to return an HTML page that
1151 describes the results of the upload. This page will contain information
1152 about which storage servers were used for the upload, how long each
1153 operation took, etc.
1155 This accepts format= and mutable=true query string arguments. Refer to
1156 `Writing/Uploading A File`_ for information on the behavior of format= and
1159 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=upload``
1161 This uploads a file, and attaches it as a new child of the given directory,
1162 which must be mutable. The file must be provided as the "file" field of an
1163 HTML-encoded form body, produced in response to an HTML form like this::
1165 <form action="." method="POST" enctype="multipart/form-data">
1166 <input type="hidden" name="t" value="upload" />
1167 <input type="file" name="file" />
1168 <input type="submit" value="Upload" />
1171 A "name=" argument can be provided to specify the new child's name,
1172 otherwise it will be taken from the "filename" field of the upload form
1173 (most web browsers will copy the last component of the original file's
1174 pathname into this field). To avoid confusion, name= is not allowed to
1177 If there is already a child with that name, and it is a mutable file, then
1178 its contents are replaced with the data being uploaded. If it is not a
1179 mutable file, the default behavior is to remove the existing child before
1180 creating a new one. To prevent this (and make the operation return an error
1181 instead of overwriting the old child), add a "replace=false" argument, as
1182 "?t=upload&replace=false". With replace=false, this operation will return an
1183 HTTP 409 "Conflict" error if there is already an object at the given
1184 location, rather than overwriting the existing object. Note that "true",
1185 "t", and "1" are all synonyms for "True", and "false", "f", and "0" are
1186 synonyms for "False". the parameter is case-insensitive.
1188 This will create additional intermediate directories as necessary, although
1189 since it is expected to be triggered by a form that was retrieved by "GET
1190 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1193 This accepts format= and mutable=true query string arguments. Refer to
1194 `Writing/Uploading A File`_ for information on the behavior of format= and
1197 If a "when_done=URL" argument is provided, the HTTP response will cause the
1198 web browser to redirect to the given URL. This provides a convenient way to
1199 return the browser to the directory that was just modified. Without a
1200 when_done= argument, the HTTP response will simply contain the file-cap of
1201 the file that was just uploaded (a write-cap for mutable files, or a
1202 read-cap for immutable files).
1204 ``POST /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=upload``
1206 This also uploads a file and attaches it as a new child of the given
1207 directory, which must be mutable. It is a slight variant of the previous
1208 operation, as the URL refers to the target file rather than the parent
1209 directory. It is otherwise identical: this accepts mutable= and when_done=
1212 ``POST /uri/$FILECAP?t=upload``
1214 This modifies the contents of an existing mutable file in-place. An error is
1215 signalled if $FILECAP does not refer to a mutable file. It behaves just like
1216 the "PUT /uri/$FILECAP" form, but uses a POST for the benefit of HTML forms
1220 Attaching An Existing File Or Directory (by URI)
1221 ------------------------------------------------
1223 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=uri&name=CHILDNAME&uri=CHILDCAP``
1225 This attaches a given read- or write- cap "CHILDCAP" to the designated
1226 directory, with a specified child name. This behaves much like the PUT t=uri
1227 operation, and is a lot like a UNIX hardlink. It is subject to the same
1228 restrictions as that operation on the use of cap formats unknown to the
1231 This will create additional intermediate directories as necessary, although
1232 since it is expected to be triggered by a form that was retrieved by "GET
1233 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1236 This accepts the same replace= argument as POST t=upload.
1242 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME``
1244 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=unlink&name=CHILDNAME``
1246 This instructs the node to remove a child object (file or subdirectory) from
1247 the given directory, which must be mutable. Note that the entire subtree is
1248 unlinked from the parent. Unlike deleting a subdirectory in a UNIX local
1249 filesystem, the subtree need not be empty; if it isn't, then other references
1250 into the subtree will see that the child subdirectories are not modified by
1251 this operation. Only the link from the given directory to its child is severed.
1253 In Tahoe-LAFS v1.9.0 and later, t=unlink can be used as a synonym for t=delete.
1254 If interoperability with older web-API servers is required, t=delete should
1261 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=rename&from_name=OLD&to_name=NEW``
1263 This instructs the node to rename a child of the given directory, which must
1264 be mutable. This has a similar effect to removing the child, then adding the
1265 same child-cap under the new name, except that it preserves metadata. This
1266 operation cannot move the child to a different directory.
1268 This operation will replace any existing child of the new name, making it
1269 behave like the UNIX "``mv -f``" command.
1274 ``GET /uri?uri=$CAP``
1276 This causes a redirect to /uri/$CAP, and retains any additional query
1277 arguments (like filename= or save=). This is for the convenience of web
1278 forms which allow the user to paste in a read- or write- cap (obtained
1279 through some out-of-band channel, like IM or email).
1281 Note that this form merely redirects to the specific file or directory
1282 indicated by the $CAP: unlike the GET /uri/$DIRCAP form, you cannot
1283 traverse to children by appending additional path segments to the URL.
1285 ``GET /uri/$DIRCAP/[SUBDIRS../]?t=rename-form&name=$CHILDNAME``
1287 This provides a useful facility to browser-based user interfaces. It
1288 returns a page containing a form targetting the "POST $DIRCAP t=rename"
1289 functionality described above, with the provided $CHILDNAME present in the
1290 'from_name' field of that form. I.e. this presents a form offering to
1291 rename $CHILDNAME, requesting the new name, and submitting POST rename.
1293 ``GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri``
1295 This returns the file- or directory- cap for the specified object.
1297 ``GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=readonly-uri``
1299 This returns a read-only file- or directory- cap for the specified object.
1300 If the object is an immutable file, this will return the same value as
1304 Debugging and Testing Features
1305 ------------------------------
1307 These URLs are less-likely to be helpful to the casual Tahoe user, and are
1308 mainly intended for developers.
1310 ``POST $URL?t=check``
1312 This triggers the FileChecker to determine the current "health" of the
1313 given file or directory, by counting how many shares are available. The
1314 page that is returned will display the results. This can be used as a "show
1315 me detailed information about this file" page.
1317 If a verify=true argument is provided, the node will perform a more
1318 intensive check, downloading and verifying every single bit of every share.
1320 If an add-lease=true argument is provided, the node will also add (or
1321 renew) a lease to every share it encounters. Each lease will keep the share
1322 alive for a certain period of time (one month by default). Once the last
1323 lease expires or is explicitly cancelled, the storage server is allowed to
1326 If an output=JSON argument is provided, the response will be
1327 machine-readable JSON instead of human-oriented HTML. The data is a
1328 dictionary with the following keys::
1330 storage-index: a base32-encoded string with the objects's storage index,
1331 or an empty string for LIT files
1332 summary: a string, with a one-line summary of the stats of the file
1333 results: a dictionary that describes the state of the file. For LIT files,
1334 this dictionary has only the 'healthy' key, which will always be
1335 True. For distributed files, this dictionary has the following
1337 count-shares-good: the number of good shares that were found
1338 count-shares-needed: 'k', the number of shares required for recovery
1339 count-shares-expected: 'N', the number of total shares generated
1340 count-good-share-hosts: this was intended to be the number of distinct
1341 storage servers with good shares. It is currently
1342 (as of Tahoe-LAFS v1.8.0) computed incorrectly;
1344 count-wrong-shares: for mutable files, the number of shares for
1345 versions other than the 'best' one (highest
1346 sequence number, highest roothash). These are
1348 count-recoverable-versions: for mutable files, the number of
1349 recoverable versions of the file. For
1350 a healthy file, this will equal 1.
1351 count-unrecoverable-versions: for mutable files, the number of
1352 unrecoverable versions of the file.
1353 For a healthy file, this will be 0.
1354 count-corrupt-shares: the number of shares with integrity failures
1355 list-corrupt-shares: a list of "share locators", one for each share
1356 that was found to be corrupt. Each share locator
1357 is a list of (serverid, storage_index, sharenum).
1358 needs-rebalancing: (bool) True if there are multiple shares on a single
1359 storage server, indicating a reduction in reliability
1360 that could be resolved by moving shares to new
1362 servers-responding: list of base32-encoded storage server identifiers,
1363 one for each server which responded to the share
1365 healthy: (bool) True if the file is completely healthy, False otherwise.
1366 Healthy files have at least N good shares. Overlapping shares
1367 do not currently cause a file to be marked unhealthy. If there
1368 are at least N good shares, then corrupt shares do not cause the
1369 file to be marked unhealthy, although the corrupt shares will be
1370 listed in the results (list-corrupt-shares) and should be manually
1371 removed to wasting time in subsequent downloads (as the
1372 downloader rediscovers the corruption and uses alternate shares).
1373 Future compatibility: the meaning of this field may change to
1374 reflect whether the servers-of-happiness criterion is met
1376 sharemap: dict mapping share identifier to list of serverids
1377 (base32-encoded strings). This indicates which servers are
1378 holding which shares. For immutable files, the shareid is
1379 an integer (the share number, from 0 to N-1). For
1380 immutable files, it is a string of the form
1381 'seq%d-%s-sh%d', containing the sequence number, the
1382 roothash, and the share number.
1384 ``POST $URL?t=start-deep-check`` (must add &ophandle=XYZ)
1386 This initiates a recursive walk of all files and directories reachable from
1387 the target, performing a check on each one just like t=check. The result
1388 page will contain a summary of the results, including details on any
1389 file/directory that was not fully healthy.
1391 t=start-deep-check can only be invoked on a directory. An error (400
1392 BAD_REQUEST) will be signalled if it is invoked on a file. The recursive
1393 walker will deal with loops safely.
1395 This accepts the same verify= and add-lease= arguments as t=check.
1397 Since this operation can take a long time (perhaps a second per object),
1398 the ophandle= argument is required (see "Slow Operations, Progress, and
1399 Cancelling" above). The response to this POST will be a redirect to the
1400 corresponding /operations/$HANDLE page (with output=HTML or output=JSON to
1401 match the output= argument given to the POST). The deep-check operation
1402 will continue to run in the background, and the /operations page should be
1403 used to find out when the operation is done.
1405 Detailed check results for non-healthy files and directories will be
1406 available under /operations/$HANDLE/$STORAGEINDEX, and the HTML status will
1407 contain links to these detailed results.
1409 The HTML /operations/$HANDLE page for incomplete operations will contain a
1410 meta-refresh tag, set to 60 seconds, so that a browser which uses
1411 deep-check will automatically poll until the operation has completed.
1413 The JSON page (/options/$HANDLE?output=JSON) will contain a
1414 machine-readable JSON dictionary with the following keys::
1416 finished: a boolean, True if the operation is complete, else False. Some
1417 of the remaining keys may not be present until the operation
1419 root-storage-index: a base32-encoded string with the storage index of the
1420 starting point of the deep-check operation
1421 count-objects-checked: count of how many objects were checked. Note that
1422 non-distributed objects (i.e. small immutable LIT
1423 files) are not checked, since for these objects,
1424 the data is contained entirely in the URI.
1425 count-objects-healthy: how many of those objects were completely healthy
1426 count-objects-unhealthy: how many were damaged in some way
1427 count-corrupt-shares: how many shares were found to have corruption,
1428 summed over all objects examined
1429 list-corrupt-shares: a list of "share identifiers", one for each share
1430 that was found to be corrupt. Each share identifier
1431 is a list of (serverid, storage_index, sharenum).
1432 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1433 each file that was not fully healthy. 'pathname' is
1434 a list of strings (which can be joined by "/"
1435 characters to turn it into a single string),
1436 relative to the directory on which deep-check was
1437 invoked. The 'check-results' field is the same as
1438 that returned by t=check&output=JSON, described
1440 stats: a dictionary with the same keys as the t=start-deep-stats command
1443 ``POST $URL?t=stream-deep-check``
1445 This initiates a recursive walk of all files and directories reachable from
1446 the target, performing a check on each one just like t=check. For each
1447 unique object (duplicates are skipped), a single line of JSON is emitted to
1448 the HTTP response channel (or an error indication, see below). When the walk
1449 is complete, a final line of JSON is emitted which contains the accumulated
1450 file-size/count "deep-stats" data.
1452 This command takes the same arguments as t=start-deep-check.
1454 A CLI tool can split the response stream on newlines into "response units",
1455 and parse each response unit as JSON. Each such parsed unit will be a
1456 dictionary, and will contain at least the "type" key: a string, one of
1457 "file", "directory", or "stats".
1459 For all units that have a type of "file" or "directory", the dictionary will
1460 contain the following keys::
1462 "path": a list of strings, with the path that is traversed to reach the
1464 "cap": a write-cap URI for the file or directory, if available, else a
1466 "verifycap": a verify-cap URI for the file or directory
1467 "repaircap": an URI for the weakest cap that can still be used to repair
1469 "storage-index": a base32 storage index for the object
1470 "check-results": a copy of the dictionary which would be returned by
1471 t=check&output=json, with three top-level keys:
1472 "storage-index", "summary", and "results", and a variety
1473 of counts and sharemaps in the "results" value.
1475 Note that non-distributed files (i.e. LIT files) will have values of None
1476 for verifycap, repaircap, and storage-index, since these files can neither
1477 be verified nor repaired, and are not stored on the storage servers.
1478 Likewise the check-results dictionary will be limited: an empty string for
1479 storage-index, and a results dictionary with only the "healthy" key.
1481 The last unit in the stream will have a type of "stats", and will contain
1482 the keys described in the "start-deep-stats" operation, below.
1484 If any errors occur during the traversal (specifically if a directory is
1485 unrecoverable, such that further traversal is not possible), an error
1486 indication is written to the response body, instead of the usual line of
1487 JSON. This error indication line will begin with the string "ERROR:" (in all
1488 caps), and contain a summary of the error on the rest of the line. The
1489 remaining lines of the response body will be a python exception. The client
1490 application should look for the ERROR: and stop processing JSON as soon as
1491 it is seen. Note that neither a file being unrecoverable nor a directory
1492 merely being unhealthy will cause traversal to stop. The line just before
1493 the ERROR: will describe the directory that was untraversable, since the
1494 unit is emitted to the HTTP response body before the child is traversed.
1497 ``POST $URL?t=check&repair=true``
1499 This performs a health check of the given file or directory, and if the
1500 checker determines that the object is not healthy (some shares are missing
1501 or corrupted), it will perform a "repair". During repair, any missing
1502 shares will be regenerated and uploaded to new servers.
1504 This accepts the same verify=true and add-lease= arguments as t=check. When
1505 an output=JSON argument is provided, the machine-readable JSON response
1506 will contain the following keys::
1508 storage-index: a base32-encoded string with the objects's storage index,
1509 or an empty string for LIT files
1510 repair-attempted: (bool) True if repair was attempted
1511 repair-successful: (bool) True if repair was attempted and the file was
1512 fully healthy afterwards. False if no repair was
1513 attempted, or if a repair attempt failed.
1514 pre-repair-results: a dictionary that describes the state of the file
1515 before any repair was performed. This contains exactly
1516 the same keys as the 'results' value of the t=check
1517 response, described above.
1518 post-repair-results: a dictionary that describes the state of the file
1519 after any repair was performed. If no repair was
1520 performed, post-repair-results and pre-repair-results
1521 will be the same. This contains exactly the same keys
1522 as the 'results' value of the t=check response,
1525 ``POST $URL?t=start-deep-check&repair=true`` (must add &ophandle=XYZ)
1527 This triggers a recursive walk of all files and directories, performing a
1528 t=check&repair=true on each one.
1530 Like t=start-deep-check without the repair= argument, this can only be
1531 invoked on a directory. An error (400 BAD_REQUEST) will be signalled if it
1532 is invoked on a file. The recursive walker will deal with loops safely.
1534 This accepts the same verify= and add-lease= arguments as
1535 t=start-deep-check. It uses the same ophandle= mechanism as
1536 start-deep-check. When an output=JSON argument is provided, the response
1537 will contain the following keys::
1539 finished: (bool) True if the operation has completed, else False
1540 root-storage-index: a base32-encoded string with the storage index of the
1541 starting point of the deep-check operation
1542 count-objects-checked: count of how many objects were checked
1544 count-objects-healthy-pre-repair: how many of those objects were completely
1545 healthy, before any repair
1546 count-objects-unhealthy-pre-repair: how many were damaged in some way
1547 count-objects-healthy-post-repair: how many of those objects were completely
1548 healthy, after any repair
1549 count-objects-unhealthy-post-repair: how many were damaged in some way
1551 count-repairs-attempted: repairs were attempted on this many objects.
1552 count-repairs-successful: how many repairs resulted in healthy objects
1553 count-repairs-unsuccessful: how many repairs resulted did not results in
1554 completely healthy objects
1555 count-corrupt-shares-pre-repair: how many shares were found to have
1556 corruption, summed over all objects
1557 examined, before any repair
1558 count-corrupt-shares-post-repair: how many shares were found to have
1559 corruption, summed over all objects
1560 examined, after any repair
1561 list-corrupt-shares: a list of "share identifiers", one for each share
1562 that was found to be corrupt (before any repair).
1563 Each share identifier is a list of (serverid,
1564 storage_index, sharenum).
1565 list-remaining-corrupt-shares: like list-corrupt-shares, but mutable shares
1566 that were successfully repaired are not
1567 included. These are shares that need
1568 manual processing. Since immutable shares
1569 cannot be modified by clients, all corruption
1570 in immutable shares will be listed here.
1571 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1572 each file that was not fully healthy. 'pathname' is
1573 relative to the directory on which deep-check was
1574 invoked. The 'check-results' field is the same as
1575 that returned by t=check&repair=true&output=JSON,
1577 stats: a dictionary with the same keys as the t=start-deep-stats command
1580 ``POST $URL?t=stream-deep-check&repair=true``
1582 This triggers a recursive walk of all files and directories, performing a
1583 t=check&repair=true on each one. For each unique object (duplicates are
1584 skipped), a single line of JSON is emitted to the HTTP response channel (or
1585 an error indication). When the walk is complete, a final line of JSON is
1586 emitted which contains the accumulated file-size/count "deep-stats" data.
1588 This emits the same data as t=stream-deep-check (without the repair=true),
1589 except that the "check-results" field is replaced with a
1590 "check-and-repair-results" field, which contains the keys returned by
1591 t=check&repair=true&output=json (i.e. repair-attempted, repair-successful,
1592 pre-repair-results, and post-repair-results). The output does not contain
1593 the summary dictionary that is provied by t=start-deep-check&repair=true
1594 (the one with count-objects-checked and list-unhealthy-files), since the
1595 receiving client is expected to calculate those values itself from the
1596 stream of per-object check-and-repair-results.
1598 Note that the "ERROR:" indication will only be emitted if traversal stops,
1599 which will only occur if an unrecoverable directory is encountered. If a
1600 file or directory repair fails, the traversal will continue, and the repair
1601 failure will be indicated in the JSON data (in the "repair-successful" key).
1603 ``POST $DIRURL?t=start-manifest`` (must add &ophandle=XYZ)
1605 This operation generates a "manfest" of the given directory tree, mostly
1606 for debugging. This is a table of (path, filecap/dircap), for every object
1607 reachable from the starting directory. The path will be slash-joined, and
1608 the filecap/dircap will contain a link to the object in question. This page
1609 gives immediate access to every object in the virtual filesystem subtree.
1611 This operation uses the same ophandle= mechanism as deep-check. The
1612 corresponding /operations/$HANDLE page has three different forms. The
1613 default is output=HTML.
1615 If output=text is added to the query args, the results will be a text/plain
1616 list. The first line is special: it is either "finished: yes" or "finished:
1617 no"; if the operation is not finished, you must periodically reload the
1618 page until it completes. The rest of the results are a plaintext list, with
1619 one file/dir per line, slash-separated, with the filecap/dircap separated
1622 If output=JSON is added to the queryargs, then the results will be a
1623 JSON-formatted dictionary with six keys. Note that because large directory
1624 structures can result in very large JSON results, the full results will not
1625 be available until the operation is complete (i.e. until output["finished"]
1628 finished (bool): if False then you must reload the page until True
1629 origin_si (base32 str): the storage index of the starting point
1630 manifest: list of (path, cap) tuples, where path is a list of strings.
1631 verifycaps: list of (printable) verify cap strings
1632 storage-index: list of (base32) storage index strings
1633 stats: a dictionary with the same keys as the t=start-deep-stats command
1636 ``POST $DIRURL?t=start-deep-size`` (must add &ophandle=XYZ)
1638 This operation generates a number (in bytes) containing the sum of the
1639 filesize of all directories and immutable files reachable from the given
1640 directory. This is a rough lower bound of the total space consumed by this
1641 subtree. It does not include space consumed by mutable files, nor does it
1642 take expansion or encoding overhead into account. Later versions of the
1643 code may improve this estimate upwards.
1645 The /operations/$HANDLE status output consists of two lines of text::
1650 ``POST $DIRURL?t=start-deep-stats`` (must add &ophandle=XYZ)
1652 This operation performs a recursive walk of all files and directories
1653 reachable from the given directory, and generates a collection of
1654 statistics about those objects.
1656 The result (obtained from the /operations/$OPHANDLE page) is a
1657 JSON-serialized dictionary with the following keys (note that some of these
1658 keys may be missing until 'finished' is True)::
1660 finished: (bool) True if the operation has finished, else False
1661 count-immutable-files: count of how many CHK files are in the set
1662 count-mutable-files: same, for mutable files (does not include directories)
1663 count-literal-files: same, for LIT files (data contained inside the URI)
1664 count-files: sum of the above three
1665 count-directories: count of directories
1666 count-unknown: count of unrecognized objects (perhaps from the future)
1667 size-immutable-files: total bytes for all CHK files in the set, =deep-size
1668 size-mutable-files (TODO): same, for current version of all mutable files
1669 size-literal-files: same, for LIT files
1670 size-directories: size of directories (includes size-literal-files)
1671 size-files-histogram: list of (minsize, maxsize, count) buckets,
1672 with a histogram of filesizes, 5dB/bucket,
1673 for both literal and immutable files
1674 largest-directory: number of children in the largest directory
1675 largest-immutable-file: number of bytes in the largest CHK file
1677 size-mutable-files is not implemented, because it would require extra
1678 queries to each mutable file to get their size. This may be implemented in
1681 Assuming no sharing, the basic space consumed by a single root directory is
1682 the sum of size-immutable-files, size-mutable-files, and size-directories.
1683 The actual disk space used by the shares is larger, because of the
1684 following sources of overhead::
1687 expansion due to erasure coding
1688 share management data (leases)
1689 backend (ext3) minimum block size
1691 ``POST $URL?t=stream-manifest``
1693 This operation performs a recursive walk of all files and directories
1694 reachable from the given starting point. For each such unique object
1695 (duplicates are skipped), a single line of JSON is emitted to the HTTP
1696 response channel (or an error indication, see below). When the walk is
1697 complete, a final line of JSON is emitted which contains the accumulated
1698 file-size/count "deep-stats" data.
1700 A CLI tool can split the response stream on newlines into "response units",
1701 and parse each response unit as JSON. Each such parsed unit will be a
1702 dictionary, and will contain at least the "type" key: a string, one of
1703 "file", "directory", or "stats".
1705 For all units that have a type of "file" or "directory", the dictionary will
1706 contain the following keys::
1708 "path": a list of strings, with the path that is traversed to reach the
1710 "cap": a write-cap URI for the file or directory, if available, else a
1712 "verifycap": a verify-cap URI for the file or directory
1713 "repaircap": an URI for the weakest cap that can still be used to repair
1715 "storage-index": a base32 storage index for the object
1717 Note that non-distributed files (i.e. LIT files) will have values of None
1718 for verifycap, repaircap, and storage-index, since these files can neither
1719 be verified nor repaired, and are not stored on the storage servers.
1721 The last unit in the stream will have a type of "stats", and will contain
1722 the keys described in the "start-deep-stats" operation, below.
1724 If any errors occur during the traversal (specifically if a directory is
1725 unrecoverable, such that further traversal is not possible), an error
1726 indication is written to the response body, instead of the usual line of
1727 JSON. This error indication line will begin with the string "ERROR:" (in all
1728 caps), and contain a summary of the error on the rest of the line. The
1729 remaining lines of the response body will be a python exception. The client
1730 application should look for the ERROR: and stop processing JSON as soon as
1731 it is seen. The line just before the ERROR: will describe the directory that
1732 was untraversable, since the manifest entry is emitted to the HTTP response
1733 body before the child is traversed.
1739 The portion of the web namespace that begins with "/uri" (and "/named") is
1740 dedicated to giving users (both humans and programs) access to the Tahoe
1741 virtual filesystem. The rest of the namespace provides status information
1742 about the state of the Tahoe node.
1744 ``GET /`` (the root page)
1746 This is the "Welcome Page", and contains a few distinct sections::
1748 Node information: library versions, local nodeid, services being provided.
1750 Filesystem Access Forms: create a new directory, view a file/directory by
1751 URI, upload a file (unlinked), download a file by
1754 Grid Status: introducer information, helper information, connected storage
1759 This page lists all active uploads and downloads, and contains a short list
1760 of recent upload/download operations. Each operation has a link to a page
1761 that describes file sizes, servers that were involved, and the time consumed
1762 in each phase of the operation.
1764 A GET of /status/?t=json will contain a machine-readable subset of the same
1765 data. It returns a JSON-encoded dictionary. The only key defined at this
1766 time is "active", with a value that is a list of operation dictionaries, one
1767 for each active operation. Once an operation is completed, it will no longer
1768 appear in data["active"] .
1770 Each op-dict contains a "type" key, one of "upload", "download",
1771 "mapupdate", "publish", or "retrieve" (the first two are for immutable
1772 files, while the latter three are for mutable files and directories).
1774 The "upload" op-dict will contain the following keys::
1776 type (string): "upload"
1777 storage-index-string (string): a base32-encoded storage index
1778 total-size (int): total size of the file
1779 status (string): current status of the operation
1780 progress-hash (float): 1.0 when the file has been hashed
1781 progress-ciphertext (float): 1.0 when the file has been encrypted.
1782 progress-encode-push (float): 1.0 when the file has been encoded and
1783 pushed to the storage servers. For helper
1784 uploads, the ciphertext value climbs to 1.0
1785 first, then encoding starts. For unassisted
1786 uploads, ciphertext and encode-push progress
1787 will climb at the same pace.
1789 The "download" op-dict will contain the following keys::
1791 type (string): "download"
1792 storage-index-string (string): a base32-encoded storage index
1793 total-size (int): total size of the file
1794 status (string): current status of the operation
1795 progress (float): 1.0 when the file has been fully downloaded
1797 Front-ends which want to report progress information are advised to simply
1798 average together all the progress-* indicators. A slightly more accurate
1799 value can be found by ignoring the progress-hash value (since the current
1800 implementation hashes synchronously, so clients will probably never see
1801 progress-hash!=1.0).
1803 ``GET /provisioning/``
1805 This page provides a basic tool to predict the likely storage and bandwidth
1806 requirements of a large Tahoe grid. It provides forms to input things like
1807 total number of users, number of files per user, average file size, number
1808 of servers, expansion ratio, hard drive failure rate, etc. It then provides
1809 numbers like how many disks per server will be needed, how many read
1810 operations per second should be expected, and the likely MTBF for files in
1811 the grid. This information is very preliminary, and the model upon which it
1812 is based still needs a lot of work.
1814 ``GET /helper_status/``
1816 If the node is running a helper (i.e. if [helper]enabled is set to True in
1817 tahoe.cfg), then this page will provide a list of all the helper operations
1818 currently in progress. If "?t=json" is added to the URL, it will return a
1819 JSON-formatted list of helper statistics, which can then be used to produce
1820 graphs to indicate how busy the helper is.
1822 ``GET /statistics/``
1824 This page provides "node statistics", which are collected from a variety of
1827 load_monitor: every second, the node schedules a timer for one second in
1828 the future, then measures how late the subsequent callback
1829 is. The "load_average" is this tardiness, measured in
1830 seconds, averaged over the last minute. It is an indication
1831 of a busy node, one which is doing more work than can be
1832 completed in a timely fashion. The "max_load" value is the
1833 highest value that has been seen in the last 60 seconds.
1835 cpu_monitor: every minute, the node uses time.clock() to measure how much
1836 CPU time it has used, and it uses this value to produce
1837 1min/5min/15min moving averages. These values range from 0%
1838 (0.0) to 100% (1.0), and indicate what fraction of the CPU
1839 has been used by the Tahoe node. Not all operating systems
1840 provide meaningful data to time.clock(): they may report 100%
1841 CPU usage at all times.
1843 uploader: this counts how many immutable files (and bytes) have been
1844 uploaded since the node was started
1846 downloader: this counts how many immutable files have been downloaded
1847 since the node was started
1849 publishes: this counts how many mutable files (including directories) have
1850 been modified since the node was started
1852 retrieves: this counts how many mutable files (including directories) have
1853 been read since the node was started
1855 There are other statistics that are tracked by the node. The "raw stats"
1856 section shows a formatted dump of all of them.
1858 By adding "?t=json" to the URL, the node will return a JSON-formatted
1859 dictionary of stats values, which can be used by other tools to produce
1860 graphs of node behavior. The misc/munin/ directory in the source
1861 distribution provides some tools to produce these graphs.
1863 ``GET /`` (introducer status)
1865 For Introducer nodes, the welcome page displays information about both
1866 clients and servers which are connected to the introducer. Servers make
1867 "service announcements", and these are listed in a table. Clients will
1868 subscribe to hear about service announcements, and these subscriptions are
1869 listed in a separate table. Both tables contain information about what
1870 version of Tahoe is being run by the remote node, their advertised and
1871 outbound IP addresses, their nodeid and nickname, and how long they have
1874 By adding "?t=json" to the URL, the node will return a JSON-formatted
1875 dictionary of stats values, which can be used to produce graphs of connected
1876 clients over time. This dictionary has the following keys::
1878 ["subscription_summary"] : a dictionary mapping service name (like
1879 "storage") to an integer with the number of
1880 clients that have subscribed to hear about that
1882 ["announcement_summary"] : a dictionary mapping service name to an integer
1883 with the number of servers which are announcing
1885 ["announcement_distinct_hosts"] : a dictionary mapping service name to an
1886 integer which represents the number of
1887 distinct hosts that are providing that
1888 service. If two servers have announced
1889 FURLs which use the same hostnames (but
1890 different ports and tubids), they are
1891 considered to be on the same host.
1894 Static Files in /public_html
1895 ============================
1897 The web-API server will take any request for a URL that starts with /static
1898 and serve it from a configurable directory which defaults to
1899 $BASEDIR/public_html . This is configured by setting the "[node]web.static"
1900 value in $BASEDIR/tahoe.cfg . If this is left at the default value of
1901 "public_html", then http://localhost:3456/static/subdir/foo.html will be
1902 served with the contents of the file $BASEDIR/public_html/subdir/foo.html .
1904 This can be useful to serve a javascript application which provides a
1905 prettier front-end to the rest of the Tahoe web-API.
1908 Safety and Security Issues -- Names vs. URIs
1909 ============================================
1911 Summary: use explicit file- and dir- caps whenever possible, to reduce the
1912 potential for surprises when the filesystem structure is changed.
1914 Tahoe provides a mutable filesystem, but the ways that the filesystem can
1915 change are limited. The only thing that can change is that the mapping from
1916 child names to child objects that each directory contains can be changed by
1917 adding a new child name pointing to an object, removing an existing child name,
1918 or changing an existing child name to point to a different object.
1920 Obviously if you query Tahoe for information about the filesystem and then act
1921 to change the filesystem (such as by getting a listing of the contents of a
1922 directory and then adding a file to the directory), then the filesystem might
1923 have been changed after you queried it and before you acted upon it. However,
1924 if you use the URI instead of the pathname of an object when you act upon the
1925 object, then the only change that can happen is if the object is a directory
1926 then the set of child names it has might be different. If, on the other hand,
1927 you act upon the object using its pathname, then a different object might be in
1928 that place, which can result in more kinds of surprises.
1930 For example, suppose you are writing code which recursively downloads the
1931 contents of a directory. The first thing your code does is fetch the listing
1932 of the contents of the directory. For each child that it fetched, if that
1933 child is a file then it downloads the file, and if that child is a directory
1934 then it recurses into that directory. Now, if the download and the recurse
1935 actions are performed using the child's name, then the results might be
1936 wrong, because for example a child name that pointed to a sub-directory when
1937 you listed the directory might have been changed to point to a file (in which
1938 case your attempt to recurse into it would result in an error and the file
1939 would be skipped), or a child name that pointed to a file when you listed the
1940 directory might now point to a sub-directory (in which case your attempt to
1941 download the child would result in a file containing HTML text describing the
1944 If your recursive algorithm uses the uri of the child instead of the name of
1945 the child, then those kinds of mistakes just can't happen. Note that both the
1946 child's name and the child's URI are included in the results of listing the
1947 parent directory, so it isn't any harder to use the URI for this purpose.
1949 The read and write caps in a given directory node are separate URIs, and
1950 can't be assumed to point to the same object even if they were retrieved in
1951 the same operation (although the web-API server attempts to ensure this
1952 in most cases). If you need to rely on that property, you should explicitly
1953 verify it. More generally, you should not make assumptions about the
1954 internal consistency of the contents of mutable directories. As a result
1955 of the signatures on mutable object versions, it is guaranteed that a given
1956 version was written in a single update, but -- as in the case of a file --
1957 the contents may have been chosen by a malicious writer in a way that is
1958 designed to confuse applications that rely on their consistency.
1960 In general, use names if you want "whatever object (whether file or
1961 directory) is found by following this name (or sequence of names) when my
1962 request reaches the server". Use URIs if you want "this particular object".
1968 Tahoe uses both mutable and immutable files. Mutable files can be created
1969 explicitly by doing an upload with ?mutable=true added, or implicitly by
1970 creating a new directory (since a directory is just a special way to
1971 interpret a given mutable file).
1973 Mutable files suffer from the same consistency-vs-availability tradeoff that
1974 all distributed data storage systems face. It is not possible to
1975 simultaneously achieve perfect consistency and perfect availability in the
1976 face of network partitions (servers being unreachable or faulty).
1978 Tahoe tries to achieve a reasonable compromise, but there is a basic rule in
1979 place, known as the Prime Coordination Directive: "Don't Do That". What this
1980 means is that if write-access to a mutable file is available to several
1981 parties, then those parties are responsible for coordinating their activities
1982 to avoid multiple simultaneous updates. This could be achieved by having
1983 these parties talk to each other and using some sort of locking mechanism, or
1984 by serializing all changes through a single writer.
1986 The consequences of performing uncoordinated writes can vary. Some of the
1987 writers may lose their changes, as somebody else wins the race condition. In
1988 many cases the file will be left in an "unhealthy" state, meaning that there
1989 are not as many redundant shares as we would like (reducing the reliability
1990 of the file against server failures). In the worst case, the file can be left
1991 in such an unhealthy state that no version is recoverable, even the old ones.
1992 It is this small possibility of data loss that prompts us to issue the Prime
1993 Coordination Directive.
1995 Tahoe nodes implement internal serialization to make sure that a single Tahoe
1996 node cannot conflict with itself. For example, it is safe to issue two
1997 directory modification requests to a single tahoe node's web-API server at the
1998 same time, because the Tahoe node will internally delay one of them until
1999 after the other has finished being applied. (This feature was introduced in
2000 Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing
2001 web requests themselves).
2003 For more details, please see the "Consistency vs Availability" and "The Prime
2004 Coordination Directive" sections of `mutable.rst <../specifications/mutable.rst>`_.
2010 Gateway nodes may find it necessary to prohibit access to certain files. The
2011 web-API has a facility to block access to filecaps by their storage index,
2012 returning a 403 "Forbidden" error instead of the original file.
2014 This blacklist is recorded in $NODEDIR/access.blacklist, and contains one
2015 blocked file per line. Comment lines (starting with ``#``) are ignored. Each
2016 line consists of the storage-index (in the usual base32 format as displayed
2017 by the "More Info" page, or by the "tahoe debug dump-cap" command), followed
2018 by whitespace, followed by a reason string, which will be included in the 403
2019 error message. This could hold a URL to a page that explains why the file is
2020 blocked, for example.
2022 So for example, if you found a need to block access to a file with filecap
2023 ``URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861``,
2024 you could do the following::
2026 tahoe debug dump-cap URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861
2027 -> storage index: whpepioyrnff7orecjolvbudeu
2028 echo "whpepioyrnff7orecjolvbudeu my puppy told me to" >>$NODEDIR/access.blacklist
2029 tahoe restart $NODEDIR
2030 tahoe get URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861
2031 -> error, 403 Access Prohibited: my puppy told me to
2033 The ``access.blacklist`` file will be checked each time a file or directory
2034 is accessed: the file's ``mtime`` is used to decide whether it need to be
2035 reloaded. Therefore no node restart is necessary when creating the initial
2036 blacklist, nor when adding second, third, or additional entries to the list.
2037 When modifying the file, be careful to update it atomically, otherwise a
2038 request may arrive while the file is only halfway written, and the partial
2039 file may be incorrectly parsed.
2041 The blacklist is applied to all access paths (including FTP, SFTP, and CLI
2042 operations), not just the web-API. The blacklist also applies to directories.
2043 If a directory is blacklisted, the gateway will refuse access to both that
2044 directory and any child files/directories underneath it, when accessed via
2045 "DIRCAP/SUBDIR/FILENAME" -style URLs. Users who go directly to the child
2046 file/dir will bypass the blacklist.
2048 The node will log the SI of the file being blocked, and the reason code, into
2049 the ``logs/twistd.log`` file.
2052 .. [1] URLs and HTTP and UTF-8, Oh My
2054 HTTP does not provide a mechanism to specify the character set used to
2055 encode non-ASCII names in URLs
2056 (`RFC3986#2.1 <http://tools.ietf.org/html/rfc3986#section-2.1>`_).
2057 We prefer the convention that the ``filename=`` argument shall be a
2058 URL-escaped UTF-8 encoded Unicode string.
2059 For example, suppose we want to provoke the server into using a filename of
2060 "f i a n c e-acute e" (i.e. f i a n c U+00E9 e). The UTF-8 encoding of this
2061 is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\\xC3\\xA9e", as python's
2062 ``repr()`` function would show). To encode this into a URL, the non-printable
2063 characters must be escaped with the urlencode ``%XX`` mechanism, giving
2064 us "fianc%C3%A9e". Thus, the first line of the HTTP request will be
2065 "``GET /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1``". Not all
2066 browsers provide this: IE7 by default uses the Latin-1 encoding, which is
2067 "fianc%E9e" (although it has a configuration option to send URLs as UTF-8).
2069 The response header will need to indicate a non-ASCII filename. The actual
2070 mechanism to do this is not clear. For ASCII filenames, the response header
2073 Content-Disposition: attachment; filename="english.txt"
2075 If Tahoe were to enforce the UTF-8 convention, it would need to decode the
2076 URL argument into a Unicode string, and then encode it back into a sequence
2077 of bytes when creating the response header. One possibility would be to use
2078 unencoded UTF-8. Developers suggest that IE7 might accept this::
2080 #1: Content-Disposition: attachment; filename="fianc\xC3\xA9e"
2081 (note, the last four bytes of that line, not including the newline, are
2082 0xC3 0xA9 0x65 0x22)
2084 `RFC2231#4 <http://tools.ietf.org/html/rfc2231#section-4>`_
2085 (dated 1997): suggests that the following might work, and
2086 `some developers have reported <http://markmail.org/message/dsjyokgl7hv64ig3>`_
2087 that it is supported by Firefox (but not IE7)::
2089 #2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
2091 My reading of `RFC2616#19.5.1 <http://tools.ietf.org/html/rfc2616#section-19.5.1>`_
2092 (which defines Content-Disposition) says that the filename= parameter is
2093 defined to be wrapped in quotes (presumably to allow spaces without breaking
2094 the parsing of subsequent parameters), which would give us::
2096 #3: Content-Disposition: attachment; filename*=utf-8''"fianc%C3%A9e"
2098 However this is contrary to the examples in the email thread listed above.
2100 Developers report that IE7 (when it is configured for UTF-8 URL encoding,
2101 which is not the default in Asian countries), will accept::
2103 #4: Content-Disposition: attachment; filename=fianc%C3%A9e
2105 However, for maximum compatibility, Tahoe simply copies bytes from the URL
2106 into the response header, rather than enforcing the UTF-8 convention. This
2107 means it does not try to decode the filename from the URL argument, nor does
2108 it encode the filename into the response header.