1 .. -*- coding: utf-8-with-signature -*-
3 ==========================
4 The Tahoe REST-ful Web API
5 ==========================
7 1. `Enabling the web-API port`_
8 2. `Basic Concepts: GET, PUT, DELETE, POST`_
13 4. `Slow Operations, Progress, and Cancelling`_
14 5. `Programmatic Operations`_
17 2. `Writing/Uploading a File`_
18 3. `Creating a New Directory`_
19 4. `Getting Information About a File Or Directory (as JSON)`_
20 5. `Attaching an Existing File or Directory by its read- or write-cap`_
21 6. `Adding Multiple Files or Directories to a Parent Directory at Once`_
22 7. `Unlinking a File or Directory`_
24 6. `Browser Operations: Human-Oriented Interfaces`_
26 1. `Viewing a Directory (as HTML)`_
27 2. `Viewing/Downloading a File`_
28 3. `Getting Information About a File Or Directory (as HTML)`_
29 4. `Creating a Directory`_
30 5. `Uploading a File`_
31 6. `Attaching an Existing File Or Directory (by URI)`_
32 7. `Unlinking a Child`_
33 8. `Renaming a Child`_
34 9. `Relinking ("Moving") a Child`_
35 10. `Other Utilities`_
36 11. `Debugging and Testing Features`_
38 7. `Other Useful Pages`_
39 8. `Static Files in /public_html`_
40 9. `Safety and Security Issues -- Names vs. URIs`_
41 10. `Concurrency Issues`_
42 11. `Access Blacklist`_
45 Enabling the web-API port
46 =========================
48 Every Tahoe node is capable of running a built-in HTTP server. To enable
49 this, just write a port number into the "[node]web.port" line of your node's
50 tahoe.cfg file. For example, writing "web.port = 3456" into the "[node]"
51 section of $NODEDIR/tahoe.cfg will cause the node to run a webserver on port
54 This string is actually a Twisted "strports" specification, meaning you can
55 get more control over the interface to which the server binds by supplying
56 additional arguments. For more details, see the documentation on
57 `twisted.application.strports`_.
59 Writing "tcp:3456:interface=127.0.0.1" into the web.port line does the same
60 but binds to the loopback interface, ensuring that only the programs on the
61 local host can connect. Using "ssl:3456:privateKey=mykey.pem:certKey=cert.pem"
64 This webport can be set when the node is created by passing a --webport
65 option to the 'tahoe create-node' command. By default, the node listens on
66 port 3456, on the loopback (127.0.0.1) interface.
68 .. _twisted.application.strports: https://twistedmatrix.com/documents/current/api/twisted.application.strports.html
71 Basic Concepts: GET, PUT, DELETE, POST
72 ======================================
74 As described in `docs/architecture.rst`_, each file and directory in a Tahoe
75 virtual filesystem is referenced by an identifier that combines the
76 designation of the object with the authority to do something with it (such as
77 read or modify the contents). This identifier is called a "read-cap" or
78 "write-cap", depending upon whether it enables read-only or read-write
79 access. These "caps" are also referred to as URIs (which may be confusing
80 because they are not currently RFC3986_-compliant URIs).
82 The Tahoe web-based API is "REST-ful", meaning it implements the concepts of
83 "REpresentational State Transfer": the original scheme by which the World
84 Wide Web was intended to work. Each object (file or directory) is referenced
85 by a URL that includes the read- or write- cap. HTTP methods (GET, PUT, and
86 DELETE) are used to manipulate these objects. You can think of the URL as a
87 noun, and the method as a verb.
89 In REST, the GET method is used to retrieve information about an object, or
90 to retrieve some representation of the object itself. When the object is a
91 file, the basic GET method will simply return the contents of that file.
92 Other variations (generally implemented by adding query parameters to the
93 URL) will return information about the object, such as metadata. GET
94 operations are required to have no side-effects.
96 PUT is used to upload new objects into the filesystem, or to replace an
97 existing link or the contents of a mutable file. DELETE is used to unlink
98 objects from directories. Both PUT and DELETE are required to be idempotent:
99 performing the same operation multiple times must have the same side-effects
100 as only performing it once.
102 POST is used for more complicated actions that cannot be expressed as a GET,
103 PUT, or DELETE. POST operations can be thought of as a method call: sending
104 some message to the object referenced by the URL. In Tahoe, POST is also used
105 for operations that must be triggered by an HTML form (including upload and
106 unlinking), because otherwise a regular web browser has no way to accomplish
107 these tasks. In general, everything that can be done with a PUT or DELETE can
108 also be done with a POST.
110 Tahoe's web API is designed for two different kinds of consumer. The first is
111 a program that needs to manipulate the virtual file system. Such programs are
112 expected to use the RESTful interface described above. The second is a human
113 using a standard web browser to work with the filesystem. This user is given
114 a series of HTML pages with links to download files, and forms that use POST
115 actions to upload, rename, and unlink files.
117 When an error occurs, the HTTP response code will be set to an appropriate
118 400-series code (like 404 Not Found for an unknown childname, or 400 Bad Request
119 when the parameters to a web-API operation are invalid), and the HTTP response
120 body will usually contain a few lines of explanation as to the cause of the
121 error and possible responses. Unusual exceptions may result in a 500 Internal
122 Server Error as a catch-all, with a default response body containing
123 a Nevow-generated HTML-ized representation of the Python exception stack trace
124 that caused the problem. CLI programs which want to copy the response body to
125 stderr should provide an "Accept: text/plain" header to their requests to get
126 a plain text stack trace instead. If the Accept header contains ``*/*``, or
127 ``text/*``, or text/html (or if there is no Accept header), HTML tracebacks will
130 .. _RFC3986: https://tools.ietf.org/html/rfc3986
131 .. _docs/architecture.rst: ../architecture.rst
137 Tahoe uses a variety of read- and write- caps to identify files and
138 directories. The most common of these is the "immutable file read-cap", which
139 is used for most uploaded files. These read-caps look like the following::
141 URI:CHK:ime6pvkaxuetdfah2p2f35pe54:4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a:3:10:202
143 The next most common is a "directory write-cap", which provides both read and
144 write access to a directory, and look like this::
146 URI:DIR2:djrdkfawoqihigoett4g6auz6a:jx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq
148 There are also "directory read-caps", which start with "URI:DIR2-RO:", and
149 give read-only access to a directory. Finally there are also mutable file
150 read- and write- caps, which start with "URI:SSK", and give access to mutable
153 (Later versions of Tahoe will make these strings shorter, and will remove the
154 unfortunate colons, which must be escaped when these caps are embedded in
157 To refer to any Tahoe object through the web API, you simply need to combine
158 a prefix (which indicates the HTTP server to use) with the cap (which
159 indicates which object inside that server to access). Since the default Tahoe
160 webport is 3456, the most common prefix is one that will use a local node
161 listening on this port::
163 http://127.0.0.1:3456/uri/ + $CAP
165 So, to access the directory named above, the URL would be::
167 http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/
169 (note that the colons in the directory-cap are url-encoded into "%3A"
172 Likewise, to access the file named above, use::
174 http://127.0.0.1:3456/uri/URI%3ACHK%3Aime6pvkaxuetdfah2p2f35pe54%3A4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a%3A3%3A10%3A202
176 In the rest of this document, we'll use "$DIRCAP" as shorthand for a read-cap
177 or write-cap that refers to a directory, and "$FILECAP" to abbreviate a cap
178 that refers to a file (whether mutable or immutable). So those URLs above can
181 http://127.0.0.1:3456/uri/$DIRCAP/
182 http://127.0.0.1:3456/uri/$FILECAP
184 The operation summaries below will abbreviate these further, by eliding the
185 server prefix. They will be displayed like this::
190 /cap can be used as a synonym for /uri. If interoperability with older web-API
191 servers is required, /uri should be used.
196 Tahoe directories contain named child entries, just like directories in a regular
197 local filesystem. These child entries, called "dirnodes", consist of a name,
198 metadata, a write slot, and a read slot. The write and read slots normally contain
199 a write-cap and read-cap referring to the same object, which can be either a file
200 or a subdirectory. The write slot may be empty (actually, both may be empty,
201 but that is unusual).
203 If you have a Tahoe URL that refers to a directory, and want to reference a
204 named child inside it, just append the child name to the URL. For example, if
205 our sample directory contains a file named "welcome.txt", we can refer to
208 http://127.0.0.1:3456/uri/$DIRCAP/welcome.txt
210 (or http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/welcome.txt)
212 Multiple levels of subdirectories can be handled this way::
214 http://127.0.0.1:3456/uri/$DIRCAP/tahoe-source/docs/architecture.rst
216 In this document, when we need to refer to a URL that references a file using
217 this child-of-some-directory format, we'll use the following string::
219 /uri/$DIRCAP/[SUBDIRS../]FILENAME
221 The "[SUBDIRS../]" part means that there are zero or more (optional)
222 subdirectory names in the middle of the URL. The "FILENAME" at the end means
223 that this whole URL refers to a file of some sort, rather than to a
226 When we need to refer specifically to a directory in this way, we'll write::
228 /uri/$DIRCAP/[SUBDIRS../]SUBDIR
231 Note that all components of pathnames in URLs are required to be UTF-8
232 encoded, so "resume.doc" (with an acute accent on both E's) would be accessed
235 http://127.0.0.1:3456/uri/$DIRCAP/r%C3%A9sum%C3%A9.doc
237 Also note that the filenames inside upload POST forms are interpreted using
238 whatever character set was provided in the conventional '_charset' field, and
239 defaults to UTF-8 if not otherwise specified. The JSON representation of each
240 directory contains native Unicode strings. Tahoe directories are specified to
241 contain Unicode filenames, and cannot contain binary strings that are not
242 representable as such.
244 All Tahoe operations that refer to existing files or directories must include
245 a suitable read- or write- cap in the URL: the web-API server won't add one
246 for you. If you don't know the cap, you can't access the file. This allows
247 the security properties of Tahoe caps to be extended across the web-API
251 Slow Operations, Progress, and Cancelling
252 =========================================
254 Certain operations can be expected to take a long time. The "t=deep-check",
255 described below, will recursively visit every file and directory reachable
256 from a given starting point, which can take minutes or even hours for
257 extremely large directory structures. A single long-running HTTP request is a
258 fragile thing: proxies, NAT boxes, browsers, and users may all grow impatient
259 with waiting and give up on the connection.
261 For this reason, long-running operations have an "operation handle", which
262 can be used to poll for status/progress messages while the operation
263 proceeds. This handle can also be used to cancel the operation. These handles
264 are created by the client, and passed in as a an "ophandle=" query argument
265 to the POST or PUT request which starts the operation. The following
266 operations can then be used to retrieve status:
268 ``GET /operations/$HANDLE?output=HTML (with or without t=status)``
270 ``GET /operations/$HANDLE?output=JSON (same)``
272 These two retrieve the current status of the given operation. Each operation
273 presents a different sort of information, but in general the page retrieved
276 * whether the operation is complete, or if it is still running
277 * how much of the operation is complete, and how much is left, if possible
279 Note that the final status output can be quite large: a deep-manifest of a
280 directory structure with 300k directories and 200k unique files is about
281 275MB of JSON, and might take two minutes to generate. For this reason, the
282 full status is not provided until the operation has completed.
284 The HTML form will include a meta-refresh tag, which will cause a regular
285 web browser to reload the status page about 60 seconds later. This tag will
286 be removed once the operation has completed.
288 There may be more status information available under
289 /operations/$HANDLE/$ETC : i.e., the handle forms the root of a URL space.
291 ``POST /operations/$HANDLE?t=cancel``
293 This terminates the operation, and returns an HTML page explaining what was
294 cancelled. If the operation handle has already expired (see below), this
295 POST will return a 404, which indicates that the operation is no longer
296 running (either it was completed or terminated). The response body will be
297 the same as a GET /operations/$HANDLE on this operation handle, and the
298 handle will be expired immediately afterwards.
300 The operation handle will eventually expire, to avoid consuming an unbounded
301 amount of memory. The handle's time-to-live can be reset at any time, by
302 passing a retain-for= argument (with a count of seconds) to either the
303 initial POST that starts the operation, or the subsequent GET request which
304 asks about the operation. For example, if a 'GET
305 /operations/$HANDLE?output=JSON&retain-for=600' query is performed, the
306 handle will remain active for 600 seconds (10 minutes) after the GET was
309 In addition, if the GET includes a release-after-complete=True argument, and
310 the operation has completed, the operation handle will be released
313 If a retain-for= argument is not used, the default handle lifetimes are:
315 * handles will remain valid at least until their operation finishes
316 * uncollected handles for finished operations (i.e. handles for
317 operations that have finished but for which the GET page has not been
318 accessed since completion) will remain valid for four days, or for
319 the total time consumed by the operation, whichever is greater.
320 * collected handles (i.e. the GET page has been retrieved at least once
321 since the operation completed) will remain valid for one day.
323 Many "slow" operations can begin to use unacceptable amounts of memory when
324 operating on large directory structures. The memory usage increases when the
325 ophandle is polled, as the results must be copied into a JSON string, sent
326 over the wire, then parsed by a client. So, as an alternative, many "slow"
327 operations have streaming equivalents. These equivalents do not use operation
328 handles. Instead, they emit line-oriented status results immediately. Client
329 code can cancel the operation by simply closing the HTTP connection.
332 Programmatic Operations
333 =======================
335 Now that we know how to build URLs that refer to files and directories in a
336 Tahoe virtual filesystem, what sorts of operations can we do with those URLs?
337 This section contains a catalog of GET, PUT, DELETE, and POST operations that
338 can be performed on these URLs. This set of operations are aimed at programs
339 that use HTTP to communicate with a Tahoe node. A later section describes
340 operations that are intended for web browsers.
346 ``GET /uri/$FILECAP``
348 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME``
350 This will retrieve the contents of the given file. The HTTP response body
351 will contain the sequence of bytes that make up the file.
353 The "Range:" header can be used to restrict which portions of the file are
354 returned (see RFC 2616 section 14.35.1 "Byte Ranges"), however Tahoe only
355 supports a single "bytes" range and never provides a `multipart/byteranges`
356 response. An attempt to begin a read past the end of the file will provoke a
357 416 Requested Range Not Satisfiable error, but normal overruns (reads which
358 start at the beginning or middle and go beyond the end) are simply
361 To view files in a web browser, you may want more control over the
362 Content-Type and Content-Disposition headers. Please see the next section
363 "Browser Operations", for details on how to modify these URLs for that
367 Writing/Uploading a File
368 ------------------------
370 ``PUT /uri/$FILECAP``
372 ``PUT /uri/$DIRCAP/[SUBDIRS../]FILENAME``
374 Upload a file, using the data from the HTTP request body, and add whatever
375 child links and subdirectories are necessary to make the file available at
376 the given location. Once this operation succeeds, a GET on the same URL will
377 retrieve the same contents that were just uploaded. This will create any
378 necessary intermediate subdirectories.
380 To use the /uri/$FILECAP form, $FILECAP must be a write-cap for a mutable file.
382 In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a
383 writeable mutable file, that file's contents will be overwritten
384 in-place. If it is a read-cap for a mutable file, an error will occur.
385 If it is an immutable file, the old file will be discarded, and a new
386 one will be put in its place. If the target file is a writable mutable
387 file, you may also specify an "offset" parameter -- a byte offset that
388 determines where in the mutable file the data from the HTTP request
389 body is placed. This operation is relatively efficient for MDMF mutable
390 files, and is relatively inefficient (but still supported) for SDMF
391 mutable files. If no offset parameter is specified, then the entire
392 file is replaced with the data from the HTTP request body. For an
393 immutable file, the "offset" parameter is not valid.
395 When creating a new file, you can control the type of file created by
396 specifying a format= argument in the query string. format=MDMF creates an
397 MDMF mutable file. format=SDMF creates an SDMF mutable file. format=CHK
398 creates an immutable file. The value of the format argument is
399 case-insensitive. If no format is specified, the newly-created file will be
400 immutable (but see below).
402 For compatibility with previous versions of Tahoe-LAFS, the web-API will
403 also accept a mutable=true argument in the query string. If mutable=true is
404 given, then the new file will be mutable, and its format will be the default
405 mutable file format, as configured by the [client]mutable.format option of
406 tahoe.cfg on the Tahoe-LAFS node hosting the webapi server. Use of
407 mutable=true is discouraged; new code should use format= instead of
408 mutable=true (unless it needs to be compatible with web-API servers older
409 than v1.9.0). If neither format= nor mutable=true are given, the
410 newly-created file will be immutable.
412 This returns the file-cap of the resulting file. If a new file was created
413 by this method, the HTTP response code (as dictated by rfc2616) will be set
414 to 201 CREATED. If an existing file was replaced or modified, the response
417 Note that the 'curl -T localfile http://127.0.0.1:3456/uri/$DIRCAP/foo.txt'
418 command can be used to invoke this operation.
422 This uploads a file, and produces a file-cap for the contents, but does not
423 attach the file into the filesystem. No directories will be modified by
424 this operation. The file-cap is returned as the body of the HTTP response.
426 This method accepts format= and mutable=true as query string arguments, and
427 interprets those arguments in the same way as the linked forms of PUT
428 described immediately above.
430 Creating a New Directory
431 ------------------------
433 ``POST /uri?t=mkdir``
437 Create a new empty directory and return its write-cap as the HTTP response
438 body. This does not make the newly created directory visible from the
439 filesystem. The "PUT" operation is provided for backwards compatibility:
440 new code should use POST.
442 This supports a format= argument in the query string. The format=
443 argument, if specified, controls the format of the directory. format=MDMF
444 indicates that the directory should be stored as an MDMF file; format=SDMF
445 indicates that the directory should be stored as an SDMF file. The value of
446 the format= argument is case-insensitive. If no format= argument is
447 given, the directory's format is determined by the default mutable file
448 format, as configured on the Tahoe-LAFS node responding to the request.
450 ``POST /uri?t=mkdir-with-children``
452 Create a new directory, populated with a set of child nodes, and return its
453 write-cap as the HTTP response body. The new directory is not attached to
454 any other directory: the returned write-cap is the only reference to it.
456 The format of the directory can be controlled with the format= argument in
457 the query string, as described above.
459 Initial children are provided as the body of the POST form (this is more
460 efficient than doing separate mkdir and set_children operations). If the
461 body is empty, the new directory will be empty. If not empty, the body will
462 be interpreted as a UTF-8 JSON-encoded dictionary of children with which the
463 new directory should be populated, using the same format as would be
464 returned in the 'children' value of the t=json GET request, described below.
465 Each dictionary key should be a child name, and each value should be a list
466 of [TYPE, PROPDICT], where PROPDICT contains "rw_uri", "ro_uri", and
467 "metadata" keys (all others are ignored). For example, the PUT request body
471 "Fran\u00e7ais": [ "filenode", {
472 "ro_uri": "URI:CHK:...",
474 "ctime": 1202777696.7564139,
475 "mtime": 1202777696.7564139,
477 "linkcrtime": 1202777696.7564139,
478 "linkmotime": 1202777696.7564139
480 "subdir": [ "dirnode", {
481 "rw_uri": "URI:DIR2:...",
482 "ro_uri": "URI:DIR2-RO:...",
484 "ctime": 1202778102.7589991,
485 "mtime": 1202778111.2160511,
487 "linkcrtime": 1202777696.7564139,
488 "linkmotime": 1202777696.7564139
492 For forward-compatibility, a mutable directory can also contain caps in
493 a format that is unknown to the web-API server. When such caps are retrieved
494 from a mutable directory in a "ro_uri" field, they will be prefixed with
495 the string "ro.", indicating that they must not be decoded without
496 checking that they are read-only. The "ro." prefix must not be stripped
497 off without performing this check. (Future versions of the web-API server
498 will perform it where necessary.)
500 If both the "rw_uri" and "ro_uri" fields are present in a given PROPDICT,
501 and the web-API server recognizes the rw_uri as a write cap, then it will
502 reset the ro_uri to the corresponding read cap and discard the original
503 contents of ro_uri (in order to ensure that the two caps correspond to the
504 same object and that the ro_uri is in fact read-only). However this may not
505 happen for caps in a format unknown to the web-API server. Therefore, when
506 writing a directory the web-API client should ensure that the contents
507 of "rw_uri" and "ro_uri" for a given PROPDICT are a consistent
508 (write cap, read cap) pair if possible. If the web-API client only has
509 one cap and does not know whether it is a write cap or read cap, then
510 it is acceptable to set "rw_uri" to that cap and omit "ro_uri". The
511 client must not put a write cap into a "ro_uri" field.
513 The metadata may have a "no-write" field. If this is set to true in the
514 metadata of a link, it will not be possible to open that link for writing
515 via the SFTP frontend; see FTP-and-SFTP.rst_ for details. Also, if the
516 "no-write" field is set to true in the metadata of a link to a mutable
517 child, it will cause the link to be diminished to read-only.
519 Note that the web-API-using client application must not provide the
520 "Content-Type: multipart/form-data" header that usually accompanies HTML
521 form submissions, since the body is not formatted this way. Doing so will
522 cause a server error as the lower-level code misparses the request body.
524 Child file names should each be expressed as a Unicode string, then used as
525 keys of the dictionary. The dictionary should then be converted into JSON,
526 and the resulting string encoded into UTF-8. This UTF-8 bytestring should
527 then be used as the POST body.
529 ``POST /uri?t=mkdir-immutable``
531 Like t=mkdir-with-children above, but the new directory will be
532 deep-immutable. This means that the directory itself is immutable, and that
533 it can only contain objects that are treated as being deep-immutable, like
534 immutable files, literal files, and deep-immutable directories.
536 For forward-compatibility, a deep-immutable directory can also contain caps
537 in a format that is unknown to the web-API server. When such caps are retrieved
538 from a deep-immutable directory in a "ro_uri" field, they will be prefixed
539 with the string "imm.", indicating that they must not be decoded without
540 checking that they are immutable. The "imm." prefix must not be stripped
541 off without performing this check. (Future versions of the web-API server
542 will perform it where necessary.)
544 The cap for each child may be given either in the "rw_uri" or "ro_uri"
545 field of the PROPDICT (not both). If a cap is given in the "rw_uri" field,
546 then the web-API server will check that it is an immutable read-cap of a
547 *known* format, and give an error if it is not. If a cap is given in the
548 "ro_uri" field, then the web-API server will still check whether known
549 caps are immutable, but for unknown caps it will simply assume that the
550 cap can be stored, as described above. Note that an attacker would be
551 able to store any cap in an immutable directory, so this check when
552 creating the directory is only to help non-malicious clients to avoid
553 accidentally giving away more authority than intended.
555 A non-empty request body is mandatory, since after the directory is created,
556 it will not be possible to add more children to it.
558 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir``
560 ``PUT /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir``
562 Create new directories as necessary to make sure that the named target
563 ($DIRCAP/SUBDIRS../SUBDIR) is a directory. This will create additional
564 intermediate mutable directories as necessary. If the named target directory
565 already exists, this will make no changes to it.
567 If the final directory is created, it will be empty.
569 This accepts a format= argument in the query string, which controls the
570 format of the named target directory, if it does not already exist. format=
571 is interpreted in the same way as in the POST /uri?t=mkdir form. Note that
572 format= only controls the format of the named target directory;
573 intermediate directories, if created, are created based on the default
574 mutable type, as configured on the Tahoe-LAFS server responding to the
577 This operation will return an error if a blocking file is present at any of
578 the parent names, preventing the server from creating the necessary parent
579 directory; or if it would require changing an immutable directory.
581 The write-cap of the new directory will be returned as the HTTP response
584 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-with-children``
586 Like /uri?t=mkdir-with-children, but the final directory is created as a
587 child of an existing mutable directory. This will create additional
588 intermediate mutable directories as necessary. If the final directory is
589 created, it will be populated with initial children from the POST request
590 body, as described above.
592 This accepts a format= argument in the query string, which controls the
593 format of the target directory, if the target directory is created as part
594 of the operation. format= is interpreted in the same way as in the POST/
595 uri?t=mkdir-with-children operation. Note that format= only controls the
596 format of the named target directory; intermediate directories, if created,
597 are created using the default mutable type setting, as configured on the
598 Tahoe-LAFS server responding to the request.
600 This operation will return an error if a blocking file is present at any of
601 the parent names, preventing the server from creating the necessary parent
602 directory; or if it would require changing an immutable directory; or if
603 the immediate parent directory already has a a child named SUBDIR.
605 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-immutable``
607 Like /uri?t=mkdir-immutable, but the final directory is created as a child
608 of an existing mutable directory. The final directory will be deep-immutable,
609 and will be populated with the children specified as a JSON dictionary in
610 the POST request body.
612 In Tahoe 1.6 this operation creates intermediate mutable directories if
613 necessary, but that behaviour should not be relied on; see ticket #920.
615 This operation will return an error if the parent directory is immutable,
616 or already has a child named SUBDIR.
618 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME``
620 Create a new empty mutable directory and attach it to the given existing
621 directory. This will create additional intermediate directories as necessary.
623 This accepts a format= argument in the query string, which controls the
624 format of the named target directory, if it does not already exist. format=
625 is interpreted in the same way as in the POST /uri?t=mkdir form. Note that
626 format= only controls the format of the named target directory;
627 intermediate directories, if created, are created based on the default
628 mutable type, as configured on the Tahoe-LAFS server responding to the
631 This operation will return an error if a blocking file is present at any of
632 the parent names, preventing the server from creating the necessary parent
633 directory, or if it would require changing any immutable directory.
635 The URL of this operation points to the parent of the bottommost new directory,
636 whereas the /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir operation above has a URL
637 that points directly to the bottommost new directory.
639 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME``
641 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME, but the new directory will
642 be populated with initial children via the POST request body. This command
643 will create additional intermediate mutable directories as necessary.
645 This accepts a format= argument in the query string, which controls the
646 format of the target directory, if the target directory is created as part
647 of the operation. format= is interpreted in the same way as in the POST/
648 uri?t=mkdir-with-children operation. Note that format= only controls the
649 format of the named target directory; intermediate directories, if created,
650 are created using the default mutable type setting, as configured on the
651 Tahoe-LAFS server responding to the request.
653 This operation will return an error if a blocking file is present at any of
654 the parent names, preventing the server from creating the necessary parent
655 directory; or if it would require changing an immutable directory; or if
656 the immediate parent directory already has a a child named NAME.
658 Note that the name= argument must be passed as a queryarg, because the POST
659 request body is used for the initial children JSON.
661 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-immutable&name=NAME``
663 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME, but the
664 final directory will be deep-immutable. The children are specified as a
665 JSON dictionary in the POST request body. Again, the name= argument must be
666 passed as a queryarg.
668 In Tahoe 1.6 this operation creates intermediate mutable directories if
669 necessary, but that behaviour should not be relied on; see ticket #920.
671 This operation will return an error if the parent directory is immutable,
672 or already has a child named NAME.
674 .. _FTP-and-SFTP.rst: FTP-and-SFTP.rst
677 Getting Information About a File Or Directory (as JSON)
678 -------------------------------------------------------
680 ``GET /uri/$FILECAP?t=json``
682 ``GET /uri/$DIRCAP?t=json``
684 ``GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json``
686 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json``
688 This returns a machine-parseable JSON-encoded description of the given
689 object. The JSON always contains a list, and the first element of the list is
690 always a flag that indicates whether the referenced object is a file or a
691 directory. If it is a capability to a file, then the information includes
692 file size and URI, like this::
694 GET /uri/$FILECAP?t=json :
698 "verify_uri": verify_uri,
704 If it is a capability to a directory followed by a path from that directory
705 to a file, then the information also includes metadata from the link to the
706 file in the parent directory, like this::
708 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json
712 "verify_uri": verify_uri,
717 "ctime": 1202777696.7564139,
718 "mtime": 1202777696.7564139,
720 "linkcrtime": 1202777696.7564139,
721 "linkmotime": 1202777696.7564139
724 If it is a directory, then it includes information about the children of
725 this directory, as a mapping from child name to a set of data about the
726 child (the same data that would appear in a corresponding GET?t=json of the
727 child itself). The child entries also include metadata about each child,
728 including link-creation- and link-change- timestamps. The output looks like
731 GET /uri/$DIRCAP?t=json :
732 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json :
735 "rw_uri": read_write_uri,
736 "ro_uri": read_only_uri,
737 "verify_uri": verify_uri,
741 "foo.txt": [ "filenode",
746 "ctime": 1202777696.7564139,
747 "mtime": 1202777696.7564139,
749 "linkcrtime": 1202777696.7564139,
750 "linkmotime": 1202777696.7564139
752 "subdir": [ "dirnode",
757 "ctime": 1202778102.7589991,
758 "mtime": 1202778111.2160511,
760 "linkcrtime": 1202777696.7564139,
761 "linkmotime": 1202777696.7564139
765 In the above example, note how 'children' is a dictionary in which the keys
766 are child names and the values depend upon whether the child is a file or a
767 directory. The value is mostly the same as the JSON representation of the
768 child object (except that directories do not recurse -- the "children"
769 entry of the child is omitted, and the directory view includes the metadata
770 that is stored on the directory edge).
772 The rw_uri field will be present in the information about a directory
773 if and only if you have read-write access to that directory. The verify_uri
774 field will be present if and only if the object has a verify-cap
775 (non-distributed LIT files do not have verify-caps).
777 If the cap is of an unknown format, then the file size and verify_uri will
780 GET /uri/$UNKNOWNCAP?t=json :
783 "ro_uri": unknown_read_uri
786 GET /uri/$DIRCAP/[SUBDIRS../]UNKNOWNCHILDNAME?t=json :
789 "rw_uri": unknown_write_uri,
790 "ro_uri": unknown_read_uri,
793 "ctime": 1202777696.7564139,
794 "mtime": 1202777696.7564139,
796 "linkcrtime": 1202777696.7564139,
797 "linkmotime": 1202777696.7564139
800 As in the case of file nodes, the metadata will only be present when the
801 capability is to a directory followed by a path. The "mutable" field is also
802 not always present; when it is absent, the mutability of the object is not
808 The value of the 'tahoe':'linkmotime' key is updated whenever a link to a
809 child is set. The value of the 'tahoe':'linkcrtime' key is updated whenever
810 a link to a child is created -- i.e. when there was not previously a link
813 Note however, that if the edge in the Tahoe filesystem points to a mutable
814 file and the contents of that mutable file is changed, then the
815 'tahoe':'linkmotime' value on that edge will *not* be updated, since the
816 edge itself wasn't updated -- only the mutable file was.
818 The timestamps are represented as a number of seconds since the UNIX epoch
819 (1970-01-01 00:00:00 UTC), with leap seconds not being counted in the long
822 In Tahoe earlier than v1.4.0, 'mtime' and 'ctime' keys were populated
823 instead of the 'tahoe':'linkmotime' and 'tahoe':'linkcrtime' keys. Starting
824 in Tahoe v1.4.0, the 'linkmotime'/'linkcrtime' keys in the 'tahoe' sub-dict
825 are populated. However, prior to Tahoe v1.7beta, a bug caused the 'tahoe'
826 sub-dict to be deleted by web-API requests in which new metadata is
827 specified, and not to be added to existing child links that lack it.
829 From Tahoe v1.7.0 onward, the 'mtime' and 'ctime' fields are no longer
830 populated or updated (see ticket #924), except by "tahoe backup" as
831 explained below. For backward compatibility, when an existing link is
832 updated and 'tahoe':'linkcrtime' is not present in the previous metadata
833 but 'ctime' is, the old value of 'ctime' is used as the new value of
834 'tahoe':'linkcrtime'.
836 The reason we added the new fields in Tahoe v1.4.0 is that there is a
837 "set_children" API (described below) which you can use to overwrite the
838 values of the 'mtime'/'ctime' pair, and this API is used by the
839 "tahoe backup" command (in Tahoe v1.3.0 and later) to set the 'mtime' and
840 'ctime' values when backing up files from a local filesystem into the
841 Tahoe filesystem. As of Tahoe v1.4.0, the set_children API cannot be used
842 to set anything under the 'tahoe' key of the metadata dict -- if you
843 include 'tahoe' keys in your 'metadata' arguments then it will silently
846 Therefore, if the 'tahoe' sub-dict is present, you can rely on the
847 'linkcrtime' and 'linkmotime' values therein to have the semantics described
848 above. (This is assuming that only official Tahoe clients have been used to
849 write those links, and that their system clocks were set to what you expected
850 -- there is nothing preventing someone from editing their Tahoe client or
851 writing their own Tahoe client which would overwrite those values however
852 they like, and there is nothing to constrain their system clock from taking
855 When an edge is created or updated by "tahoe backup", the 'mtime' and
856 'ctime' keys on that edge are set as follows:
858 * 'mtime' is set to the timestamp read from the local filesystem for the
859 "mtime" of the local file in question, which means the last time the
860 contents of that file were changed.
862 * On Windows, 'ctime' is set to the creation timestamp for the file
863 read from the local filesystem. On other platforms, 'ctime' is set to
864 the UNIX "ctime" of the local file, which means the last time that
865 either the contents or the metadata of the local file was changed.
867 There are several ways that the 'ctime' field could be confusing:
869 1. You might be confused about whether it reflects the time of the creation
870 of a link in the Tahoe filesystem (by a version of Tahoe < v1.7.0) or a
871 timestamp copied in by "tahoe backup" from a local filesystem.
873 2. You might be confused about whether it is a copy of the file creation
874 time (if "tahoe backup" was run on a Windows system) or of the last
875 contents-or-metadata change (if "tahoe backup" was run on a different
878 3. You might be confused by the fact that changing the contents of a
879 mutable file in Tahoe doesn't have any effect on any links pointing at
880 that file in any directories, although "tahoe backup" sets the link
881 'ctime'/'mtime' to reflect timestamps about the local file corresponding
882 to the Tahoe file to which the link points.
884 4. Also, quite apart from Tahoe, you might be confused about the meaning
885 of the "ctime" in UNIX local filesystems, which people sometimes think
886 means file creation time, but which actually means, in UNIX local
887 filesystems, the most recent time that the file contents or the file
888 metadata (such as owner, permission bits, extended attributes, etc.)
889 has changed. Note that although "ctime" does not mean file creation time
890 in UNIX, links created by a version of Tahoe prior to v1.7.0, and never
891 written by "tahoe backup", will have 'ctime' set to the link creation
895 Attaching an Existing File or Directory by its read- or write-cap
896 -----------------------------------------------------------------
898 ``PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri``
900 This attaches a child object (either a file or directory) to a specified
901 location in the virtual filesystem. The child object is referenced by its
902 read- or write- cap, as provided in the HTTP request body. This will create
903 intermediate directories as necessary.
905 This is similar to a UNIX hardlink: by referencing a previously-uploaded file
906 (or previously-created directory) instead of uploading/creating a new one,
907 you can create two references to the same object.
909 The read- or write- cap of the child is provided in the body of the HTTP
910 request, and this same cap is returned in the response body.
912 The default behavior is to overwrite any existing object at the same
913 location. To prevent this (and make the operation return an error instead
914 of overwriting), add a "replace=false" argument, as "?t=uri&replace=false".
915 With replace=false, this operation will return an HTTP 409 "Conflict" error
916 if there is already an object at the given location, rather than
917 overwriting the existing object. To allow the operation to overwrite a
918 file, but return an error when trying to overwrite a directory, use
919 "replace=only-files" (this behavior is closer to the traditional UNIX "mv"
920 command). Note that "true", "t", and "1" are all synonyms for "True", and
921 "false", "f", and "0" are synonyms for "False", and the parameter is
924 Note that this operation does not take its child cap in the form of
925 separate "rw_uri" and "ro_uri" fields. Therefore, it cannot accept a
926 child cap in a format unknown to the web-API server, unless its URI
927 starts with "ro." or "imm.". This restriction is necessary because the
928 server is not able to attenuate an unknown write cap to a read cap.
929 Unknown URIs starting with "ro." or "imm.", on the other hand, are
930 assumed to represent read caps. The client should not prefix a write
931 cap with "ro." or "imm." and pass it to this operation, since that
932 would result in granting the cap's write authority to holders of the
936 Adding Multiple Files or Directories to a Parent Directory at Once
937 ------------------------------------------------------------------
939 ``POST /uri/$DIRCAP/[SUBDIRS..]?t=set_children``
941 ``POST /uri/$DIRCAP/[SUBDIRS..]?t=set-children`` (Tahoe >= v1.6)
943 This command adds multiple children to a directory in a single operation.
944 It reads the request body and interprets it as a JSON-encoded description
945 of the child names and read/write-caps that should be added.
947 The body should be a JSON-encoded dictionary, in the same format as the
948 "children" value returned by the "GET /uri/$DIRCAP?t=json" operation
949 described above. In this format, each key is a child names, and the
950 corresponding value is a tuple of (type, childinfo). "type" is ignored, and
951 "childinfo" is a dictionary that contains "rw_uri", "ro_uri", and
952 "metadata" keys. You can take the output of "GET /uri/$DIRCAP1?t=json" and
953 use it as the input to "POST /uri/$DIRCAP2?t=set_children" to make DIR2
954 look very much like DIR1 (except for any existing children of DIR2 that
955 were not overwritten, and any existing "tahoe" metadata keys as described
958 When the set_children request contains a child name that already exists in
959 the target directory, this command defaults to overwriting that child with
960 the new value (both child cap and metadata, but if the JSON data does not
961 contain a "metadata" key, the old child's metadata is preserved). The
962 command takes a boolean "overwrite=" query argument to control this
963 behavior. If you use "?t=set_children&overwrite=false", then an attempt to
964 replace an existing child will instead cause an error.
966 Any "tahoe" key in the new child's "metadata" value is ignored. Any
967 existing "tahoe" metadata is preserved. The metadata["tahoe"] value is
968 reserved for metadata generated by the tahoe node itself. The only two keys
969 currently placed here are "linkcrtime" and "linkmotime". For details, see
970 the section above entitled "Getting Information About a File Or Directory (as
971 JSON)", in the "About the metadata" subsection.
973 Note that this command was introduced with the name "set_children", which
974 uses an underscore rather than a hyphen as other multi-word command names
975 do. The variant with a hyphen is now accepted, but clients that desire
976 backward compatibility should continue to use "set_children".
979 Unlinking a File or Directory
980 -----------------------------
982 ``DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME``
984 This removes the given name from its parent directory. CHILDNAME is the
985 name to be removed, and $DIRCAP/SUBDIRS.. indicates the directory that will
988 Note that this does not actually delete the file or directory that the name
989 points to from the tahoe grid -- it only unlinks the named reference from
990 this directory. If there are other names in this directory or in other
991 directories that point to the resource, then it will remain accessible
992 through those paths. Even if all names pointing to this object are removed
993 from their parent directories, then someone with possession of its read-cap
994 can continue to access the object through that cap.
996 The object will only become completely unreachable once 1: there are no
997 reachable directories that reference it, and 2: nobody is holding a read-
998 or write- cap to the object. (This behavior is very similar to the way
999 hardlinks and anonymous files work in traditional UNIX filesystems).
1001 This operation will not modify more than a single directory. Intermediate
1002 directories which were implicitly created by PUT or POST methods will *not*
1003 be automatically removed by DELETE.
1005 This method returns the file- or directory- cap of the object that was just
1009 Browser Operations: Human-oriented interfaces
1010 =============================================
1012 This section describes the HTTP operations that provide support for humans
1013 running a web browser. Most of these operations use HTML forms that use POST
1014 to drive the Tahoe node. This section is intended for HTML authors who want
1015 to write web pages that contain forms and buttons which manipulate the Tahoe
1018 Note that for all POST operations, the arguments listed can be provided
1019 either as URL query arguments or as form body fields. URL query arguments are
1020 separated from the main URL by "?", and from each other by "&". For example,
1021 "POST /uri/$DIRCAP?t=upload&mutable=true". Form body fields are usually
1022 specified by using <input type="hidden"> elements. For clarity, the
1023 descriptions below display the most significant arguments as URL query args.
1026 Viewing a Directory (as HTML)
1027 -----------------------------
1029 ``GET /uri/$DIRCAP/[SUBDIRS../]``
1031 This returns an HTML page, intended to be displayed to a human by a web
1032 browser, which contains HREF links to all files and directories reachable
1033 from this directory. These HREF links do not have a t= argument, meaning
1034 that a human who follows them will get pages also meant for a human. It also
1035 contains forms to upload new files, and to unlink files and directories
1036 from their parent directory. Those forms use POST methods to do their job.
1039 Viewing/Downloading a File
1040 --------------------------
1042 ``GET /uri/$FILECAP``
1044 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME``
1046 This will retrieve the contents of the given file. The HTTP response body
1047 will contain the sequence of bytes that make up the file.
1049 If you want the HTTP response to include a useful Content-Type header,
1050 either use the second form (which starts with a $DIRCAP), or add a
1051 "filename=foo" query argument, like "GET /uri/$FILECAP?filename=foo.jpg".
1052 The bare "GET /uri/$FILECAP" does not give the Tahoe node enough information
1053 to determine a Content-Type (since Tahoe immutable files are merely
1054 sequences of bytes, not typed+named file objects).
1056 If the URL has both filename= and "save=true" in the query arguments, then
1057 the server to add a "Content-Disposition: attachment" header, along with a
1058 filename= parameter. When a user clicks on such a link, most browsers will
1059 offer to let the user save the file instead of displaying it inline (indeed,
1060 most browsers will refuse to display it inline). "true", "t", "1", and other
1061 case-insensitive equivalents are all treated the same.
1063 Character-set handling in URLs and HTTP headers is a dubious art [1]_. For
1064 maximum compatibility, Tahoe simply copies the bytes from the filename=
1065 argument into the Content-Disposition header's filename= parameter, without
1066 trying to interpret them in any particular way.
1069 ``GET /named/$FILECAP/FILENAME``
1071 This is an alternate download form which makes it easier to get the correct
1072 filename. The Tahoe server will provide the contents of the given file, with
1073 a Content-Type header derived from the given filename. This form is used to
1074 get browsers to use the "Save Link As" feature correctly, and also helps
1075 command-line tools like "wget" and "curl" use the right filename. Note that
1076 this form can *only* be used with file caps; it is an error to use a
1077 directory cap after the /named/ prefix.
1079 URLs may also use /file/$FILECAP/FILENAME as a synonym for
1080 /named/$FILECAP/FILENAME.
1082 Getting Information About a File Or Directory (as HTML)
1083 -------------------------------------------------------
1085 ``GET /uri/$FILECAP?t=info``
1087 ``GET /uri/$DIRCAP/?t=info``
1089 ``GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR/?t=info``
1091 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=info``
1093 This returns a human-oriented HTML page with more detail about the selected
1094 file or directory object. This page contains the following items:
1098 * JSON representation
1099 * raw contents (text/plain)
1100 * access caps (URIs): verify-cap, read-cap, write-cap (for mutable objects)
1101 * check/verify/repair form
1102 * deep-check/deep-size/deep-stats/manifest (for directories)
1103 * replace-contents form (for mutable files)
1106 Creating a Directory
1107 --------------------
1109 ``POST /uri?t=mkdir``
1111 This creates a new empty directory, but does not attach it to the virtual
1114 If a "redirect_to_result=true" argument is provided, then the HTTP response
1115 will cause the web browser to be redirected to a /uri/$DIRCAP page that
1116 gives access to the newly-created directory. If you bookmark this page,
1117 you'll be able to get back to the directory again in the future. This is the
1118 recommended way to start working with a Tahoe server: create a new unlinked
1119 directory (using redirect_to_result=true), then bookmark the resulting
1120 /uri/$DIRCAP page. There is a "create directory" button on the Welcome page
1121 to invoke this action.
1123 This accepts a format= argument in the query string. Refer to the
1124 documentation of the PUT /uri?t=mkdir operation in `Creating A
1125 New Directory`_ for information on the behavior of the format= argument.
1127 If "redirect_to_result=true" is not provided (or is given a value of
1128 "false"), then the HTTP response body will simply be the write-cap of the
1131 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME``
1133 This creates a new empty directory as a child of the designated SUBDIR. This
1134 will create additional intermediate directories as necessary.
1136 This accepts a format= argument in the query string. Refer to the
1137 documentation of POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME in
1138 `Creating a New Directory`_ for information on the behavior of the format=
1141 If a "when_done=URL" argument is provided, the HTTP response will cause the
1142 web browser to redirect to the given URL. This provides a convenient way to
1143 return the browser to the directory that was just modified. Without a
1144 when_done= argument, the HTTP response will simply contain the write-cap of
1145 the directory that was just created.
1151 ``POST /uri?t=upload``
1153 This uploads a file, and produces a file-cap for the contents, but does not
1154 attach the file into the filesystem. No directories will be modified by
1157 The file must be provided as the "file" field of an HTML encoded form body,
1158 produced in response to an HTML form like this::
1160 <form action="/uri" method="POST" enctype="multipart/form-data">
1161 <input type="hidden" name="t" value="upload" />
1162 <input type="file" name="file" />
1163 <input type="submit" value="Upload Unlinked" />
1166 If a "when_done=URL" argument is provided, the response body will cause the
1167 browser to redirect to the given URL. If the when_done= URL has the string
1168 "%(uri)s" in it, that string will be replaced by a URL-escaped form of the
1169 newly created file-cap. (Note that without this substitution, there is no
1170 way to access the file that was just uploaded).
1172 The default (in the absence of when_done=) is to return an HTML page that
1173 describes the results of the upload. This page will contain information
1174 about which storage servers were used for the upload, how long each
1175 operation took, etc.
1177 This accepts format= and mutable=true query string arguments. Refer to
1178 `Writing/Uploading a File`_ for information on the behavior of format= and
1181 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=upload``
1183 This uploads a file, and attaches it as a new child of the given directory,
1184 which must be mutable. The file must be provided as the "file" field of an
1185 HTML-encoded form body, produced in response to an HTML form like this::
1187 <form action="." method="POST" enctype="multipart/form-data">
1188 <input type="hidden" name="t" value="upload" />
1189 <input type="file" name="file" />
1190 <input type="submit" value="Upload" />
1193 A "name=" argument can be provided to specify the new child's name,
1194 otherwise it will be taken from the "filename" field of the upload form
1195 (most web browsers will copy the last component of the original file's
1196 pathname into this field). To avoid confusion, name= is not allowed to
1199 If there is already a child with that name, and it is a mutable file, then
1200 its contents are replaced with the data being uploaded. If it is not a
1201 mutable file, the default behavior is to remove the existing child before
1202 creating a new one. To prevent this (and make the operation return an error
1203 instead of overwriting the old child), add a "replace=false" argument, as
1204 "?t=upload&replace=false". With replace=false, this operation will return an
1205 HTTP 409 "Conflict" error if there is already an object at the given
1206 location, rather than overwriting the existing object. Note that "true",
1207 "t", and "1" are all synonyms for "True", and "false", "f", and "0" are
1208 synonyms for "False". the parameter is case-insensitive.
1210 This will create additional intermediate directories as necessary, although
1211 since it is expected to be triggered by a form that was retrieved by "GET
1212 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1215 This accepts format= and mutable=true query string arguments. Refer to
1216 `Writing/Uploading a File`_ for information on the behavior of format= and
1219 If a "when_done=URL" argument is provided, the HTTP response will cause the
1220 web browser to redirect to the given URL. This provides a convenient way to
1221 return the browser to the directory that was just modified. Without a
1222 when_done= argument, the HTTP response will simply contain the file-cap of
1223 the file that was just uploaded (a write-cap for mutable files, or a
1224 read-cap for immutable files).
1226 ``POST /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=upload``
1228 This also uploads a file and attaches it as a new child of the given
1229 directory, which must be mutable. It is a slight variant of the previous
1230 operation, as the URL refers to the target file rather than the parent
1231 directory. It is otherwise identical: this accepts mutable= and when_done=
1234 ``POST /uri/$FILECAP?t=upload``
1236 This modifies the contents of an existing mutable file in-place. An error is
1237 signalled if $FILECAP does not refer to a mutable file. It behaves just like
1238 the "PUT /uri/$FILECAP" form, but uses a POST for the benefit of HTML forms
1242 Attaching An Existing File Or Directory (by URI)
1243 ------------------------------------------------
1245 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=uri&name=CHILDNAME&uri=CHILDCAP``
1247 This attaches a given read- or write- cap "CHILDCAP" to the designated
1248 directory, with a specified child name. This behaves much like the PUT t=uri
1249 operation, and is a lot like a UNIX hardlink. It is subject to the same
1250 restrictions as that operation on the use of cap formats unknown to the
1253 This will create additional intermediate directories as necessary, although
1254 since it is expected to be triggered by a form that was retrieved by "GET
1255 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1258 This accepts the same replace= argument as POST t=upload.
1264 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME``
1266 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=unlink&name=CHILDNAME`` (Tahoe >= v1.9)
1268 This instructs the node to remove a child object (file or subdirectory) from
1269 the given directory, which must be mutable. Note that the entire subtree is
1270 unlinked from the parent. Unlike deleting a subdirectory in a UNIX local
1271 filesystem, the subtree need not be empty; if it isn't, then other references
1272 into the subtree will see that the child subdirectories are not modified by
1273 this operation. Only the link from the given directory to its child is severed.
1275 In Tahoe-LAFS v1.9.0 and later, t=unlink can be used as a synonym for t=delete.
1276 If interoperability with older web-API servers is required, t=delete should
1283 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=rename&from_name=OLD&to_name=NEW``
1285 This instructs the node to rename a child of the given directory, which must
1286 be mutable. This has a similar effect to removing the child, then adding the
1287 same child-cap under the new name, except that it preserves metadata. This
1288 operation cannot move the child to a different directory.
1290 The default behavior is to overwrite any existing link at the destination
1291 (replace=true). To prevent this (and make the operation return an error
1292 instead of overwriting), add a "replace=false" argument. With replace=false,
1293 this operation will return an HTTP 409 "Conflict" error if the destination
1294 is not the same link as the source and there is already a link at the
1295 destination, rather than overwriting the existing link. To allow the
1296 operation to overwrite a link to a file, but return an HTTP 409 error when
1297 trying to overwrite a link to a directory, use "replace=only-files" (this
1298 behavior is closer to the traditional UNIX "mv" command). Note that "true",
1299 "t", and "1" are all synonyms for "True"; "false", "f", and "0" are synonyms
1300 for "False"; and the parameter is case-insensitive.
1303 Relinking ("Moving") a Child
1304 ----------------------------
1306 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=relink&from_name=OLD&to_dir=$NEWDIRCAP/[NEWSUBDIRS../]&to_name=NEW``
1307 ``[&replace=true|false|only-files]`` (Tahoe >= v1.10)
1309 This instructs the node to move a child of the given source directory, into
1310 a different directory and/or to a different name. The command is named
1311 ``relink`` because what it does is add a new link to the child from the new
1312 location, then remove the old link. Nothing is actually "moved": the child
1313 is still reachable through any path from which it was formerly reachable,
1314 and the storage space occupied by its ciphertext is not affected.
1316 The source and destination directories must be writeable. If {{{to_dir}}} is
1317 not present, the child link is renamed within the same directory. If
1318 {{{to_name}}} is not present then it defaults to {{{from_name}}}. If the
1319 destination link (directory and name) is the same as the source link, the
1320 operation has no effect.
1322 Metadata from the source directory entry is preserved. Multiple levels of
1323 descent in the source and destination paths are supported.
1325 This operation will return an HTTP 404 "Not Found" error if
1326 ``$DIRCAP/[SUBDIRS../]``, the child being moved, or the destination
1327 directory does not exist. It will return an HTTP 400 "Bad Request" error
1328 if any entry in the source or destination paths is not a directory.
1330 The default behavior is to overwrite any existing link at the destination
1331 (replace=true). To prevent this (and make the operation return an error
1332 instead of overwriting), add a "replace=false" argument. With replace=false,
1333 this operation will return an HTTP 409 "Conflict" error if the destination
1334 is not the same link as the source and there is already a link at the
1335 destination, rather than overwriting the existing link. To allow the
1336 operation to overwrite a link to a file, but return an HTTP 409 error when
1337 trying to overwrite a link to a directory, use "replace=only-files" (this
1338 behavior is closer to the traditional UNIX "mv" command). Note that "true",
1339 "t", and "1" are all synonyms for "True"; "false", "f", and "0" are synonyms
1340 for "False"; and the parameter is case-insensitive.
1342 When relinking into a different directory, for safety, the child link is
1343 not removed from the old directory until it has been successfully added to
1344 the new directory. This implies that in case of a crash or failure, the
1345 link to the child will not be lost, but it could be linked at both the old
1348 The source link should not be the same as any link (directory and child name)
1349 in the ``to_dir`` path. This restriction is not enforced, but it may be
1350 enforced in a future version. If it were violated then the result would be
1351 to create a cycle in the directory structure that is not necessarily reachable
1352 from the root of the destination path (``$NEWDIRCAP``), which could result in
1353 data loss, as described in ticket `#943`_.
1355 .. _`#943`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/943
1361 ``GET /uri?uri=$CAP``
1363 This causes a redirect to /uri/$CAP, and retains any additional query
1364 arguments (like filename= or save=). This is for the convenience of web
1365 forms which allow the user to paste in a read- or write- cap (obtained
1366 through some out-of-band channel, like IM or email).
1368 Note that this form merely redirects to the specific file or directory
1369 indicated by the $CAP: unlike the GET /uri/$DIRCAP form, you cannot
1370 traverse to children by appending additional path segments to the URL.
1372 ``GET /uri/$DIRCAP/[SUBDIRS../]?t=rename-form&name=$CHILDNAME``
1374 This provides a useful facility to browser-based user interfaces. It
1375 returns a page containing a form targetting the "POST $DIRCAP t=rename"
1376 functionality described above, with the provided $CHILDNAME present in the
1377 'from_name' field of that form. I.e. this presents a form offering to
1378 rename $CHILDNAME, requesting the new name, and submitting POST rename.
1379 This same URL format can also be used with "move-form" with the expected
1382 ``GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri``
1384 This returns the file- or directory- cap for the specified object.
1386 ``GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=readonly-uri``
1388 This returns a read-only file- or directory- cap for the specified object.
1389 If the object is an immutable file, this will return the same value as
1393 Debugging and Testing Features
1394 ------------------------------
1396 These URLs are less-likely to be helpful to the casual Tahoe user, and are
1397 mainly intended for developers.
1399 ``POST $URL?t=check``
1401 This triggers the FileChecker to determine the current "health" of the
1402 given file or directory, by counting how many shares are available. The
1403 page that is returned will display the results. This can be used as a "show
1404 me detailed information about this file" page.
1406 If a verify=true argument is provided, the node will perform a more
1407 intensive check, downloading and verifying every single bit of every share.
1409 If an add-lease=true argument is provided, the node will also add (or
1410 renew) a lease to every share it encounters. Each lease will keep the share
1411 alive for a certain period of time (one month by default). Once the last
1412 lease expires or is explicitly cancelled, the storage server is allowed to
1415 If an output=JSON argument is provided, the response will be
1416 machine-readable JSON instead of human-oriented HTML. The data is a
1417 dictionary with the following keys::
1419 storage-index: a base32-encoded string with the objects's storage index,
1420 or an empty string for LIT files
1421 summary: a string, with a one-line summary of the stats of the file
1422 results: a dictionary that describes the state of the file. For LIT files,
1423 this dictionary has only the 'healthy' key, which will always be
1424 True. For distributed files, this dictionary has the following
1426 count-happiness: the servers-of-happiness level of the file, as
1427 defined in `docs/specifications/servers-of-happiness.rst`_.
1428 count-shares-good: the number of good shares that were found
1429 count-shares-needed: 'k', the number of shares required for recovery
1430 count-shares-expected: 'N', the number of total shares generated
1431 count-good-share-hosts: the number of distinct storage servers with
1432 good shares. Note that a high value does not
1433 necessarily imply good share distribution,
1434 because some of these servers may only hold
1436 count-wrong-shares: for mutable files, the number of shares for
1437 versions other than the 'best' one (highest
1438 sequence number, highest roothash). These are
1439 either old, or created by an uncoordinated or
1440 not fully successful write.
1441 count-recoverable-versions: for mutable files, the number of
1442 recoverable versions of the file. For
1443 a healthy file, this will equal 1.
1444 count-unrecoverable-versions: for mutable files, the number of
1445 unrecoverable versions of the file.
1446 For a healthy file, this will be 0.
1447 count-corrupt-shares: the number of shares with integrity failures
1448 list-corrupt-shares: a list of "share locators", one for each share
1449 that was found to be corrupt. Each share locator
1450 is a list of (serverid, storage_index, sharenum).
1451 servers-responding: list of base32-encoded storage server identifiers,
1452 one for each server which responded to the share
1454 healthy: (bool) True if the file is completely healthy, False otherwise.
1455 Healthy files have at least N good shares. Overlapping shares
1456 do not currently cause a file to be marked unhealthy. If there
1457 are at least N good shares, then corrupt shares do not cause the
1458 file to be marked unhealthy, although the corrupt shares will be
1459 listed in the results (list-corrupt-shares) and should be manually
1460 removed to wasting time in subsequent downloads (as the
1461 downloader rediscovers the corruption and uses alternate shares).
1462 Future compatibility: the meaning of this field may change to
1463 reflect whether the servers-of-happiness criterion is met
1465 sharemap: dict mapping share identifier to list of serverids
1466 (base32-encoded strings). This indicates which servers are
1467 holding which shares. For immutable files, the shareid is
1468 an integer (the share number, from 0 to N-1). For
1469 immutable files, it is a string of the form
1470 'seq%d-%s-sh%d', containing the sequence number, the
1471 roothash, and the share number.
1473 Before Tahoe-LAFS v1.11, the `results` dictionary also had a `needs-rebalancing`
1474 field, but that has been removed since it was computed incorrectly.
1476 .. _`docs/specifications/servers-of-happiness.rst`: ../specifications/servers-of-happiness.rst
1479 ``POST $URL?t=start-deep-check`` (must add &ophandle=XYZ)
1481 This initiates a recursive walk of all files and directories reachable from
1482 the target, performing a check on each one just like t=check. The result
1483 page will contain a summary of the results, including details on any
1484 file/directory that was not fully healthy.
1486 t=start-deep-check can only be invoked on a directory. An error (400
1487 BAD_REQUEST) will be signalled if it is invoked on a file. The recursive
1488 walker will deal with loops safely.
1490 This accepts the same verify= and add-lease= arguments as t=check.
1492 Since this operation can take a long time (perhaps a second per object),
1493 the ophandle= argument is required (see "Slow Operations, Progress, and
1494 Cancelling" above). The response to this POST will be a redirect to the
1495 corresponding /operations/$HANDLE page (with output=HTML or output=JSON to
1496 match the output= argument given to the POST). The deep-check operation
1497 will continue to run in the background, and the /operations page should be
1498 used to find out when the operation is done.
1500 Detailed check results for non-healthy files and directories will be
1501 available under /operations/$HANDLE/$STORAGEINDEX, and the HTML status will
1502 contain links to these detailed results.
1504 The HTML /operations/$HANDLE page for incomplete operations will contain a
1505 meta-refresh tag, set to 60 seconds, so that a browser which uses
1506 deep-check will automatically poll until the operation has completed.
1508 The JSON page (/options/$HANDLE?output=JSON) will contain a
1509 machine-readable JSON dictionary with the following keys::
1511 finished: a boolean, True if the operation is complete, else False. Some
1512 of the remaining keys may not be present until the operation
1514 root-storage-index: a base32-encoded string with the storage index of the
1515 starting point of the deep-check operation
1516 count-objects-checked: count of how many objects were checked. Note that
1517 non-distributed objects (i.e. small immutable LIT
1518 files) are not checked, since for these objects,
1519 the data is contained entirely in the URI.
1520 count-objects-healthy: how many of those objects were completely healthy
1521 count-objects-unhealthy: how many were damaged in some way
1522 count-corrupt-shares: how many shares were found to have corruption,
1523 summed over all objects examined
1524 list-corrupt-shares: a list of "share identifiers", one for each share
1525 that was found to be corrupt. Each share identifier
1526 is a list of (serverid, storage_index, sharenum).
1527 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1528 each file that was not fully healthy. 'pathname' is
1529 a list of strings (which can be joined by "/"
1530 characters to turn it into a single string),
1531 relative to the directory on which deep-check was
1532 invoked. The 'check-results' field is the same as
1533 that returned by t=check&output=JSON, described
1535 stats: a dictionary with the same keys as the t=start-deep-stats command
1538 ``POST $URL?t=stream-deep-check``
1540 This initiates a recursive walk of all files and directories reachable from
1541 the target, performing a check on each one just like t=check. For each
1542 unique object (duplicates are skipped), a single line of JSON is emitted to
1543 the HTTP response channel (or an error indication, see below). When the walk
1544 is complete, a final line of JSON is emitted which contains the accumulated
1545 file-size/count "deep-stats" data.
1547 This command takes the same arguments as t=start-deep-check.
1549 A CLI tool can split the response stream on newlines into "response units",
1550 and parse each response unit as JSON. Each such parsed unit will be a
1551 dictionary, and will contain at least the "type" key: a string, one of
1552 "file", "directory", or "stats".
1554 For all units that have a type of "file" or "directory", the dictionary will
1555 contain the following keys::
1557 "path": a list of strings, with the path that is traversed to reach the
1559 "cap": a write-cap URI for the file or directory, if available, else a
1561 "verifycap": a verify-cap URI for the file or directory
1562 "repaircap": an URI for the weakest cap that can still be used to repair
1564 "storage-index": a base32 storage index for the object
1565 "check-results": a copy of the dictionary which would be returned by
1566 t=check&output=json, with three top-level keys:
1567 "storage-index", "summary", and "results", and a variety
1568 of counts and sharemaps in the "results" value.
1570 Note that non-distributed files (i.e. LIT files) will have values of None
1571 for verifycap, repaircap, and storage-index, since these files can neither
1572 be verified nor repaired, and are not stored on the storage servers.
1573 Likewise the check-results dictionary will be limited: an empty string for
1574 storage-index, and a results dictionary with only the "healthy" key.
1576 The last unit in the stream will have a type of "stats", and will contain
1577 the keys described in the "start-deep-stats" operation, below.
1579 If any errors occur during the traversal (specifically if a directory is
1580 unrecoverable, such that further traversal is not possible), an error
1581 indication is written to the response body, instead of the usual line of
1582 JSON. This error indication line will begin with the string "ERROR:" (in all
1583 caps), and contain a summary of the error on the rest of the line. The
1584 remaining lines of the response body will be a python exception. The client
1585 application should look for the ERROR: and stop processing JSON as soon as
1586 it is seen. Note that neither a file being unrecoverable nor a directory
1587 merely being unhealthy will cause traversal to stop. The line just before
1588 the ERROR: will describe the directory that was untraversable, since the
1589 unit is emitted to the HTTP response body before the child is traversed.
1592 ``POST $URL?t=check&repair=true``
1594 This performs a health check of the given file or directory, and if the
1595 checker determines that the object is not healthy (some shares are missing
1596 or corrupted), it will perform a "repair". During repair, any missing
1597 shares will be regenerated and uploaded to new servers.
1599 This accepts the same verify=true and add-lease= arguments as t=check. When
1600 an output=JSON argument is provided, the machine-readable JSON response
1601 will contain the following keys::
1603 storage-index: a base32-encoded string with the objects's storage index,
1604 or an empty string for LIT files
1605 repair-attempted: (bool) True if repair was attempted
1606 repair-successful: (bool) True if repair was attempted and the file was
1607 fully healthy afterwards. False if no repair was
1608 attempted, or if a repair attempt failed.
1609 pre-repair-results: a dictionary that describes the state of the file
1610 before any repair was performed. This contains exactly
1611 the same keys as the 'results' value of the t=check
1612 response, described above.
1613 post-repair-results: a dictionary that describes the state of the file
1614 after any repair was performed. If no repair was
1615 performed, post-repair-results and pre-repair-results
1616 will be the same. This contains exactly the same keys
1617 as the 'results' value of the t=check response,
1620 ``POST $URL?t=start-deep-check&repair=true`` (must add &ophandle=XYZ)
1622 This triggers a recursive walk of all files and directories, performing a
1623 t=check&repair=true on each one.
1625 Like t=start-deep-check without the repair= argument, this can only be
1626 invoked on a directory. An error (400 BAD_REQUEST) will be signalled if it
1627 is invoked on a file. The recursive walker will deal with loops safely.
1629 This accepts the same verify= and add-lease= arguments as
1630 t=start-deep-check. It uses the same ophandle= mechanism as
1631 start-deep-check. When an output=JSON argument is provided, the response
1632 will contain the following keys::
1634 finished: (bool) True if the operation has completed, else False
1635 root-storage-index: a base32-encoded string with the storage index of the
1636 starting point of the deep-check operation
1637 count-objects-checked: count of how many objects were checked
1639 count-objects-healthy-pre-repair: how many of those objects were completely
1640 healthy, before any repair
1641 count-objects-unhealthy-pre-repair: how many were damaged in some way
1642 count-objects-healthy-post-repair: how many of those objects were completely
1643 healthy, after any repair
1644 count-objects-unhealthy-post-repair: how many were damaged in some way
1646 count-repairs-attempted: repairs were attempted on this many objects.
1647 count-repairs-successful: how many repairs resulted in healthy objects
1648 count-repairs-unsuccessful: how many repairs resulted did not results in
1649 completely healthy objects
1650 count-corrupt-shares-pre-repair: how many shares were found to have
1651 corruption, summed over all objects
1652 examined, before any repair
1653 count-corrupt-shares-post-repair: how many shares were found to have
1654 corruption, summed over all objects
1655 examined, after any repair
1656 list-corrupt-shares: a list of "share identifiers", one for each share
1657 that was found to be corrupt (before any repair).
1658 Each share identifier is a list of (serverid,
1659 storage_index, sharenum).
1660 list-remaining-corrupt-shares: like list-corrupt-shares, but mutable shares
1661 that were successfully repaired are not
1662 included. These are shares that need
1663 manual processing. Since immutable shares
1664 cannot be modified by clients, all corruption
1665 in immutable shares will be listed here.
1666 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1667 each file that was not fully healthy. 'pathname' is
1668 relative to the directory on which deep-check was
1669 invoked. The 'check-results' field is the same as
1670 that returned by t=check&repair=true&output=JSON,
1672 stats: a dictionary with the same keys as the t=start-deep-stats command
1675 ``POST $URL?t=stream-deep-check&repair=true``
1677 This triggers a recursive walk of all files and directories, performing a
1678 t=check&repair=true on each one. For each unique object (duplicates are
1679 skipped), a single line of JSON is emitted to the HTTP response channel (or
1680 an error indication). When the walk is complete, a final line of JSON is
1681 emitted which contains the accumulated file-size/count "deep-stats" data.
1683 This emits the same data as t=stream-deep-check (without the repair=true),
1684 except that the "check-results" field is replaced with a
1685 "check-and-repair-results" field, which contains the keys returned by
1686 t=check&repair=true&output=json (i.e. repair-attempted, repair-successful,
1687 pre-repair-results, and post-repair-results). The output does not contain
1688 the summary dictionary that is provied by t=start-deep-check&repair=true
1689 (the one with count-objects-checked and list-unhealthy-files), since the
1690 receiving client is expected to calculate those values itself from the
1691 stream of per-object check-and-repair-results.
1693 Note that the "ERROR:" indication will only be emitted if traversal stops,
1694 which will only occur if an unrecoverable directory is encountered. If a
1695 file or directory repair fails, the traversal will continue, and the repair
1696 failure will be indicated in the JSON data (in the "repair-successful" key).
1698 ``POST $DIRURL?t=start-manifest`` (must add &ophandle=XYZ)
1700 This operation generates a "manfest" of the given directory tree, mostly
1701 for debugging. This is a table of (path, filecap/dircap), for every object
1702 reachable from the starting directory. The path will be slash-joined, and
1703 the filecap/dircap will contain a link to the object in question. This page
1704 gives immediate access to every object in the virtual filesystem subtree.
1706 This operation uses the same ophandle= mechanism as deep-check. The
1707 corresponding /operations/$HANDLE page has three different forms. The
1708 default is output=HTML.
1710 If output=text is added to the query args, the results will be a text/plain
1711 list. The first line is special: it is either "finished: yes" or "finished:
1712 no"; if the operation is not finished, you must periodically reload the
1713 page until it completes. The rest of the results are a plaintext list, with
1714 one file/dir per line, slash-separated, with the filecap/dircap separated
1717 If output=JSON is added to the queryargs, then the results will be a
1718 JSON-formatted dictionary with six keys. Note that because large directory
1719 structures can result in very large JSON results, the full results will not
1720 be available until the operation is complete (i.e. until output["finished"]
1723 finished (bool): if False then you must reload the page until True
1724 origin_si (base32 str): the storage index of the starting point
1725 manifest: list of (path, cap) tuples, where path is a list of strings.
1726 verifycaps: list of (printable) verify cap strings
1727 storage-index: list of (base32) storage index strings
1728 stats: a dictionary with the same keys as the t=start-deep-stats command
1731 ``POST $DIRURL?t=start-deep-size`` (must add &ophandle=XYZ)
1733 This operation generates a number (in bytes) containing the sum of the
1734 filesize of all directories and immutable files reachable from the given
1735 directory. This is a rough lower bound of the total space consumed by this
1736 subtree. It does not include space consumed by mutable files, nor does it
1737 take expansion or encoding overhead into account. Later versions of the
1738 code may improve this estimate upwards.
1740 The /operations/$HANDLE status output consists of two lines of text::
1745 ``POST $DIRURL?t=start-deep-stats`` (must add &ophandle=XYZ)
1747 This operation performs a recursive walk of all files and directories
1748 reachable from the given directory, and generates a collection of
1749 statistics about those objects.
1751 The result (obtained from the /operations/$OPHANDLE page) is a
1752 JSON-serialized dictionary with the following keys (note that some of these
1753 keys may be missing until 'finished' is True)::
1755 finished: (bool) True if the operation has finished, else False
1756 count-immutable-files: count of how many CHK files are in the set
1757 count-mutable-files: same, for mutable files (does not include directories)
1758 count-literal-files: same, for LIT files (data contained inside the URI)
1759 count-files: sum of the above three
1760 count-directories: count of directories
1761 count-unknown: count of unrecognized objects (perhaps from the future)
1762 size-immutable-files: total bytes for all CHK files in the set, =deep-size
1763 size-mutable-files (TODO): same, for current version of all mutable files
1764 size-literal-files: same, for LIT files
1765 size-directories: size of directories (includes size-literal-files)
1766 size-files-histogram: list of (minsize, maxsize, count) buckets,
1767 with a histogram of filesizes, 5dB/bucket,
1768 for both literal and immutable files
1769 largest-directory: number of children in the largest directory
1770 largest-immutable-file: number of bytes in the largest CHK file
1772 size-mutable-files is not implemented, because it would require extra
1773 queries to each mutable file to get their size. This may be implemented in
1776 Assuming no sharing, the basic space consumed by a single root directory is
1777 the sum of size-immutable-files, size-mutable-files, and size-directories.
1778 The actual disk space used by the shares is larger, because of the
1779 following sources of overhead::
1782 expansion due to erasure coding
1783 share management data (leases)
1784 backend (ext3) minimum block size
1786 ``POST $URL?t=stream-manifest``
1788 This operation performs a recursive walk of all files and directories
1789 reachable from the given starting point. For each such unique object
1790 (duplicates are skipped), a single line of JSON is emitted to the HTTP
1791 response channel (or an error indication, see below). When the walk is
1792 complete, a final line of JSON is emitted which contains the accumulated
1793 file-size/count "deep-stats" data.
1795 A CLI tool can split the response stream on newlines into "response units",
1796 and parse each response unit as JSON. Each such parsed unit will be a
1797 dictionary, and will contain at least the "type" key: a string, one of
1798 "file", "directory", or "stats".
1800 For all units that have a type of "file" or "directory", the dictionary will
1801 contain the following keys::
1803 "path": a list of strings, with the path that is traversed to reach the
1805 "cap": a write-cap URI for the file or directory, if available, else a
1807 "verifycap": a verify-cap URI for the file or directory
1808 "repaircap": an URI for the weakest cap that can still be used to repair
1810 "storage-index": a base32 storage index for the object
1812 Note that non-distributed files (i.e. LIT files) will have values of None
1813 for verifycap, repaircap, and storage-index, since these files can neither
1814 be verified nor repaired, and are not stored on the storage servers.
1816 The last unit in the stream will have a type of "stats", and will contain
1817 the keys described in the "start-deep-stats" operation, below.
1819 If any errors occur during the traversal (specifically if a directory is
1820 unrecoverable, such that further traversal is not possible), an error
1821 indication is written to the response body, instead of the usual line of
1822 JSON. This error indication line will begin with the string "ERROR:" (in all
1823 caps), and contain a summary of the error on the rest of the line. The
1824 remaining lines of the response body will be a python exception. The client
1825 application should look for the ERROR: and stop processing JSON as soon as
1826 it is seen. The line just before the ERROR: will describe the directory that
1827 was untraversable, since the manifest entry is emitted to the HTTP response
1828 body before the child is traversed.
1834 The portion of the web namespace that begins with "/uri" (and "/named") is
1835 dedicated to giving users (both humans and programs) access to the Tahoe
1836 virtual filesystem. The rest of the namespace provides status information
1837 about the state of the Tahoe node.
1839 ``GET /`` (the root page)
1841 This is the "Welcome Page", and contains a few distinct sections::
1843 Node information: library versions, local nodeid, services being provided.
1845 Filesystem Access Forms: create a new directory, view a file/directory by
1846 URI, upload a file (unlinked), download a file by
1849 Grid Status: introducer information, helper information, connected storage
1854 This page lists all active uploads and downloads, and contains a short list
1855 of recent upload/download operations. Each operation has a link to a page
1856 that describes file sizes, servers that were involved, and the time consumed
1857 in each phase of the operation.
1859 A GET of /status/?t=json will contain a machine-readable subset of the same
1860 data. It returns a JSON-encoded dictionary. The only key defined at this
1861 time is "active", with a value that is a list of operation dictionaries, one
1862 for each active operation. Once an operation is completed, it will no longer
1863 appear in data["active"] .
1865 Each op-dict contains a "type" key, one of "upload", "download",
1866 "mapupdate", "publish", or "retrieve" (the first two are for immutable
1867 files, while the latter three are for mutable files and directories).
1869 The "upload" op-dict will contain the following keys::
1871 type (string): "upload"
1872 storage-index-string (string): a base32-encoded storage index
1873 total-size (int): total size of the file
1874 status (string): current status of the operation
1875 progress-hash (float): 1.0 when the file has been hashed
1876 progress-ciphertext (float): 1.0 when the file has been encrypted.
1877 progress-encode-push (float): 1.0 when the file has been encoded and
1878 pushed to the storage servers. For helper
1879 uploads, the ciphertext value climbs to 1.0
1880 first, then encoding starts. For unassisted
1881 uploads, ciphertext and encode-push progress
1882 will climb at the same pace.
1884 The "download" op-dict will contain the following keys::
1886 type (string): "download"
1887 storage-index-string (string): a base32-encoded storage index
1888 total-size (int): total size of the file
1889 status (string): current status of the operation
1890 progress (float): 1.0 when the file has been fully downloaded
1892 Front-ends which want to report progress information are advised to simply
1893 average together all the progress-* indicators. A slightly more accurate
1894 value can be found by ignoring the progress-hash value (since the current
1895 implementation hashes synchronously, so clients will probably never see
1896 progress-hash!=1.0).
1898 ``GET /helper_status/``
1900 If the node is running a helper (i.e. if [helper]enabled is set to True in
1901 tahoe.cfg), then this page will provide a list of all the helper operations
1902 currently in progress. If "?t=json" is added to the URL, it will return a
1903 JSON-formatted list of helper statistics, which can then be used to produce
1904 graphs to indicate how busy the helper is.
1906 ``GET /statistics/``
1908 This page provides "node statistics", which are collected from a variety of
1911 load_monitor: every second, the node schedules a timer for one second in
1912 the future, then measures how late the subsequent callback
1913 is. The "load_average" is this tardiness, measured in
1914 seconds, averaged over the last minute. It is an indication
1915 of a busy node, one which is doing more work than can be
1916 completed in a timely fashion. The "max_load" value is the
1917 highest value that has been seen in the last 60 seconds.
1919 cpu_monitor: every minute, the node uses time.clock() to measure how much
1920 CPU time it has used, and it uses this value to produce
1921 1min/5min/15min moving averages. These values range from 0%
1922 (0.0) to 100% (1.0), and indicate what fraction of the CPU
1923 has been used by the Tahoe node. Not all operating systems
1924 provide meaningful data to time.clock(): they may report 100%
1925 CPU usage at all times.
1927 uploader: this counts how many immutable files (and bytes) have been
1928 uploaded since the node was started
1930 downloader: this counts how many immutable files have been downloaded
1931 since the node was started
1933 publishes: this counts how many mutable files (including directories) have
1934 been modified since the node was started
1936 retrieves: this counts how many mutable files (including directories) have
1937 been read since the node was started
1939 There are other statistics that are tracked by the node. The "raw stats"
1940 section shows a formatted dump of all of them.
1942 By adding "?t=json" to the URL, the node will return a JSON-formatted
1943 dictionary of stats values, which can be used by other tools to produce
1944 graphs of node behavior. The misc/munin/ directory in the source
1945 distribution provides some tools to produce these graphs.
1947 ``GET /`` (introducer status)
1949 For Introducer nodes, the welcome page displays information about both
1950 clients and servers which are connected to the introducer. Servers make
1951 "service announcements", and these are listed in a table. Clients will
1952 subscribe to hear about service announcements, and these subscriptions are
1953 listed in a separate table. Both tables contain information about what
1954 version of Tahoe is being run by the remote node, their advertised and
1955 outbound IP addresses, their nodeid and nickname, and how long they have
1958 By adding "?t=json" to the URL, the node will return a JSON-formatted
1959 dictionary of stats values, which can be used to produce graphs of connected
1960 clients over time. This dictionary has the following keys::
1962 ["subscription_summary"] : a dictionary mapping service name (like
1963 "storage") to an integer with the number of
1964 clients that have subscribed to hear about that
1966 ["announcement_summary"] : a dictionary mapping service name to an integer
1967 with the number of servers which are announcing
1969 ["announcement_distinct_hosts"] : a dictionary mapping service name to an
1970 integer which represents the number of
1971 distinct hosts that are providing that
1972 service. If two servers have announced
1973 FURLs which use the same hostnames (but
1974 different ports and tubids), they are
1975 considered to be on the same host.
1978 Static Files in /public_html
1979 ============================
1981 The web-API server will take any request for a URL that starts with /static
1982 and serve it from a configurable directory which defaults to
1983 $BASEDIR/public_html . This is configured by setting the "[node]web.static"
1984 value in $BASEDIR/tahoe.cfg . If this is left at the default value of
1985 "public_html", then http://127.0.0.1:3456/static/subdir/foo.html will be
1986 served with the contents of the file $BASEDIR/public_html/subdir/foo.html .
1988 This can be useful to serve a javascript application which provides a
1989 prettier front-end to the rest of the Tahoe web-API.
1992 Safety and Security Issues -- Names vs. URIs
1993 ============================================
1995 Summary: use explicit file- and dir- caps whenever possible, to reduce the
1996 potential for surprises when the filesystem structure is changed.
1998 Tahoe provides a mutable filesystem, but the ways that the filesystem can
1999 change are limited. The only thing that can change is that the mapping from
2000 child names to child objects that each directory contains can be changed by
2001 adding a new child name pointing to an object, removing an existing child name,
2002 or changing an existing child name to point to a different object.
2004 Obviously if you query Tahoe for information about the filesystem and then act
2005 to change the filesystem (such as by getting a listing of the contents of a
2006 directory and then adding a file to the directory), then the filesystem might
2007 have been changed after you queried it and before you acted upon it. However,
2008 if you use the URI instead of the pathname of an object when you act upon the
2009 object, then the only change that can happen is if the object is a directory
2010 then the set of child names it has might be different. If, on the other hand,
2011 you act upon the object using its pathname, then a different object might be in
2012 that place, which can result in more kinds of surprises.
2014 For example, suppose you are writing code which recursively downloads the
2015 contents of a directory. The first thing your code does is fetch the listing
2016 of the contents of the directory. For each child that it fetched, if that
2017 child is a file then it downloads the file, and if that child is a directory
2018 then it recurses into that directory. Now, if the download and the recurse
2019 actions are performed using the child's name, then the results might be
2020 wrong, because for example a child name that pointed to a sub-directory when
2021 you listed the directory might have been changed to point to a file (in which
2022 case your attempt to recurse into it would result in an error and the file
2023 would be skipped), or a child name that pointed to a file when you listed the
2024 directory might now point to a sub-directory (in which case your attempt to
2025 download the child would result in a file containing HTML text describing the
2028 If your recursive algorithm uses the uri of the child instead of the name of
2029 the child, then those kinds of mistakes just can't happen. Note that both the
2030 child's name and the child's URI are included in the results of listing the
2031 parent directory, so it isn't any harder to use the URI for this purpose.
2033 The read and write caps in a given directory node are separate URIs, and
2034 can't be assumed to point to the same object even if they were retrieved in
2035 the same operation (although the web-API server attempts to ensure this
2036 in most cases). If you need to rely on that property, you should explicitly
2037 verify it. More generally, you should not make assumptions about the
2038 internal consistency of the contents of mutable directories. As a result
2039 of the signatures on mutable object versions, it is guaranteed that a given
2040 version was written in a single update, but -- as in the case of a file --
2041 the contents may have been chosen by a malicious writer in a way that is
2042 designed to confuse applications that rely on their consistency.
2044 In general, use names if you want "whatever object (whether file or
2045 directory) is found by following this name (or sequence of names) when my
2046 request reaches the server". Use URIs if you want "this particular object".
2052 Tahoe uses both mutable and immutable files. Mutable files can be created
2053 explicitly by doing an upload with ?mutable=true added, or implicitly by
2054 creating a new directory (since a directory is just a special way to
2055 interpret a given mutable file).
2057 Mutable files suffer from the same consistency-vs-availability tradeoff that
2058 all distributed data storage systems face. It is not possible to
2059 simultaneously achieve perfect consistency and perfect availability in the
2060 face of network partitions (servers being unreachable or faulty).
2062 Tahoe tries to achieve a reasonable compromise, but there is a basic rule in
2063 place, known as the Prime Coordination Directive: "Don't Do That". What this
2064 means is that if write-access to a mutable file is available to several
2065 parties, then those parties are responsible for coordinating their activities
2066 to avoid multiple simultaneous updates. This could be achieved by having
2067 these parties talk to each other and using some sort of locking mechanism, or
2068 by serializing all changes through a single writer.
2070 The consequences of performing uncoordinated writes can vary. Some of the
2071 writers may lose their changes, as somebody else wins the race condition. In
2072 many cases the file will be left in an "unhealthy" state, meaning that there
2073 are not as many redundant shares as we would like (reducing the reliability
2074 of the file against server failures). In the worst case, the file can be left
2075 in such an unhealthy state that no version is recoverable, even the old ones.
2076 It is this small possibility of data loss that prompts us to issue the Prime
2077 Coordination Directive.
2079 Tahoe nodes implement internal serialization to make sure that a single Tahoe
2080 node cannot conflict with itself. For example, it is safe to issue two
2081 directory modification requests to a single tahoe node's web-API server at the
2082 same time, because the Tahoe node will internally delay one of them until
2083 after the other has finished being applied. (This feature was introduced in
2084 Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing
2085 web requests themselves).
2087 For more details, please see the "Consistency vs Availability" and "The Prime
2088 Coordination Directive" sections of mutable.rst_.
2090 .. _mutable.rst: ../specifications/mutable.rst
2096 Gateway nodes may find it necessary to prohibit access to certain files. The
2097 web-API has a facility to block access to filecaps by their storage index,
2098 returning a 403 "Forbidden" error instead of the original file.
2100 This blacklist is recorded in $NODEDIR/access.blacklist, and contains one
2101 blocked file per line. Comment lines (starting with ``#``) are ignored. Each
2102 line consists of the storage-index (in the usual base32 format as displayed
2103 by the "More Info" page, or by the "tahoe debug dump-cap" command), followed
2104 by whitespace, followed by a reason string, which will be included in the 403
2105 error message. This could hold a URL to a page that explains why the file is
2106 blocked, for example.
2108 So for example, if you found a need to block access to a file with filecap
2109 ``URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861``,
2110 you could do the following::
2112 tahoe debug dump-cap URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861
2113 -> storage index: whpepioyrnff7orecjolvbudeu
2114 echo "whpepioyrnff7orecjolvbudeu my puppy told me to" >>$NODEDIR/access.blacklist
2115 tahoe restart $NODEDIR
2116 tahoe get URI:CHK:n7r3m6wmomelk4sep3kw5cvduq:os7ijw5c3maek7pg65e5254k2fzjflavtpejjyhshpsxuqzhcwwq:3:20:14861
2117 -> error, 403 Access Prohibited: my puppy told me to
2119 The ``access.blacklist`` file will be checked each time a file or directory
2120 is accessed: the file's ``mtime`` is used to decide whether it need to be
2121 reloaded. Therefore no node restart is necessary when creating the initial
2122 blacklist, nor when adding second, third, or additional entries to the list.
2123 When modifying the file, be careful to update it atomically, otherwise a
2124 request may arrive while the file is only halfway written, and the partial
2125 file may be incorrectly parsed.
2127 The blacklist is applied to all access paths (including SFTP, FTP, and CLI
2128 operations), not just the web-API. The blacklist also applies to directories.
2129 If a directory is blacklisted, the gateway will refuse access to both that
2130 directory and any child files/directories underneath it, when accessed via
2131 "DIRCAP/SUBDIR/FILENAME" -style URLs. Users who go directly to the child
2132 file/dir will bypass the blacklist.
2134 The node will log the SI of the file being blocked, and the reason code, into
2135 the ``logs/twistd.log`` file.
2138 .. [1] URLs and HTTP and UTF-8, Oh My
2140 HTTP does not provide a mechanism to specify the character set used to
2141 encode non-ASCII names in URLs (`RFC3986#2.1`_). We prefer the convention
2142 that the ``filename=`` argument shall be a URL-escaped UTF-8 encoded Unicode
2143 string. For example, suppose we want to provoke the server into using a
2144 filename of "f i a n c e-acute e" (i.e. f i a n c U+00E9 e). The UTF-8
2145 encoding of this is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or
2146 "fianc\\xC3\\xA9e", as python's ``repr()`` function would show). To encode
2147 this into a URL, the non-printable characters must be escaped with the
2148 urlencode ``%XX`` mechanism, giving us "fianc%C3%A9e". Thus, the first line
2149 of the HTTP request will be "``GET
2150 /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1``". Not all browsers
2151 provide this: IE7 by default uses the Latin-1 encoding, which is "fianc%E9e"
2152 (although it has a configuration option to send URLs as UTF-8).
2154 The response header will need to indicate a non-ASCII filename. The actual
2155 mechanism to do this is not clear. For ASCII filenames, the response header
2158 Content-Disposition: attachment; filename="english.txt"
2160 If Tahoe were to enforce the UTF-8 convention, it would need to decode the
2161 URL argument into a Unicode string, and then encode it back into a sequence
2162 of bytes when creating the response header. One possibility would be to use
2163 unencoded UTF-8. Developers suggest that IE7 might accept this::
2165 #1: Content-Disposition: attachment; filename="fianc\xC3\xA9e"
2166 (note, the last four bytes of that line, not including the newline, are
2167 0xC3 0xA9 0x65 0x22)
2169 `RFC2231#4`_ (dated 1997): suggests that the following might work, and `some
2170 developers have reported`_ that it is supported by Firefox (but not IE7)::
2172 #2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
2174 My reading of `RFC2616#19.5.1`_ (which defines Content-Disposition) says
2175 that the filename= parameter is defined to be wrapped in quotes (presumably
2176 to allow spaces without breaking the parsing of subsequent parameters),
2177 which would give us::
2179 #3: Content-Disposition: attachment; filename*=utf-8''"fianc%C3%A9e"
2181 However this is contrary to the examples in the email thread listed above.
2183 Developers report that IE7 (when it is configured for UTF-8 URL encoding,
2184 which is not the default in Asian countries), will accept::
2186 #4: Content-Disposition: attachment; filename=fianc%C3%A9e
2188 However, for maximum compatibility, Tahoe simply copies bytes from the URL
2189 into the response header, rather than enforcing the UTF-8 convention. This
2190 means it does not try to decode the filename from the URL argument, nor does
2191 it encode the filename into the response header.
2193 .. _RFC3986#2.1: https://tools.ietf.org/html/rfc3986#section-2.1
2194 .. _RFC2231#4: https://tools.ietf.org/html/rfc2231#section-4
2195 .. _some developers have reported: http://markmail.org/message/dsjyokgl7hv64ig3
2196 .. _RFC2616#19.5.1: https://tools.ietf.org/html/rfc2616#section-19.5.1