1 ==========================
2 The Tahoe REST-ful Web API
3 ==========================
5 1. `Enabling the web-API port`_
6 2. `Basic Concepts: GET, PUT, DELETE, POST`_
11 4. `Slow Operations, Progress, and Cancelling`_
12 5. `Programmatic Operations`_
15 2. `Writing/Uploading a File`_
16 3. `Creating a New Directory`_
17 4. `Get Information About A File Or Directory (as JSON)`_
18 5. `Attaching an existing File or Directory by its read- or write-cap`_
19 6. `Adding multiple files or directories to a parent directory at once`_
20 7. `Deleting a File or Directory`_
22 6. `Browser Operations: Human-Oriented Interfaces`_
24 1. `Viewing A Directory (as HTML)`_
25 2. `Viewing/Downloading a File`_
26 3. `Get Information About A File Or Directory (as HTML)`_
27 4. `Creating a Directory`_
28 5. `Uploading a File`_
29 6. `Attaching An Existing File Or Directory (by URI)`_
30 7. `Deleting A Child`_
31 8. `Renaming A Child`_
33 10. `Debugging and Testing Features`_
35 7. `Other Useful Pages`_
36 8. `Static Files in /public_html`_
37 9. `Safety and security issues -- names vs. URIs`_
38 10. `Concurrency Issues`_
40 Enabling the web-API port
41 =========================
43 Every Tahoe node is capable of running a built-in HTTP server. To enable
44 this, just write a port number into the "[node]web.port" line of your node's
45 tahoe.cfg file. For example, writing "web.port = 3456" into the "[node]"
46 section of $NODEDIR/tahoe.cfg will cause the node to run a webserver on port
49 This string is actually a Twisted "strports" specification, meaning you can
50 get more control over the interface to which the server binds by supplying
51 additional arguments. For more details, see the documentation on
52 `twisted.application.strports
53 <http://twistedmatrix.com/documents/current/api/twisted.application.strports.html>`_.
55 Writing "tcp:3456:interface=127.0.0.1" into the web.port line does the same
56 but binds to the loopback interface, ensuring that only the programs on the
57 local host can connect. Using "ssl:3456:privateKey=mykey.pem:certKey=cert.pem"
60 This webport can be set when the node is created by passing a --webport
61 option to the 'tahoe create-node' command. By default, the node listens on
62 port 3456, on the loopback (127.0.0.1) interface.
64 Basic Concepts: GET, PUT, DELETE, POST
65 ======================================
67 As described in `docs/architecture.rst <../architecture.rst>`_, each file
68 and directory in a Tahoe virtual filesystem is referenced by an identifier
69 that combines the designation of the object with the authority to do something
70 with it (such as read or modify the contents). This identifier is called a
71 "read-cap" or "write-cap", depending upon whether it enables read-only or
72 read-write access. These "caps" are also referred to as URIs (which may be
73 confusing because they are not currently `RFC3986
74 <http://tools.ietf.org/html/rfc3986>`_-compliant URIs).
76 The Tahoe web-based API is "REST-ful", meaning it implements the concepts of
77 "REpresentational State Transfer": the original scheme by which the World
78 Wide Web was intended to work. Each object (file or directory) is referenced
79 by a URL that includes the read- or write- cap. HTTP methods (GET, PUT, and
80 DELETE) are used to manipulate these objects. You can think of the URL as a
81 noun, and the method as a verb.
83 In REST, the GET method is used to retrieve information about an object, or
84 to retrieve some representation of the object itself. When the object is a
85 file, the basic GET method will simply return the contents of that file.
86 Other variations (generally implemented by adding query parameters to the
87 URL) will return information about the object, such as metadata. GET
88 operations are required to have no side-effects.
90 PUT is used to upload new objects into the filesystem, or to replace an
91 existing object. DELETE it used to delete objects from the filesystem. Both
92 PUT and DELETE are required to be idempotent: performing the same operation
93 multiple times must have the same side-effects as only performing it once.
95 POST is used for more complicated actions that cannot be expressed as a GET,
96 PUT, or DELETE. POST operations can be thought of as a method call: sending
97 some message to the object referenced by the URL. In Tahoe, POST is also used
98 for operations that must be triggered by an HTML form (including upload and
99 delete), because otherwise a regular web browser has no way to accomplish
100 these tasks. In general, everything that can be done with a PUT or DELETE can
101 also be done with a POST.
103 Tahoe's web API is designed for two different kinds of consumer. The first is
104 a program that needs to manipulate the virtual file system. Such programs are
105 expected to use the RESTful interface described above. The second is a human
106 using a standard web browser to work with the filesystem. This user is given
107 a series of HTML pages with links to download files, and forms that use POST
108 actions to upload, rename, and delete files.
110 When an error occurs, the HTTP response code will be set to an appropriate
111 400-series code (like 404 Not Found for an unknown childname, or 400 Bad Request
112 when the parameters to a webapi operation are invalid), and the HTTP response
113 body will usually contain a few lines of explanation as to the cause of the
114 error and possible responses. Unusual exceptions may result in a 500 Internal
115 Server Error as a catch-all, with a default response body containing
116 a Nevow-generated HTML-ized representation of the Python exception stack trace
117 that caused the problem. CLI programs which want to copy the response body to
118 stderr should provide an "Accept: text/plain" header to their requests to get
119 a plain text stack trace instead. If the Accept header contains ``*/*``, or
120 ``text/*``, or text/html (or if there is no Accept header), HTML tracebacks will
126 Tahoe uses a variety of read- and write- caps to identify files and
127 directories. The most common of these is the "immutable file read-cap", which
128 is used for most uploaded files. These read-caps look like the following::
130 URI:CHK:ime6pvkaxuetdfah2p2f35pe54:4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a:3:10:202
132 The next most common is a "directory write-cap", which provides both read and
133 write access to a directory, and look like this::
135 URI:DIR2:djrdkfawoqihigoett4g6auz6a:jx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq
137 There are also "directory read-caps", which start with "URI:DIR2-RO:", and
138 give read-only access to a directory. Finally there are also mutable file
139 read- and write- caps, which start with "URI:SSK", and give access to mutable
142 (Later versions of Tahoe will make these strings shorter, and will remove the
143 unfortunate colons, which must be escaped when these caps are embedded in
146 To refer to any Tahoe object through the web API, you simply need to combine
147 a prefix (which indicates the HTTP server to use) with the cap (which
148 indicates which object inside that server to access). Since the default Tahoe
149 webport is 3456, the most common prefix is one that will use a local node
150 listening on this port::
152 http://127.0.0.1:3456/uri/ + $CAP
154 So, to access the directory named above (which happens to be the
155 publically-writeable sample directory on the Tahoe test grid, described at
156 http://allmydata.org/trac/tahoe/wiki/TestGrid), the URL would be::
158 http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/
160 (note that the colons in the directory-cap are url-encoded into "%3A"
163 Likewise, to access the file named above, use::
165 http://127.0.0.1:3456/uri/URI%3ACHK%3Aime6pvkaxuetdfah2p2f35pe54%3A4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a%3A3%3A10%3A202
167 In the rest of this document, we'll use "$DIRCAP" as shorthand for a read-cap
168 or write-cap that refers to a directory, and "$FILECAP" to abbreviate a cap
169 that refers to a file (whether mutable or immutable). So those URLs above can
172 http://127.0.0.1:3456/uri/$DIRCAP/
173 http://127.0.0.1:3456/uri/$FILECAP
175 The operation summaries below will abbreviate these further, by eliding the
176 server prefix. They will be displayed like this::
185 Tahoe directories contain named child entries, just like directories in a regular
186 local filesystem. These child entries, called "dirnodes", consist of a name,
187 metadata, a write slot, and a read slot. The write and read slots normally contain
188 a write-cap and read-cap referring to the same object, which can be either a file
189 or a subdirectory. The write slot may be empty (actually, both may be empty,
190 but that is unusual).
192 If you have a Tahoe URL that refers to a directory, and want to reference a
193 named child inside it, just append the child name to the URL. For example, if
194 our sample directory contains a file named "welcome.txt", we can refer to
197 http://127.0.0.1:3456/uri/$DIRCAP/welcome.txt
199 (or http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/welcome.txt)
201 Multiple levels of subdirectories can be handled this way::
203 http://127.0.0.1:3456/uri/$DIRCAP/tahoe-source/docs/architecture.rst
205 In this document, when we need to refer to a URL that references a file using
206 this child-of-some-directory format, we'll use the following string::
208 /uri/$DIRCAP/[SUBDIRS../]FILENAME
210 The "[SUBDIRS../]" part means that there are zero or more (optional)
211 subdirectory names in the middle of the URL. The "FILENAME" at the end means
212 that this whole URL refers to a file of some sort, rather than to a
215 When we need to refer specifically to a directory in this way, we'll write::
217 /uri/$DIRCAP/[SUBDIRS../]SUBDIR
220 Note that all components of pathnames in URLs are required to be UTF-8
221 encoded, so "resume.doc" (with an acute accent on both E's) would be accessed
224 http://127.0.0.1:3456/uri/$DIRCAP/r%C3%A9sum%C3%A9.doc
226 Also note that the filenames inside upload POST forms are interpreted using
227 whatever character set was provided in the conventional '_charset' field, and
228 defaults to UTF-8 if not otherwise specified. The JSON representation of each
229 directory contains native unicode strings. Tahoe directories are specified to
230 contain unicode filenames, and cannot contain binary strings that are not
231 representable as such.
233 All Tahoe operations that refer to existing files or directories must include
234 a suitable read- or write- cap in the URL: the webapi server won't add one
235 for you. If you don't know the cap, you can't access the file. This allows
236 the security properties of Tahoe caps to be extended across the webapi
239 Slow Operations, Progress, and Cancelling
240 =========================================
242 Certain operations can be expected to take a long time. The "t=deep-check",
243 described below, will recursively visit every file and directory reachable
244 from a given starting point, which can take minutes or even hours for
245 extremely large directory structures. A single long-running HTTP request is a
246 fragile thing: proxies, NAT boxes, browsers, and users may all grow impatient
247 with waiting and give up on the connection.
249 For this reason, long-running operations have an "operation handle", which
250 can be used to poll for status/progress messages while the operation
251 proceeds. This handle can also be used to cancel the operation. These handles
252 are created by the client, and passed in as a an "ophandle=" query argument
253 to the POST or PUT request which starts the operation. The following
254 operations can then be used to retrieve status:
256 ``GET /operations/$HANDLE?output=HTML (with or without t=status)``
258 ``GET /operations/$HANDLE?output=JSON (same)``
260 These two retrieve the current status of the given operation. Each operation
261 presents a different sort of information, but in general the page retrieved
264 * whether the operation is complete, or if it is still running
265 * how much of the operation is complete, and how much is left, if possible
267 Note that the final status output can be quite large: a deep-manifest of a
268 directory structure with 300k directories and 200k unique files is about
269 275MB of JSON, and might take two minutes to generate. For this reason, the
270 full status is not provided until the operation has completed.
272 The HTML form will include a meta-refresh tag, which will cause a regular
273 web browser to reload the status page about 60 seconds later. This tag will
274 be removed once the operation has completed.
276 There may be more status information available under
277 /operations/$HANDLE/$ETC : i.e., the handle forms the root of a URL space.
279 ``POST /operations/$HANDLE?t=cancel``
281 This terminates the operation, and returns an HTML page explaining what was
282 cancelled. If the operation handle has already expired (see below), this
283 POST will return a 404, which indicates that the operation is no longer
284 running (either it was completed or terminated). The response body will be
285 the same as a GET /operations/$HANDLE on this operation handle, and the
286 handle will be expired immediately afterwards.
288 The operation handle will eventually expire, to avoid consuming an unbounded
289 amount of memory. The handle's time-to-live can be reset at any time, by
290 passing a retain-for= argument (with a count of seconds) to either the
291 initial POST that starts the operation, or the subsequent GET request which
292 asks about the operation. For example, if a 'GET
293 /operations/$HANDLE?output=JSON&retain-for=600' query is performed, the
294 handle will remain active for 600 seconds (10 minutes) after the GET was
297 In addition, if the GET includes a release-after-complete=True argument, and
298 the operation has completed, the operation handle will be released
301 If a retain-for= argument is not used, the default handle lifetimes are:
303 * handles will remain valid at least until their operation finishes
304 * uncollected handles for finished operations (i.e. handles for
305 operations that have finished but for which the GET page has not been
306 accessed since completion) will remain valid for four days, or for
307 the total time consumed by the operation, whichever is greater.
308 * collected handles (i.e. the GET page has been retrieved at least once
309 since the operation completed) will remain valid for one day.
311 Many "slow" operations can begin to use unacceptable amounts of memory when
312 operating on large directory structures. The memory usage increases when the
313 ophandle is polled, as the results must be copied into a JSON string, sent
314 over the wire, then parsed by a client. So, as an alternative, many "slow"
315 operations have streaming equivalents. These equivalents do not use operation
316 handles. Instead, they emit line-oriented status results immediately. Client
317 code can cancel the operation by simply closing the HTTP connection.
319 Programmatic Operations
320 =======================
322 Now that we know how to build URLs that refer to files and directories in a
323 Tahoe virtual filesystem, what sorts of operations can we do with those URLs?
324 This section contains a catalog of GET, PUT, DELETE, and POST operations that
325 can be performed on these URLs. This set of operations are aimed at programs
326 that use HTTP to communicate with a Tahoe node. A later section describes
327 operations that are intended for web browsers.
332 ``GET /uri/$FILECAP``
334 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME``
336 This will retrieve the contents of the given file. The HTTP response body
337 will contain the sequence of bytes that make up the file.
339 To view files in a web browser, you may want more control over the
340 Content-Type and Content-Disposition headers. Please see the next section
341 "Browser Operations", for details on how to modify these URLs for that
344 Writing/Uploading A File
345 ------------------------
347 ``PUT /uri/$FILECAP``
349 ``PUT /uri/$DIRCAP/[SUBDIRS../]FILENAME``
351 Upload a file, using the data from the HTTP request body, and add whatever
352 child links and subdirectories are necessary to make the file available at
353 the given location. Once this operation succeeds, a GET on the same URL will
354 retrieve the same contents that were just uploaded. This will create any
355 necessary intermediate subdirectories.
357 To use the /uri/$FILECAP form, $FILECAP must be a write-cap for a mutable file.
359 In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a
360 writeable mutable file, that file's contents will be overwritten in-place. If
361 it is a read-cap for a mutable file, an error will occur. If it is an
362 immutable file, the old file will be discarded, and a new one will be put in
365 When creating a new file, if "mutable=true" is in the query arguments, the
366 operation will create a mutable file instead of an immutable one.
368 This returns the file-cap of the resulting file. If a new file was created
369 by this method, the HTTP response code (as dictated by rfc2616) will be set
370 to 201 CREATED. If an existing file was replaced or modified, the response
373 Note that the 'curl -T localfile http://127.0.0.1:3456/uri/$DIRCAP/foo.txt'
374 command can be used to invoke this operation.
378 This uploads a file, and produces a file-cap for the contents, but does not
379 attach the file into the filesystem. No directories will be modified by
380 this operation. The file-cap is returned as the body of the HTTP response.
382 If "mutable=true" is in the query arguments, the operation will create a
383 mutable file, and return its write-cap in the HTTP respose. The default is
384 to create an immutable file, returning the read-cap as a response.
386 Creating A New Directory
387 ------------------------
389 ``POST /uri?t=mkdir``
393 Create a new empty directory and return its write-cap as the HTTP response
394 body. This does not make the newly created directory visible from the
395 filesystem. The "PUT" operation is provided for backwards compatibility:
396 new code should use POST.
398 ``POST /uri?t=mkdir-with-children``
400 Create a new directory, populated with a set of child nodes, and return its
401 write-cap as the HTTP response body. The new directory is not attached to
402 any other directory: the returned write-cap is the only reference to it.
404 Initial children are provided as the body of the POST form (this is more
405 efficient than doing separate mkdir and set_children operations). If the
406 body is empty, the new directory will be empty. If not empty, the body will
407 be interpreted as a UTF-8 JSON-encoded dictionary of children with which the
408 new directory should be populated, using the same format as would be
409 returned in the 'children' value of the t=json GET request, described below.
410 Each dictionary key should be a child name, and each value should be a list
411 of [TYPE, PROPDICT], where PROPDICT contains "rw_uri", "ro_uri", and
412 "metadata" keys (all others are ignored). For example, the PUT request body
416 "Fran\u00e7ais": [ "filenode", {
417 "ro_uri": "URI:CHK:...",
420 "ctime": 1202777696.7564139,
421 "mtime": 1202777696.7564139,
423 "linkcrtime": 1202777696.7564139,
424 "linkmotime": 1202777696.7564139
426 "subdir": [ "dirnode", {
427 "rw_uri": "URI:DIR2:...",
428 "ro_uri": "URI:DIR2-RO:...",
430 "ctime": 1202778102.7589991,
431 "mtime": 1202778111.2160511,
433 "linkcrtime": 1202777696.7564139,
434 "linkmotime": 1202777696.7564139
438 For forward-compatibility, a mutable directory can also contain caps in
439 a format that is unknown to the webapi server. When such caps are retrieved
440 from a mutable directory in a "ro_uri" field, they will be prefixed with
441 the string "ro.", indicating that they must not be decoded without
442 checking that they are read-only. The "ro." prefix must not be stripped
443 off without performing this check. (Future versions of the webapi server
444 will perform it where necessary.)
446 If both the "rw_uri" and "ro_uri" fields are present in a given PROPDICT,
447 and the webapi server recognizes the rw_uri as a write cap, then it will
448 reset the ro_uri to the corresponding read cap and discard the original
449 contents of ro_uri (in order to ensure that the two caps correspond to the
450 same object and that the ro_uri is in fact read-only). However this may not
451 happen for caps in a format unknown to the webapi server. Therefore, when
452 writing a directory the webapi client should ensure that the contents
453 of "rw_uri" and "ro_uri" for a given PROPDICT are a consistent
454 (write cap, read cap) pair if possible. If the webapi client only has
455 one cap and does not know whether it is a write cap or read cap, then
456 it is acceptable to set "rw_uri" to that cap and omit "ro_uri". The
457 client must not put a write cap into a "ro_uri" field.
459 The metadata may have a "no-write" field. If this is set to true in the
460 metadata of a link, it will not be possible to open that link for writing
461 via the SFTP frontend; see `FTP-and-SFTP.rst`_ for details.
462 Also, if the "no-write" field is set to true in the metadata of a link to
463 a mutable child, it will cause the link to be diminished to read-only.
465 .. _FTP-and-SFTP.rst: http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/frontents/FTP-and-SFTP.rst
467 Note that the webapi-using client application must not provide the
468 "Content-Type: multipart/form-data" header that usually accompanies HTML
469 form submissions, since the body is not formatted this way. Doing so will
470 cause a server error as the lower-level code misparses the request body.
472 Child file names should each be expressed as a unicode string, then used as
473 keys of the dictionary. The dictionary should then be converted into JSON,
474 and the resulting string encoded into UTF-8. This UTF-8 bytestring should
475 then be used as the POST body.
477 ``POST /uri?t=mkdir-immutable``
479 Like t=mkdir-with-children above, but the new directory will be
480 deep-immutable. This means that the directory itself is immutable, and that
481 it can only contain objects that are treated as being deep-immutable, like
482 immutable files, literal files, and deep-immutable directories.
484 For forward-compatibility, a deep-immutable directory can also contain caps
485 in a format that is unknown to the webapi server. When such caps are retrieved
486 from a deep-immutable directory in a "ro_uri" field, they will be prefixed
487 with the string "imm.", indicating that they must not be decoded without
488 checking that they are immutable. The "imm." prefix must not be stripped
489 off without performing this check. (Future versions of the webapi server
490 will perform it where necessary.)
492 The cap for each child may be given either in the "rw_uri" or "ro_uri"
493 field of the PROPDICT (not both). If a cap is given in the "rw_uri" field,
494 then the webapi server will check that it is an immutable read-cap of a
495 *known* format, and give an error if it is not. If a cap is given in the
496 "ro_uri" field, then the webapi server will still check whether known
497 caps are immutable, but for unknown caps it will simply assume that the
498 cap can be stored, as described above. Note that an attacker would be
499 able to store any cap in an immutable directory, so this check when
500 creating the directory is only to help non-malicious clients to avoid
501 accidentally giving away more authority than intended.
503 A non-empty request body is mandatory, since after the directory is created,
504 it will not be possible to add more children to it.
506 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir``
508 ``PUT /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir``
510 Create new directories as necessary to make sure that the named target
511 ($DIRCAP/SUBDIRS../SUBDIR) is a directory. This will create additional
512 intermediate mutable directories as necessary. If the named target directory
513 already exists, this will make no changes to it.
515 If the final directory is created, it will be empty.
517 This operation will return an error if a blocking file is present at any of
518 the parent names, preventing the server from creating the necessary parent
519 directory; or if it would require changing an immutable directory.
521 The write-cap of the new directory will be returned as the HTTP response
524 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-with-children``
526 Like /uri?t=mkdir-with-children, but the final directory is created as a
527 child of an existing mutable directory. This will create additional
528 intermediate mutable directories as necessary. If the final directory is
529 created, it will be populated with initial children from the POST request
530 body, as described above.
532 This operation will return an error if a blocking file is present at any of
533 the parent names, preventing the server from creating the necessary parent
534 directory; or if it would require changing an immutable directory; or if
535 the immediate parent directory already has a a child named SUBDIR.
537 ``POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-immutable``
539 Like /uri?t=mkdir-immutable, but the final directory is created as a child
540 of an existing mutable directory. The final directory will be deep-immutable,
541 and will be populated with the children specified as a JSON dictionary in
542 the POST request body.
544 In Tahoe 1.6 this operation creates intermediate mutable directories if
545 necessary, but that behaviour should not be relied on; see ticket #920.
547 This operation will return an error if the parent directory is immutable,
548 or already has a child named SUBDIR.
550 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME``
552 Create a new empty mutable directory and attach it to the given existing
553 directory. This will create additional intermediate directories as necessary.
555 This operation will return an error if a blocking file is present at any of
556 the parent names, preventing the server from creating the necessary parent
557 directory, or if it would require changing any immutable directory.
559 The URL of this operation points to the parent of the bottommost new directory,
560 whereas the /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir operation above has a URL
561 that points directly to the bottommost new directory.
563 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME``
565 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME, but the new directory will
566 be populated with initial children via the POST request body. This command
567 will create additional intermediate mutable directories as necessary.
569 This operation will return an error if a blocking file is present at any of
570 the parent names, preventing the server from creating the necessary parent
571 directory; or if it would require changing an immutable directory; or if
572 the immediate parent directory already has a a child named NAME.
574 Note that the name= argument must be passed as a queryarg, because the POST
575 request body is used for the initial children JSON.
577 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-immutable&name=NAME``
579 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME, but the
580 final directory will be deep-immutable. The children are specified as a
581 JSON dictionary in the POST request body. Again, the name= argument must be
582 passed as a queryarg.
584 In Tahoe 1.6 this operation creates intermediate mutable directories if
585 necessary, but that behaviour should not be relied on; see ticket #920.
587 This operation will return an error if the parent directory is immutable,
588 or already has a child named NAME.
590 Get Information About A File Or Directory (as JSON)
591 ---------------------------------------------------
593 ``GET /uri/$FILECAP?t=json``
595 ``GET /uri/$DIRCAP?t=json``
597 ``GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json``
599 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json``
601 This returns a machine-parseable JSON-encoded description of the given
602 object. The JSON always contains a list, and the first element of the list is
603 always a flag that indicates whether the referenced object is a file or a
604 directory. If it is a capability to a file, then the information includes
605 file size and URI, like this::
607 GET /uri/$FILECAP?t=json :
611 "verify_uri": verify_uri,
616 If it is a capability to a directory followed by a path from that directory
617 to a file, then the information also includes metadata from the link to the
618 file in the parent directory, like this::
620 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json
624 "verify_uri": verify_uri,
628 "ctime": 1202777696.7564139,
629 "mtime": 1202777696.7564139,
631 "linkcrtime": 1202777696.7564139,
632 "linkmotime": 1202777696.7564139
635 If it is a directory, then it includes information about the children of
636 this directory, as a mapping from child name to a set of data about the
637 child (the same data that would appear in a corresponding GET?t=json of the
638 child itself). The child entries also include metadata about each child,
639 including link-creation- and link-change- timestamps. The output looks like
642 GET /uri/$DIRCAP?t=json :
643 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json :
646 "rw_uri": read_write_uri,
647 "ro_uri": read_only_uri,
648 "verify_uri": verify_uri,
651 "foo.txt": [ "filenode", {
655 "ctime": 1202777696.7564139,
656 "mtime": 1202777696.7564139,
658 "linkcrtime": 1202777696.7564139,
659 "linkmotime": 1202777696.7564139
661 "subdir": [ "dirnode", {
665 "ctime": 1202778102.7589991,
666 "mtime": 1202778111.2160511,
668 "linkcrtime": 1202777696.7564139,
669 "linkmotime": 1202777696.7564139
673 In the above example, note how 'children' is a dictionary in which the keys
674 are child names and the values depend upon whether the child is a file or a
675 directory. The value is mostly the same as the JSON representation of the
676 child object (except that directories do not recurse -- the "children"
677 entry of the child is omitted, and the directory view includes the metadata
678 that is stored on the directory edge).
680 The rw_uri field will be present in the information about a directory
681 if and only if you have read-write access to that directory. The verify_uri
682 field will be present if and only if the object has a verify-cap
683 (non-distributed LIT files do not have verify-caps).
685 If the cap is of an unknown format, then the file size and verify_uri will
688 GET /uri/$UNKNOWNCAP?t=json :
691 "ro_uri": unknown_read_uri
694 GET /uri/$DIRCAP/[SUBDIRS../]UNKNOWNCHILDNAME?t=json :
697 "rw_uri": unknown_write_uri,
698 "ro_uri": unknown_read_uri,
701 "ctime": 1202777696.7564139,
702 "mtime": 1202777696.7564139,
704 "linkcrtime": 1202777696.7564139,
705 "linkmotime": 1202777696.7564139
708 As in the case of file nodes, the metadata will only be present when the
709 capability is to a directory followed by a path. The "mutable" field is also
710 not always present; when it is absent, the mutability of the object is not
716 The value of the 'tahoe':'linkmotime' key is updated whenever a link to a
717 child is set. The value of the 'tahoe':'linkcrtime' key is updated whenever
718 a link to a child is created -- i.e. when there was not previously a link
721 Note however, that if the edge in the Tahoe filesystem points to a mutable
722 file and the contents of that mutable file is changed, then the
723 'tahoe':'linkmotime' value on that edge will *not* be updated, since the
724 edge itself wasn't updated -- only the mutable file was.
726 The timestamps are represented as a number of seconds since the UNIX epoch
727 (1970-01-01 00:00:00 UTC), with leap seconds not being counted in the long
730 In Tahoe earlier than v1.4.0, 'mtime' and 'ctime' keys were populated
731 instead of the 'tahoe':'linkmotime' and 'tahoe':'linkcrtime' keys. Starting
732 in Tahoe v1.4.0, the 'linkmotime'/'linkcrtime' keys in the 'tahoe' sub-dict
733 are populated. However, prior to Tahoe v1.7beta, a bug caused the 'tahoe'
734 sub-dict to be deleted by webapi requests in which new metadata is
735 specified, and not to be added to existing child links that lack it.
737 From Tahoe v1.7.0 onward, the 'mtime' and 'ctime' fields are no longer
738 populated or updated (see ticket #924), except by "tahoe backup" as
739 explained below. For backward compatibility, when an existing link is
740 updated and 'tahoe':'linkcrtime' is not present in the previous metadata
741 but 'ctime' is, the old value of 'ctime' is used as the new value of
742 'tahoe':'linkcrtime'.
744 The reason we added the new fields in Tahoe v1.4.0 is that there is a
745 "set_children" API (described below) which you can use to overwrite the
746 values of the 'mtime'/'ctime' pair, and this API is used by the
747 "tahoe backup" command (in Tahoe v1.3.0 and later) to set the 'mtime' and
748 'ctime' values when backing up files from a local filesystem into the
749 Tahoe filesystem. As of Tahoe v1.4.0, the set_children API cannot be used
750 to set anything under the 'tahoe' key of the metadata dict -- if you
751 include 'tahoe' keys in your 'metadata' arguments then it will silently
754 Therefore, if the 'tahoe' sub-dict is present, you can rely on the
755 'linkcrtime' and 'linkmotime' values therein to have the semantics described
756 above. (This is assuming that only official Tahoe clients have been used to
757 write those links, and that their system clocks were set to what you expected
758 -- there is nothing preventing someone from editing their Tahoe client or
759 writing their own Tahoe client which would overwrite those values however
760 they like, and there is nothing to constrain their system clock from taking
763 When an edge is created or updated by "tahoe backup", the 'mtime' and
764 'ctime' keys on that edge are set as follows:
766 * 'mtime' is set to the timestamp read from the local filesystem for the
767 "mtime" of the local file in question, which means the last time the
768 contents of that file were changed.
770 * On Windows, 'ctime' is set to the creation timestamp for the file
771 read from the local filesystem. On other platforms, 'ctime' is set to
772 the UNIX "ctime" of the local file, which means the last time that
773 either the contents or the metadata of the local file was changed.
775 There are several ways that the 'ctime' field could be confusing:
777 1. You might be confused about whether it reflects the time of the creation
778 of a link in the Tahoe filesystem (by a version of Tahoe < v1.7.0) or a
779 timestamp copied in by "tahoe backup" from a local filesystem.
781 2. You might be confused about whether it is a copy of the file creation
782 time (if "tahoe backup" was run on a Windows system) or of the last
783 contents-or-metadata change (if "tahoe backup" was run on a different
786 3. You might be confused by the fact that changing the contents of a
787 mutable file in Tahoe doesn't have any effect on any links pointing at
788 that file in any directories, although "tahoe backup" sets the link
789 'ctime'/'mtime' to reflect timestamps about the local file corresponding
790 to the Tahoe file to which the link points.
792 4. Also, quite apart from Tahoe, you might be confused about the meaning
793 of the "ctime" in UNIX local filesystems, which people sometimes think
794 means file creation time, but which actually means, in UNIX local
795 filesystems, the most recent time that the file contents or the file
796 metadata (such as owner, permission bits, extended attributes, etc.)
797 has changed. Note that although "ctime" does not mean file creation time
798 in UNIX, links created by a version of Tahoe prior to v1.7.0, and never
799 written by "tahoe backup", will have 'ctime' set to the link creation
803 Attaching an existing File or Directory by its read- or write-cap
804 -----------------------------------------------------------------
806 ``PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri``
808 This attaches a child object (either a file or directory) to a specified
809 location in the virtual filesystem. The child object is referenced by its
810 read- or write- cap, as provided in the HTTP request body. This will create
811 intermediate directories as necessary.
813 This is similar to a UNIX hardlink: by referencing a previously-uploaded file
814 (or previously-created directory) instead of uploading/creating a new one,
815 you can create two references to the same object.
817 The read- or write- cap of the child is provided in the body of the HTTP
818 request, and this same cap is returned in the response body.
820 The default behavior is to overwrite any existing object at the same
821 location. To prevent this (and make the operation return an error instead
822 of overwriting), add a "replace=false" argument, as "?t=uri&replace=false".
823 With replace=false, this operation will return an HTTP 409 "Conflict" error
824 if there is already an object at the given location, rather than
825 overwriting the existing object. To allow the operation to overwrite a
826 file, but return an error when trying to overwrite a directory, use
827 "replace=only-files" (this behavior is closer to the traditional UNIX "mv"
828 command). Note that "true", "t", and "1" are all synonyms for "True", and
829 "false", "f", and "0" are synonyms for "False", and the parameter is
832 Note that this operation does not take its child cap in the form of
833 separate "rw_uri" and "ro_uri" fields. Therefore, it cannot accept a
834 child cap in a format unknown to the webapi server, unless its URI
835 starts with "ro." or "imm.". This restriction is necessary because the
836 server is not able to attenuate an unknown write cap to a read cap.
837 Unknown URIs starting with "ro." or "imm.", on the other hand, are
838 assumed to represent read caps. The client should not prefix a write
839 cap with "ro." or "imm." and pass it to this operation, since that
840 would result in granting the cap's write authority to holders of the
843 Adding multiple files or directories to a parent directory at once
844 ------------------------------------------------------------------
846 ``POST /uri/$DIRCAP/[SUBDIRS..]?t=set_children``
848 ``POST /uri/$DIRCAP/[SUBDIRS..]?t=set-children`` (Tahoe >= v1.6)
850 This command adds multiple children to a directory in a single operation.
851 It reads the request body and interprets it as a JSON-encoded description
852 of the child names and read/write-caps that should be added.
854 The body should be a JSON-encoded dictionary, in the same format as the
855 "children" value returned by the "GET /uri/$DIRCAP?t=json" operation
856 described above. In this format, each key is a child names, and the
857 corresponding value is a tuple of (type, childinfo). "type" is ignored, and
858 "childinfo" is a dictionary that contains "rw_uri", "ro_uri", and
859 "metadata" keys. You can take the output of "GET /uri/$DIRCAP1?t=json" and
860 use it as the input to "POST /uri/$DIRCAP2?t=set_children" to make DIR2
861 look very much like DIR1 (except for any existing children of DIR2 that
862 were not overwritten, and any existing "tahoe" metadata keys as described
865 When the set_children request contains a child name that already exists in
866 the target directory, this command defaults to overwriting that child with
867 the new value (both child cap and metadata, but if the JSON data does not
868 contain a "metadata" key, the old child's metadata is preserved). The
869 command takes a boolean "overwrite=" query argument to control this
870 behavior. If you use "?t=set_children&overwrite=false", then an attempt to
871 replace an existing child will instead cause an error.
873 Any "tahoe" key in the new child's "metadata" value is ignored. Any
874 existing "tahoe" metadata is preserved. The metadata["tahoe"] value is
875 reserved for metadata generated by the tahoe node itself. The only two keys
876 currently placed here are "linkcrtime" and "linkmotime". For details, see
877 the section above entitled "Get Information About A File Or Directory (as
878 JSON)", in the "About the metadata" subsection.
880 Note that this command was introduced with the name "set_children", which
881 uses an underscore rather than a hyphen as other multi-word command names
882 do. The variant with a hyphen is now accepted, but clients that desire
883 backward compatibility should continue to use "set_children".
886 Deleting a File or Directory
887 ----------------------------
889 ``DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME``
891 This removes the given name from its parent directory. CHILDNAME is the
892 name to be removed, and $DIRCAP/SUBDIRS.. indicates the directory that will
895 Note that this does not actually delete the file or directory that the name
896 points to from the tahoe grid -- it only removes the named reference from
897 this directory. If there are other names in this directory or in other
898 directories that point to the resource, then it will remain accessible
899 through those paths. Even if all names pointing to this object are removed
900 from their parent directories, then someone with possession of its read-cap
901 can continue to access the object through that cap.
903 The object will only become completely unreachable once 1: there are no
904 reachable directories that reference it, and 2: nobody is holding a read-
905 or write- cap to the object. (This behavior is very similar to the way
906 hardlinks and anonymous files work in traditional UNIX filesystems).
908 This operation will not modify more than a single directory. Intermediate
909 directories which were implicitly created by PUT or POST methods will *not*
910 be automatically removed by DELETE.
912 This method returns the file- or directory- cap of the object that was just
915 Browser Operations: Human-oriented interfaces
916 =============================================
918 This section describes the HTTP operations that provide support for humans
919 running a web browser. Most of these operations use HTML forms that use POST
920 to drive the Tahoe node. This section is intended for HTML authors who want
921 to write web pages that contain forms and buttons which manipulate the Tahoe
924 Note that for all POST operations, the arguments listed can be provided
925 either as URL query arguments or as form body fields. URL query arguments are
926 separated from the main URL by "?", and from each other by "&". For example,
927 "POST /uri/$DIRCAP?t=upload&mutable=true". Form body fields are usually
928 specified by using <input type="hidden"> elements. For clarity, the
929 descriptions below display the most significant arguments as URL query args.
931 Viewing A Directory (as HTML)
932 -----------------------------
934 ``GET /uri/$DIRCAP/[SUBDIRS../]``
936 This returns an HTML page, intended to be displayed to a human by a web
937 browser, which contains HREF links to all files and directories reachable
938 from this directory. These HREF links do not have a t= argument, meaning
939 that a human who follows them will get pages also meant for a human. It also
940 contains forms to upload new files, and to delete files and directories.
941 Those forms use POST methods to do their job.
943 Viewing/Downloading a File
944 --------------------------
946 ``GET /uri/$FILECAP``
948 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME``
950 This will retrieve the contents of the given file. The HTTP response body
951 will contain the sequence of bytes that make up the file.
953 If you want the HTTP response to include a useful Content-Type header,
954 either use the second form (which starts with a $DIRCAP), or add a
955 "filename=foo" query argument, like "GET /uri/$FILECAP?filename=foo.jpg".
956 The bare "GET /uri/$FILECAP" does not give the Tahoe node enough information
957 to determine a Content-Type (since Tahoe immutable files are merely
958 sequences of bytes, not typed+named file objects).
960 If the URL has both filename= and "save=true" in the query arguments, then
961 the server to add a "Content-Disposition: attachment" header, along with a
962 filename= parameter. When a user clicks on such a link, most browsers will
963 offer to let the user save the file instead of displaying it inline (indeed,
964 most browsers will refuse to display it inline). "true", "t", "1", and other
965 case-insensitive equivalents are all treated the same.
967 Character-set handling in URLs and HTTP headers is a dubious art [1]_. For
968 maximum compatibility, Tahoe simply copies the bytes from the filename=
969 argument into the Content-Disposition header's filename= parameter, without
970 trying to interpret them in any particular way.
973 ``GET /named/$FILECAP/FILENAME``
975 This is an alternate download form which makes it easier to get the correct
976 filename. The Tahoe server will provide the contents of the given file, with
977 a Content-Type header derived from the given filename. This form is used to
978 get browsers to use the "Save Link As" feature correctly, and also helps
979 command-line tools like "wget" and "curl" use the right filename. Note that
980 this form can *only* be used with file caps; it is an error to use a
981 directory cap after the /named/ prefix.
983 Get Information About A File Or Directory (as HTML)
984 ---------------------------------------------------
986 ``GET /uri/$FILECAP?t=info``
988 ``GET /uri/$DIRCAP/?t=info``
990 ``GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR/?t=info``
992 ``GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=info``
994 This returns a human-oriented HTML page with more detail about the selected
995 file or directory object. This page contains the following items:
999 * JSON representation
1000 * raw contents (text/plain)
1001 * access caps (URIs): verify-cap, read-cap, write-cap (for mutable objects)
1002 * check/verify/repair form
1003 * deep-check/deep-size/deep-stats/manifest (for directories)
1004 * replace-conents form (for mutable files)
1006 Creating a Directory
1007 --------------------
1009 ``POST /uri?t=mkdir``
1011 This creates a new empty directory, but does not attach it to the virtual
1014 If a "redirect_to_result=true" argument is provided, then the HTTP response
1015 will cause the web browser to be redirected to a /uri/$DIRCAP page that
1016 gives access to the newly-created directory. If you bookmark this page,
1017 you'll be able to get back to the directory again in the future. This is the
1018 recommended way to start working with a Tahoe server: create a new unlinked
1019 directory (using redirect_to_result=true), then bookmark the resulting
1020 /uri/$DIRCAP page. There is a "create directory" button on the Welcome page
1021 to invoke this action.
1023 If "redirect_to_result=true" is not provided (or is given a value of
1024 "false"), then the HTTP response body will simply be the write-cap of the
1027 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME``
1029 This creates a new empty directory as a child of the designated SUBDIR. This
1030 will create additional intermediate directories as necessary.
1032 If a "when_done=URL" argument is provided, the HTTP response will cause the
1033 web browser to redirect to the given URL. This provides a convenient way to
1034 return the browser to the directory that was just modified. Without a
1035 when_done= argument, the HTTP response will simply contain the write-cap of
1036 the directory that was just created.
1042 ``POST /uri?t=upload``
1044 This uploads a file, and produces a file-cap for the contents, but does not
1045 attach the file into the filesystem. No directories will be modified by
1048 The file must be provided as the "file" field of an HTML encoded form body,
1049 produced in response to an HTML form like this::
1051 <form action="/uri" method="POST" enctype="multipart/form-data">
1052 <input type="hidden" name="t" value="upload" />
1053 <input type="file" name="file" />
1054 <input type="submit" value="Upload Unlinked" />
1057 If a "when_done=URL" argument is provided, the response body will cause the
1058 browser to redirect to the given URL. If the when_done= URL has the string
1059 "%(uri)s" in it, that string will be replaced by a URL-escaped form of the
1060 newly created file-cap. (Note that without this substitution, there is no
1061 way to access the file that was just uploaded).
1063 The default (in the absence of when_done=) is to return an HTML page that
1064 describes the results of the upload. This page will contain information
1065 about which storage servers were used for the upload, how long each
1066 operation took, etc.
1068 If a "mutable=true" argument is provided, the operation will create a
1069 mutable file, and the response body will contain the write-cap instead of
1070 the upload results page. The default is to create an immutable file,
1071 returning the upload results page as a response.
1074 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=upload``
1076 This uploads a file, and attaches it as a new child of the given directory,
1077 which must be mutable. The file must be provided as the "file" field of an
1078 HTML-encoded form body, produced in response to an HTML form like this::
1080 <form action="." method="POST" enctype="multipart/form-data">
1081 <input type="hidden" name="t" value="upload" />
1082 <input type="file" name="file" />
1083 <input type="submit" value="Upload" />
1086 A "name=" argument can be provided to specify the new child's name,
1087 otherwise it will be taken from the "filename" field of the upload form
1088 (most web browsers will copy the last component of the original file's
1089 pathname into this field). To avoid confusion, name= is not allowed to
1092 If there is already a child with that name, and it is a mutable file, then
1093 its contents are replaced with the data being uploaded. If it is not a
1094 mutable file, the default behavior is to remove the existing child before
1095 creating a new one. To prevent this (and make the operation return an error
1096 instead of overwriting the old child), add a "replace=false" argument, as
1097 "?t=upload&replace=false". With replace=false, this operation will return an
1098 HTTP 409 "Conflict" error if there is already an object at the given
1099 location, rather than overwriting the existing object. Note that "true",
1100 "t", and "1" are all synonyms for "True", and "false", "f", and "0" are
1101 synonyms for "False". the parameter is case-insensitive.
1103 This will create additional intermediate directories as necessary, although
1104 since it is expected to be triggered by a form that was retrieved by "GET
1105 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1108 If a "mutable=true" argument is provided, any new file that is created will
1109 be a mutable file instead of an immutable one. <input type="checkbox"
1110 name="mutable" /> will give the user a way to set this option.
1112 If a "when_done=URL" argument is provided, the HTTP response will cause the
1113 web browser to redirect to the given URL. This provides a convenient way to
1114 return the browser to the directory that was just modified. Without a
1115 when_done= argument, the HTTP response will simply contain the file-cap of
1116 the file that was just uploaded (a write-cap for mutable files, or a
1117 read-cap for immutable files).
1119 ``POST /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=upload``
1121 This also uploads a file and attaches it as a new child of the given
1122 directory, which must be mutable. It is a slight variant of the previous
1123 operation, as the URL refers to the target file rather than the parent
1124 directory. It is otherwise identical: this accepts mutable= and when_done=
1127 ``POST /uri/$FILECAP?t=upload``
1129 This modifies the contents of an existing mutable file in-place. An error is
1130 signalled if $FILECAP does not refer to a mutable file. It behaves just like
1131 the "PUT /uri/$FILECAP" form, but uses a POST for the benefit of HTML forms
1134 Attaching An Existing File Or Directory (by URI)
1135 ------------------------------------------------
1137 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=uri&name=CHILDNAME&uri=CHILDCAP``
1139 This attaches a given read- or write- cap "CHILDCAP" to the designated
1140 directory, with a specified child name. This behaves much like the PUT t=uri
1141 operation, and is a lot like a UNIX hardlink. It is subject to the same
1142 restrictions as that operation on the use of cap formats unknown to the
1145 This will create additional intermediate directories as necessary, although
1146 since it is expected to be triggered by a form that was retrieved by "GET
1147 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1150 This accepts the same replace= argument as POST t=upload.
1155 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME``
1157 This instructs the node to remove a child object (file or subdirectory) from
1158 the given directory, which must be mutable. Note that the entire subtree is
1159 unlinked from the parent. Unlike deleting a subdirectory in a UNIX local
1160 filesystem, the subtree need not be empty; if it isn't, then other references
1161 into the subtree will see that the child subdirectories are not modified by
1162 this operation. Only the link from the given directory to its child is severed.
1167 ``POST /uri/$DIRCAP/[SUBDIRS../]?t=rename&from_name=OLD&to_name=NEW``
1169 This instructs the node to rename a child of the given directory, which must
1170 be mutable. This has a similar effect to removing the child, then adding the
1171 same child-cap under the new name, except that it preserves metadata. This
1172 operation cannot move the child to a different directory.
1174 This operation will replace any existing child of the new name, making it
1175 behave like the UNIX "``mv -f``" command.
1180 ``GET /uri?uri=$CAP``
1182 This causes a redirect to /uri/$CAP, and retains any additional query
1183 arguments (like filename= or save=). This is for the convenience of web
1184 forms which allow the user to paste in a read- or write- cap (obtained
1185 through some out-of-band channel, like IM or email).
1187 Note that this form merely redirects to the specific file or directory
1188 indicated by the $CAP: unlike the GET /uri/$DIRCAP form, you cannot
1189 traverse to children by appending additional path segments to the URL.
1191 ``GET /uri/$DIRCAP/[SUBDIRS../]?t=rename-form&name=$CHILDNAME``
1193 This provides a useful facility to browser-based user interfaces. It
1194 returns a page containing a form targetting the "POST $DIRCAP t=rename"
1195 functionality described above, with the provided $CHILDNAME present in the
1196 'from_name' field of that form. I.e. this presents a form offering to
1197 rename $CHILDNAME, requesting the new name, and submitting POST rename.
1199 ``GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri``
1201 This returns the file- or directory- cap for the specified object.
1203 ``GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=readonly-uri``
1205 This returns a read-only file- or directory- cap for the specified object.
1206 If the object is an immutable file, this will return the same value as
1209 Debugging and Testing Features
1210 ------------------------------
1212 These URLs are less-likely to be helpful to the casual Tahoe user, and are
1213 mainly intended for developers.
1215 ``POST $URL?t=check``
1217 This triggers the FileChecker to determine the current "health" of the
1218 given file or directory, by counting how many shares are available. The
1219 page that is returned will display the results. This can be used as a "show
1220 me detailed information about this file" page.
1222 If a verify=true argument is provided, the node will perform a more
1223 intensive check, downloading and verifying every single bit of every share.
1225 If an add-lease=true argument is provided, the node will also add (or
1226 renew) a lease to every share it encounters. Each lease will keep the share
1227 alive for a certain period of time (one month by default). Once the last
1228 lease expires or is explicitly cancelled, the storage server is allowed to
1231 If an output=JSON argument is provided, the response will be
1232 machine-readable JSON instead of human-oriented HTML. The data is a
1233 dictionary with the following keys::
1235 storage-index: a base32-encoded string with the objects's storage index,
1236 or an empty string for LIT files
1237 summary: a string, with a one-line summary of the stats of the file
1238 results: a dictionary that describes the state of the file. For LIT files,
1239 this dictionary has only the 'healthy' key, which will always be
1240 True. For distributed files, this dictionary has the following
1242 count-shares-good: the number of good shares that were found
1243 count-shares-needed: 'k', the number of shares required for recovery
1244 count-shares-expected: 'N', the number of total shares generated
1245 count-good-share-hosts: this was intended to be the number of distinct
1246 storage servers with good shares. It is currently
1247 (as of Tahoe-LAFS v1.8.0) computed incorrectly;
1249 count-wrong-shares: for mutable files, the number of shares for
1250 versions other than the 'best' one (highest
1251 sequence number, highest roothash). These are
1253 count-recoverable-versions: for mutable files, the number of
1254 recoverable versions of the file. For
1255 a healthy file, this will equal 1.
1256 count-unrecoverable-versions: for mutable files, the number of
1257 unrecoverable versions of the file.
1258 For a healthy file, this will be 0.
1259 count-corrupt-shares: the number of shares with integrity failures
1260 list-corrupt-shares: a list of "share locators", one for each share
1261 that was found to be corrupt. Each share locator
1262 is a list of (serverid, storage_index, sharenum).
1263 needs-rebalancing: (bool) True if there are multiple shares on a single
1264 storage server, indicating a reduction in reliability
1265 that could be resolved by moving shares to new
1267 servers-responding: list of base32-encoded storage server identifiers,
1268 one for each server which responded to the share
1270 healthy: (bool) True if the file is completely healthy, False otherwise.
1271 Healthy files have at least N good shares. Overlapping shares
1272 do not currently cause a file to be marked unhealthy. If there
1273 are at least N good shares, then corrupt shares do not cause the
1274 file to be marked unhealthy, although the corrupt shares will be
1275 listed in the results (list-corrupt-shares) and should be manually
1276 removed to wasting time in subsequent downloads (as the
1277 downloader rediscovers the corruption and uses alternate shares).
1278 Future compatibility: the meaning of this field may change to
1279 reflect whether the servers-of-happiness criterion is met
1281 sharemap: dict mapping share identifier to list of serverids
1282 (base32-encoded strings). This indicates which servers are
1283 holding which shares. For immutable files, the shareid is
1284 an integer (the share number, from 0 to N-1). For
1285 immutable files, it is a string of the form
1286 'seq%d-%s-sh%d', containing the sequence number, the
1287 roothash, and the share number.
1289 ``POST $URL?t=start-deep-check`` (must add &ophandle=XYZ)
1291 This initiates a recursive walk of all files and directories reachable from
1292 the target, performing a check on each one just like t=check. The result
1293 page will contain a summary of the results, including details on any
1294 file/directory that was not fully healthy.
1296 t=start-deep-check can only be invoked on a directory. An error (400
1297 BAD_REQUEST) will be signalled if it is invoked on a file. The recursive
1298 walker will deal with loops safely.
1300 This accepts the same verify= and add-lease= arguments as t=check.
1302 Since this operation can take a long time (perhaps a second per object),
1303 the ophandle= argument is required (see "Slow Operations, Progress, and
1304 Cancelling" above). The response to this POST will be a redirect to the
1305 corresponding /operations/$HANDLE page (with output=HTML or output=JSON to
1306 match the output= argument given to the POST). The deep-check operation
1307 will continue to run in the background, and the /operations page should be
1308 used to find out when the operation is done.
1310 Detailed check results for non-healthy files and directories will be
1311 available under /operations/$HANDLE/$STORAGEINDEX, and the HTML status will
1312 contain links to these detailed results.
1314 The HTML /operations/$HANDLE page for incomplete operations will contain a
1315 meta-refresh tag, set to 60 seconds, so that a browser which uses
1316 deep-check will automatically poll until the operation has completed.
1318 The JSON page (/options/$HANDLE?output=JSON) will contain a
1319 machine-readable JSON dictionary with the following keys::
1321 finished: a boolean, True if the operation is complete, else False. Some
1322 of the remaining keys may not be present until the operation
1324 root-storage-index: a base32-encoded string with the storage index of the
1325 starting point of the deep-check operation
1326 count-objects-checked: count of how many objects were checked. Note that
1327 non-distributed objects (i.e. small immutable LIT
1328 files) are not checked, since for these objects,
1329 the data is contained entirely in the URI.
1330 count-objects-healthy: how many of those objects were completely healthy
1331 count-objects-unhealthy: how many were damaged in some way
1332 count-corrupt-shares: how many shares were found to have corruption,
1333 summed over all objects examined
1334 list-corrupt-shares: a list of "share identifiers", one for each share
1335 that was found to be corrupt. Each share identifier
1336 is a list of (serverid, storage_index, sharenum).
1337 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1338 each file that was not fully healthy. 'pathname' is
1339 a list of strings (which can be joined by "/"
1340 characters to turn it into a single string),
1341 relative to the directory on which deep-check was
1342 invoked. The 'check-results' field is the same as
1343 that returned by t=check&output=JSON, described
1345 stats: a dictionary with the same keys as the t=start-deep-stats command
1348 ``POST $URL?t=stream-deep-check``
1350 This initiates a recursive walk of all files and directories reachable from
1351 the target, performing a check on each one just like t=check. For each
1352 unique object (duplicates are skipped), a single line of JSON is emitted to
1353 the HTTP response channel (or an error indication, see below). When the walk
1354 is complete, a final line of JSON is emitted which contains the accumulated
1355 file-size/count "deep-stats" data.
1357 This command takes the same arguments as t=start-deep-check.
1359 A CLI tool can split the response stream on newlines into "response units",
1360 and parse each response unit as JSON. Each such parsed unit will be a
1361 dictionary, and will contain at least the "type" key: a string, one of
1362 "file", "directory", or "stats".
1364 For all units that have a type of "file" or "directory", the dictionary will
1365 contain the following keys::
1367 "path": a list of strings, with the path that is traversed to reach the
1369 "cap": a write-cap URI for the file or directory, if available, else a
1371 "verifycap": a verify-cap URI for the file or directory
1372 "repaircap": an URI for the weakest cap that can still be used to repair
1374 "storage-index": a base32 storage index for the object
1375 "check-results": a copy of the dictionary which would be returned by
1376 t=check&output=json, with three top-level keys:
1377 "storage-index", "summary", and "results", and a variety
1378 of counts and sharemaps in the "results" value.
1380 Note that non-distributed files (i.e. LIT files) will have values of None
1381 for verifycap, repaircap, and storage-index, since these files can neither
1382 be verified nor repaired, and are not stored on the storage servers.
1383 Likewise the check-results dictionary will be limited: an empty string for
1384 storage-index, and a results dictionary with only the "healthy" key.
1386 The last unit in the stream will have a type of "stats", and will contain
1387 the keys described in the "start-deep-stats" operation, below.
1389 If any errors occur during the traversal (specifically if a directory is
1390 unrecoverable, such that further traversal is not possible), an error
1391 indication is written to the response body, instead of the usual line of
1392 JSON. This error indication line will begin with the string "ERROR:" (in all
1393 caps), and contain a summary of the error on the rest of the line. The
1394 remaining lines of the response body will be a python exception. The client
1395 application should look for the ERROR: and stop processing JSON as soon as
1396 it is seen. Note that neither a file being unrecoverable nor a directory
1397 merely being unhealthy will cause traversal to stop. The line just before
1398 the ERROR: will describe the directory that was untraversable, since the
1399 unit is emitted to the HTTP response body before the child is traversed.
1402 ``POST $URL?t=check&repair=true``
1404 This performs a health check of the given file or directory, and if the
1405 checker determines that the object is not healthy (some shares are missing
1406 or corrupted), it will perform a "repair". During repair, any missing
1407 shares will be regenerated and uploaded to new servers.
1409 This accepts the same verify=true and add-lease= arguments as t=check. When
1410 an output=JSON argument is provided, the machine-readable JSON response
1411 will contain the following keys::
1413 storage-index: a base32-encoded string with the objects's storage index,
1414 or an empty string for LIT files
1415 repair-attempted: (bool) True if repair was attempted
1416 repair-successful: (bool) True if repair was attempted and the file was
1417 fully healthy afterwards. False if no repair was
1418 attempted, or if a repair attempt failed.
1419 pre-repair-results: a dictionary that describes the state of the file
1420 before any repair was performed. This contains exactly
1421 the same keys as the 'results' value of the t=check
1422 response, described above.
1423 post-repair-results: a dictionary that describes the state of the file
1424 after any repair was performed. If no repair was
1425 performed, post-repair-results and pre-repair-results
1426 will be the same. This contains exactly the same keys
1427 as the 'results' value of the t=check response,
1430 ``POST $URL?t=start-deep-check&repair=true`` (must add &ophandle=XYZ)
1432 This triggers a recursive walk of all files and directories, performing a
1433 t=check&repair=true on each one.
1435 Like t=start-deep-check without the repair= argument, this can only be
1436 invoked on a directory. An error (400 BAD_REQUEST) will be signalled if it
1437 is invoked on a file. The recursive walker will deal with loops safely.
1439 This accepts the same verify= and add-lease= arguments as
1440 t=start-deep-check. It uses the same ophandle= mechanism as
1441 start-deep-check. When an output=JSON argument is provided, the response
1442 will contain the following keys::
1444 finished: (bool) True if the operation has completed, else False
1445 root-storage-index: a base32-encoded string with the storage index of the
1446 starting point of the deep-check operation
1447 count-objects-checked: count of how many objects were checked
1449 count-objects-healthy-pre-repair: how many of those objects were completely
1450 healthy, before any repair
1451 count-objects-unhealthy-pre-repair: how many were damaged in some way
1452 count-objects-healthy-post-repair: how many of those objects were completely
1453 healthy, after any repair
1454 count-objects-unhealthy-post-repair: how many were damaged in some way
1456 count-repairs-attempted: repairs were attempted on this many objects.
1457 count-repairs-successful: how many repairs resulted in healthy objects
1458 count-repairs-unsuccessful: how many repairs resulted did not results in
1459 completely healthy objects
1460 count-corrupt-shares-pre-repair: how many shares were found to have
1461 corruption, summed over all objects
1462 examined, before any repair
1463 count-corrupt-shares-post-repair: how many shares were found to have
1464 corruption, summed over all objects
1465 examined, after any repair
1466 list-corrupt-shares: a list of "share identifiers", one for each share
1467 that was found to be corrupt (before any repair).
1468 Each share identifier is a list of (serverid,
1469 storage_index, sharenum).
1470 list-remaining-corrupt-shares: like list-corrupt-shares, but mutable shares
1471 that were successfully repaired are not
1472 included. These are shares that need
1473 manual processing. Since immutable shares
1474 cannot be modified by clients, all corruption
1475 in immutable shares will be listed here.
1476 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1477 each file that was not fully healthy. 'pathname' is
1478 relative to the directory on which deep-check was
1479 invoked. The 'check-results' field is the same as
1480 that returned by t=check&repair=true&output=JSON,
1482 stats: a dictionary with the same keys as the t=start-deep-stats command
1485 ``POST $URL?t=stream-deep-check&repair=true``
1487 This triggers a recursive walk of all files and directories, performing a
1488 t=check&repair=true on each one. For each unique object (duplicates are
1489 skipped), a single line of JSON is emitted to the HTTP response channel (or
1490 an error indication). When the walk is complete, a final line of JSON is
1491 emitted which contains the accumulated file-size/count "deep-stats" data.
1493 This emits the same data as t=stream-deep-check (without the repair=true),
1494 except that the "check-results" field is replaced with a
1495 "check-and-repair-results" field, which contains the keys returned by
1496 t=check&repair=true&output=json (i.e. repair-attempted, repair-successful,
1497 pre-repair-results, and post-repair-results). The output does not contain
1498 the summary dictionary that is provied by t=start-deep-check&repair=true
1499 (the one with count-objects-checked and list-unhealthy-files), since the
1500 receiving client is expected to calculate those values itself from the
1501 stream of per-object check-and-repair-results.
1503 Note that the "ERROR:" indication will only be emitted if traversal stops,
1504 which will only occur if an unrecoverable directory is encountered. If a
1505 file or directory repair fails, the traversal will continue, and the repair
1506 failure will be indicated in the JSON data (in the "repair-successful" key).
1508 ``POST $DIRURL?t=start-manifest`` (must add &ophandle=XYZ)
1510 This operation generates a "manfest" of the given directory tree, mostly
1511 for debugging. This is a table of (path, filecap/dircap), for every object
1512 reachable from the starting directory. The path will be slash-joined, and
1513 the filecap/dircap will contain a link to the object in question. This page
1514 gives immediate access to every object in the virtual filesystem subtree.
1516 This operation uses the same ophandle= mechanism as deep-check. The
1517 corresponding /operations/$HANDLE page has three different forms. The
1518 default is output=HTML.
1520 If output=text is added to the query args, the results will be a text/plain
1521 list. The first line is special: it is either "finished: yes" or "finished:
1522 no"; if the operation is not finished, you must periodically reload the
1523 page until it completes. The rest of the results are a plaintext list, with
1524 one file/dir per line, slash-separated, with the filecap/dircap separated
1527 If output=JSON is added to the queryargs, then the results will be a
1528 JSON-formatted dictionary with six keys. Note that because large directory
1529 structures can result in very large JSON results, the full results will not
1530 be available until the operation is complete (i.e. until output["finished"]
1533 finished (bool): if False then you must reload the page until True
1534 origin_si (base32 str): the storage index of the starting point
1535 manifest: list of (path, cap) tuples, where path is a list of strings.
1536 verifycaps: list of (printable) verify cap strings
1537 storage-index: list of (base32) storage index strings
1538 stats: a dictionary with the same keys as the t=start-deep-stats command
1541 ``POST $DIRURL?t=start-deep-size`` (must add &ophandle=XYZ)
1543 This operation generates a number (in bytes) containing the sum of the
1544 filesize of all directories and immutable files reachable from the given
1545 directory. This is a rough lower bound of the total space consumed by this
1546 subtree. It does not include space consumed by mutable files, nor does it
1547 take expansion or encoding overhead into account. Later versions of the
1548 code may improve this estimate upwards.
1550 The /operations/$HANDLE status output consists of two lines of text::
1555 ``POST $DIRURL?t=start-deep-stats`` (must add &ophandle=XYZ)
1557 This operation performs a recursive walk of all files and directories
1558 reachable from the given directory, and generates a collection of
1559 statistics about those objects.
1561 The result (obtained from the /operations/$OPHANDLE page) is a
1562 JSON-serialized dictionary with the following keys (note that some of these
1563 keys may be missing until 'finished' is True)::
1565 finished: (bool) True if the operation has finished, else False
1566 count-immutable-files: count of how many CHK files are in the set
1567 count-mutable-files: same, for mutable files (does not include directories)
1568 count-literal-files: same, for LIT files (data contained inside the URI)
1569 count-files: sum of the above three
1570 count-directories: count of directories
1571 count-unknown: count of unrecognized objects (perhaps from the future)
1572 size-immutable-files: total bytes for all CHK files in the set, =deep-size
1573 size-mutable-files (TODO): same, for current version of all mutable files
1574 size-literal-files: same, for LIT files
1575 size-directories: size of directories (includes size-literal-files)
1576 size-files-histogram: list of (minsize, maxsize, count) buckets,
1577 with a histogram of filesizes, 5dB/bucket,
1578 for both literal and immutable files
1579 largest-directory: number of children in the largest directory
1580 largest-immutable-file: number of bytes in the largest CHK file
1582 size-mutable-files is not implemented, because it would require extra
1583 queries to each mutable file to get their size. This may be implemented in
1586 Assuming no sharing, the basic space consumed by a single root directory is
1587 the sum of size-immutable-files, size-mutable-files, and size-directories.
1588 The actual disk space used by the shares is larger, because of the
1589 following sources of overhead::
1592 expansion due to erasure coding
1593 share management data (leases)
1594 backend (ext3) minimum block size
1596 ``POST $URL?t=stream-manifest``
1598 This operation performs a recursive walk of all files and directories
1599 reachable from the given starting point. For each such unique object
1600 (duplicates are skipped), a single line of JSON is emitted to the HTTP
1601 response channel (or an error indication, see below). When the walk is
1602 complete, a final line of JSON is emitted which contains the accumulated
1603 file-size/count "deep-stats" data.
1605 A CLI tool can split the response stream on newlines into "response units",
1606 and parse each response unit as JSON. Each such parsed unit will be a
1607 dictionary, and will contain at least the "type" key: a string, one of
1608 "file", "directory", or "stats".
1610 For all units that have a type of "file" or "directory", the dictionary will
1611 contain the following keys::
1613 "path": a list of strings, with the path that is traversed to reach the
1615 "cap": a write-cap URI for the file or directory, if available, else a
1617 "verifycap": a verify-cap URI for the file or directory
1618 "repaircap": an URI for the weakest cap that can still be used to repair
1620 "storage-index": a base32 storage index for the object
1622 Note that non-distributed files (i.e. LIT files) will have values of None
1623 for verifycap, repaircap, and storage-index, since these files can neither
1624 be verified nor repaired, and are not stored on the storage servers.
1626 The last unit in the stream will have a type of "stats", and will contain
1627 the keys described in the "start-deep-stats" operation, below.
1629 If any errors occur during the traversal (specifically if a directory is
1630 unrecoverable, such that further traversal is not possible), an error
1631 indication is written to the response body, instead of the usual line of
1632 JSON. This error indication line will begin with the string "ERROR:" (in all
1633 caps), and contain a summary of the error on the rest of the line. The
1634 remaining lines of the response body will be a python exception. The client
1635 application should look for the ERROR: and stop processing JSON as soon as
1636 it is seen. The line just before the ERROR: will describe the directory that
1637 was untraversable, since the manifest entry is emitted to the HTTP response
1638 body before the child is traversed.
1643 The portion of the web namespace that begins with "/uri" (and "/named") is
1644 dedicated to giving users (both humans and programs) access to the Tahoe
1645 virtual filesystem. The rest of the namespace provides status information
1646 about the state of the Tahoe node.
1648 ``GET /`` (the root page)
1650 This is the "Welcome Page", and contains a few distinct sections::
1652 Node information: library versions, local nodeid, services being provided.
1654 Filesystem Access Forms: create a new directory, view a file/directory by
1655 URI, upload a file (unlinked), download a file by
1658 Grid Status: introducer information, helper information, connected storage
1663 This page lists all active uploads and downloads, and contains a short list
1664 of recent upload/download operations. Each operation has a link to a page
1665 that describes file sizes, servers that were involved, and the time consumed
1666 in each phase of the operation.
1668 A GET of /status/?t=json will contain a machine-readable subset of the same
1669 data. It returns a JSON-encoded dictionary. The only key defined at this
1670 time is "active", with a value that is a list of operation dictionaries, one
1671 for each active operation. Once an operation is completed, it will no longer
1672 appear in data["active"] .
1674 Each op-dict contains a "type" key, one of "upload", "download",
1675 "mapupdate", "publish", or "retrieve" (the first two are for immutable
1676 files, while the latter three are for mutable files and directories).
1678 The "upload" op-dict will contain the following keys::
1680 type (string): "upload"
1681 storage-index-string (string): a base32-encoded storage index
1682 total-size (int): total size of the file
1683 status (string): current status of the operation
1684 progress-hash (float): 1.0 when the file has been hashed
1685 progress-ciphertext (float): 1.0 when the file has been encrypted.
1686 progress-encode-push (float): 1.0 when the file has been encoded and
1687 pushed to the storage servers. For helper
1688 uploads, the ciphertext value climbs to 1.0
1689 first, then encoding starts. For unassisted
1690 uploads, ciphertext and encode-push progress
1691 will climb at the same pace.
1693 The "download" op-dict will contain the following keys::
1695 type (string): "download"
1696 storage-index-string (string): a base32-encoded storage index
1697 total-size (int): total size of the file
1698 status (string): current status of the operation
1699 progress (float): 1.0 when the file has been fully downloaded
1701 Front-ends which want to report progress information are advised to simply
1702 average together all the progress-* indicators. A slightly more accurate
1703 value can be found by ignoring the progress-hash value (since the current
1704 implementation hashes synchronously, so clients will probably never see
1705 progress-hash!=1.0).
1707 ``GET /provisioning/``
1709 This page provides a basic tool to predict the likely storage and bandwidth
1710 requirements of a large Tahoe grid. It provides forms to input things like
1711 total number of users, number of files per user, average file size, number
1712 of servers, expansion ratio, hard drive failure rate, etc. It then provides
1713 numbers like how many disks per server will be needed, how many read
1714 operations per second should be expected, and the likely MTBF for files in
1715 the grid. This information is very preliminary, and the model upon which it
1716 is based still needs a lot of work.
1718 ``GET /helper_status/``
1720 If the node is running a helper (i.e. if [helper]enabled is set to True in
1721 tahoe.cfg), then this page will provide a list of all the helper operations
1722 currently in progress. If "?t=json" is added to the URL, it will return a
1723 JSON-formatted list of helper statistics, which can then be used to produce
1724 graphs to indicate how busy the helper is.
1726 ``GET /statistics/``
1728 This page provides "node statistics", which are collected from a variety of
1731 load_monitor: every second, the node schedules a timer for one second in
1732 the future, then measures how late the subsequent callback
1733 is. The "load_average" is this tardiness, measured in
1734 seconds, averaged over the last minute. It is an indication
1735 of a busy node, one which is doing more work than can be
1736 completed in a timely fashion. The "max_load" value is the
1737 highest value that has been seen in the last 60 seconds.
1739 cpu_monitor: every minute, the node uses time.clock() to measure how much
1740 CPU time it has used, and it uses this value to produce
1741 1min/5min/15min moving averages. These values range from 0%
1742 (0.0) to 100% (1.0), and indicate what fraction of the CPU
1743 has been used by the Tahoe node. Not all operating systems
1744 provide meaningful data to time.clock(): they may report 100%
1745 CPU usage at all times.
1747 uploader: this counts how many immutable files (and bytes) have been
1748 uploaded since the node was started
1750 downloader: this counts how many immutable files have been downloaded
1751 since the node was started
1753 publishes: this counts how many mutable files (including directories) have
1754 been modified since the node was started
1756 retrieves: this counts how many mutable files (including directories) have
1757 been read since the node was started
1759 There are other statistics that are tracked by the node. The "raw stats"
1760 section shows a formatted dump of all of them.
1762 By adding "?t=json" to the URL, the node will return a JSON-formatted
1763 dictionary of stats values, which can be used by other tools to produce
1764 graphs of node behavior. The misc/munin/ directory in the source
1765 distribution provides some tools to produce these graphs.
1767 ``GET /`` (introducer status)
1769 For Introducer nodes, the welcome page displays information about both
1770 clients and servers which are connected to the introducer. Servers make
1771 "service announcements", and these are listed in a table. Clients will
1772 subscribe to hear about service announcements, and these subscriptions are
1773 listed in a separate table. Both tables contain information about what
1774 version of Tahoe is being run by the remote node, their advertised and
1775 outbound IP addresses, their nodeid and nickname, and how long they have
1778 By adding "?t=json" to the URL, the node will return a JSON-formatted
1779 dictionary of stats values, which can be used to produce graphs of connected
1780 clients over time. This dictionary has the following keys::
1782 ["subscription_summary"] : a dictionary mapping service name (like
1783 "storage") to an integer with the number of
1784 clients that have subscribed to hear about that
1786 ["announcement_summary"] : a dictionary mapping service name to an integer
1787 with the number of servers which are announcing
1789 ["announcement_distinct_hosts"] : a dictionary mapping service name to an
1790 integer which represents the number of
1791 distinct hosts that are providing that
1792 service. If two servers have announced
1793 FURLs which use the same hostnames (but
1794 different ports and tubids), they are
1795 considered to be on the same host.
1798 Static Files in /public_html
1799 ============================
1801 The webapi server will take any request for a URL that starts with /static
1802 and serve it from a configurable directory which defaults to
1803 $BASEDIR/public_html . This is configured by setting the "[node]web.static"
1804 value in $BASEDIR/tahoe.cfg . If this is left at the default value of
1805 "public_html", then http://localhost:3456/static/subdir/foo.html will be
1806 served with the contents of the file $BASEDIR/public_html/subdir/foo.html .
1808 This can be useful to serve a javascript application which provides a
1809 prettier front-end to the rest of the Tahoe webapi.
1812 Safety and security issues -- names vs. URIs
1813 ============================================
1815 Summary: use explicit file- and dir- caps whenever possible, to reduce the
1816 potential for surprises when the filesystem structure is changed.
1818 Tahoe provides a mutable filesystem, but the ways that the filesystem can
1819 change are limited. The only thing that can change is that the mapping from
1820 child names to child objects that each directory contains can be changed by
1821 adding a new child name pointing to an object, removing an existing child name,
1822 or changing an existing child name to point to a different object.
1824 Obviously if you query Tahoe for information about the filesystem and then act
1825 to change the filesystem (such as by getting a listing of the contents of a
1826 directory and then adding a file to the directory), then the filesystem might
1827 have been changed after you queried it and before you acted upon it. However,
1828 if you use the URI instead of the pathname of an object when you act upon the
1829 object, then the only change that can happen is if the object is a directory
1830 then the set of child names it has might be different. If, on the other hand,
1831 you act upon the object using its pathname, then a different object might be in
1832 that place, which can result in more kinds of surprises.
1834 For example, suppose you are writing code which recursively downloads the
1835 contents of a directory. The first thing your code does is fetch the listing
1836 of the contents of the directory. For each child that it fetched, if that
1837 child is a file then it downloads the file, and if that child is a directory
1838 then it recurses into that directory. Now, if the download and the recurse
1839 actions are performed using the child's name, then the results might be
1840 wrong, because for example a child name that pointed to a sub-directory when
1841 you listed the directory might have been changed to point to a file (in which
1842 case your attempt to recurse into it would result in an error and the file
1843 would be skipped), or a child name that pointed to a file when you listed the
1844 directory might now point to a sub-directory (in which case your attempt to
1845 download the child would result in a file containing HTML text describing the
1848 If your recursive algorithm uses the uri of the child instead of the name of
1849 the child, then those kinds of mistakes just can't happen. Note that both the
1850 child's name and the child's URI are included in the results of listing the
1851 parent directory, so it isn't any harder to use the URI for this purpose.
1853 The read and write caps in a given directory node are separate URIs, and
1854 can't be assumed to point to the same object even if they were retrieved in
1855 the same operation (although the webapi server attempts to ensure this
1856 in most cases). If you need to rely on that property, you should explicitly
1857 verify it. More generally, you should not make assumptions about the
1858 internal consistency of the contents of mutable directories. As a result
1859 of the signatures on mutable object versions, it is guaranteed that a given
1860 version was written in a single update, but -- as in the case of a file --
1861 the contents may have been chosen by a malicious writer in a way that is
1862 designed to confuse applications that rely on their consistency.
1864 In general, use names if you want "whatever object (whether file or
1865 directory) is found by following this name (or sequence of names) when my
1866 request reaches the server". Use URIs if you want "this particular object".
1871 Tahoe uses both mutable and immutable files. Mutable files can be created
1872 explicitly by doing an upload with ?mutable=true added, or implicitly by
1873 creating a new directory (since a directory is just a special way to
1874 interpret a given mutable file).
1876 Mutable files suffer from the same consistency-vs-availability tradeoff that
1877 all distributed data storage systems face. It is not possible to
1878 simultaneously achieve perfect consistency and perfect availability in the
1879 face of network partitions (servers being unreachable or faulty).
1881 Tahoe tries to achieve a reasonable compromise, but there is a basic rule in
1882 place, known as the Prime Coordination Directive: "Don't Do That". What this
1883 means is that if write-access to a mutable file is available to several
1884 parties, then those parties are responsible for coordinating their activities
1885 to avoid multiple simultaneous updates. This could be achieved by having
1886 these parties talk to each other and using some sort of locking mechanism, or
1887 by serializing all changes through a single writer.
1889 The consequences of performing uncoordinated writes can vary. Some of the
1890 writers may lose their changes, as somebody else wins the race condition. In
1891 many cases the file will be left in an "unhealthy" state, meaning that there
1892 are not as many redundant shares as we would like (reducing the reliability
1893 of the file against server failures). In the worst case, the file can be left
1894 in such an unhealthy state that no version is recoverable, even the old ones.
1895 It is this small possibility of data loss that prompts us to issue the Prime
1896 Coordination Directive.
1898 Tahoe nodes implement internal serialization to make sure that a single Tahoe
1899 node cannot conflict with itself. For example, it is safe to issue two
1900 directory modification requests to a single tahoe node's webapi server at the
1901 same time, because the Tahoe node will internally delay one of them until
1902 after the other has finished being applied. (This feature was introduced in
1903 Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing
1904 web requests themselves).
1906 For more details, please see the "Consistency vs Availability" and "The Prime
1907 Coordination Directive" sections of `mutable.rst <../specifications/mutable.rst>`_.
1910 .. [1] URLs and HTTP and UTF-8, Oh My
1912 HTTP does not provide a mechanism to specify the character set used to
1913 encode non-ascii names in URLs (rfc2396#2.1). We prefer the convention that
1914 the filename= argument shall be a URL-encoded UTF-8 encoded unicode object.
1915 For example, suppose we want to provoke the server into using a filename of
1916 "f i a n c e-acute e" (i.e. F I A N C U+00E9 E). The UTF-8 encoding of this
1917 is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\xC3\xA9e", as python's
1918 repr() function would show). To encode this into a URL, the non-printable
1919 characters must be escaped with the urlencode '%XX' mechansim, giving us
1920 "fianc%C3%A9e". Thus, the first line of the HTTP request will be "GET
1921 /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1". Not all browsers
1922 provide this: IE7 uses the Latin-1 encoding, which is fianc%E9e.
1924 The response header will need to indicate a non-ASCII filename. The actual
1925 mechanism to do this is not clear. For ASCII filenames, the response header
1928 Content-Disposition: attachment; filename="english.txt"
1930 If Tahoe were to enforce the utf-8 convention, it would need to decode the
1931 URL argument into a unicode string, and then encode it back into a sequence
1932 of bytes when creating the response header. One possibility would be to use
1933 unencoded utf-8. Developers suggest that IE7 might accept this::
1935 #1: Content-Disposition: attachment; filename="fianc\xC3\xA9e"
1936 (note, the last four bytes of that line, not including the newline, are
1937 0xC3 0xA9 0x65 0x22)
1939 `RFC2231#4 <http://tools.ietf.org/html/rfc2231#section-4>`_
1940 (dated 1997): suggests that the following might work, and
1941 `some developers have reported <http://markmail.org/message/dsjyokgl7hv64ig3>`_
1942 that it is supported by firefox (but not IE7)::
1944 #2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
1946 My reading of `RFC2616#19.5.1 <http://tools.ietf.org/html/rfc2616#section-19.5.1>`_
1947 (which defines Content-Disposition) says that the filename= parameter is
1948 defined to be wrapped in quotes (presumably to allow spaces without breaking
1949 the parsing of subsequent parameters), which would give us::
1951 #3: Content-Disposition: attachment; filename*=utf-8''"fianc%C3%A9e"
1953 However this is contrary to the examples in the email thread listed above.
1955 Developers report that IE7 (when it is configured for UTF-8 URL encoding,
1956 which is not the default in asian countries), will accept::
1958 #4: Content-Disposition: attachment; filename=fianc%C3%A9e
1960 However, for maximum compatibility, Tahoe simply copies bytes from the URL
1961 into the response header, rather than enforcing the utf-8 convention. This
1962 means it does not try to decode the filename from the URL argument, nor does
1963 it encode the filename into the response header.