2 = The Tahoe REST-ful Web API =
4 1. Enabling the web-API port
5 2. Basic Concepts: GET, PUT, DELETE, POST
6 3. URLs, Machine-Oriented Interfaces
7 4. Browser Operations: Human-Oriented Interfaces
8 5. Welcome / Debug / Status pages
9 6. Static Files in /public_html
10 7. Safety and security issues -- names vs. URIs
14 == Enabling the web-API port ==
16 Every Tahoe node is capable of running a built-in HTTP server. To enable
17 this, just write a port number into the "[node]web.port" line of your node's
18 tahoe.cfg file. For example, writing "web.port = 3456" into the "[node]"
19 section of $NODEDIR/tahoe.cfg will cause the node to run a webserver on port
22 This string is actually a Twisted "strports" specification, meaning you can
23 get more control over the interface to which the server binds by supplying
24 additional arguments. For more details, see the documentation on
25 twisted.application.strports:
26 http://twistedmatrix.com/documents/current/api/twisted.application.strports.html
28 Writing "tcp:3456:interface=127.0.0.1" into the web.port line does the same
29 but binds to the loopback interface, ensuring that only the programs on the
30 local host can connect. Using
31 "ssl:3456:privateKey=mykey.pem:certKey=cert.pem" runs an SSL server.
33 This webport can be set when the node is created by passing a --webport
34 option to the 'tahoe create-node' command. By default, the node listens on
35 port 3456, on the loopback (127.0.0.1) interface.
39 As described in architecture.txt, each file and directory in a Tahoe virtual
40 filesystem is referenced by an identifier that combines the designation of
41 the object with the authority to do something with it (such as read or modify
42 the contents). This identifier is called a "read-cap" or "write-cap",
43 depending upon whether it enables read-only or read-write access. These
44 "caps" are also referred to as URIs.
46 The Tahoe web-based API is "REST-ful", meaning it implements the concepts of
47 "REpresentational State Transfer": the original scheme by which the World
48 Wide Web was intended to work. Each object (file or directory) is referenced
49 by a URL that includes the read- or write- cap. HTTP methods (GET, PUT, and
50 DELETE) are used to manipulate these objects. You can think of the URL as a
51 noun, and the method as a verb.
53 In REST, the GET method is used to retrieve information about an object, or
54 to retrieve some representation of the object itself. When the object is a
55 file, the basic GET method will simply return the contents of that file.
56 Other variations (generally implemented by adding query parameters to the
57 URL) will return information about the object, such as metadata. GET
58 operations are required to have no side-effects.
60 PUT is used to upload new objects into the filesystem, or to replace an
61 existing object. DELETE it used to delete objects from the filesystem. Both
62 PUT and DELETE are required to be idempotent: performing the same operation
63 multiple times must have the same side-effects as only performing it once.
65 POST is used for more complicated actions that cannot be expressed as a GET,
66 PUT, or DELETE. POST operations can be thought of as a method call: sending
67 some message to the object referenced by the URL. In Tahoe, POST is also used
68 for operations that must be triggered by an HTML form (including upload and
69 delete), because otherwise a regular web browser has no way to accomplish
70 these tasks. In general, everything that can be done with a PUT or DELETE can
71 also be done with a POST.
73 Tahoe's web API is designed for two different kinds of consumer. The first is
74 a program that needs to manipulate the virtual file system. Such programs are
75 expected to use the RESTful interface described above. The second is a human
76 using a standard web browser to work with the filesystem. This user is given
77 a series of HTML pages with links to download files, and forms that use POST
78 actions to upload, rename, and delete files.
80 When an error occurs, the HTTP response code will be set to an appropriate
81 400-series code (like 404 Not Found for an unknown childname, or 400 Bad Request
82 when the parameters to a webapi operation are invalid), and the HTTP response
83 body will usually contain a few lines of explanation as to the cause of the
84 error and possible responses. Unusual exceptions may result in a
85 500 Internal Server Error as a catch-all, with a default response body containing
86 a Nevow-generated HTML-ized representation of the Python exception stack trace
87 that caused the problem. CLI programs which want to copy the response body to
88 stderr should provide an "Accept: text/plain" header to their requests to get
89 a plain text stack trace instead. If the Accept header contains */*, or
90 text/*, or text/html (or if there is no Accept header), HTML tracebacks will
95 Tahoe uses a variety of read- and write- caps to identify files and
96 directories. The most common of these is the "immutable file read-cap", which
97 is used for most uploaded files. These read-caps look like the following:
99 URI:CHK:ime6pvkaxuetdfah2p2f35pe54:4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a:3:10:202
101 The next most common is a "directory write-cap", which provides both read and
102 write access to a directory, and look like this:
104 URI:DIR2:djrdkfawoqihigoett4g6auz6a:jx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq
106 There are also "directory read-caps", which start with "URI:DIR2-RO:", and
107 give read-only access to a directory. Finally there are also mutable file
108 read- and write- caps, which start with "URI:SSK", and give access to mutable
111 (Later versions of Tahoe will make these strings shorter, and will remove the
112 unfortunate colons, which must be escaped when these caps are embedded in
115 To refer to any Tahoe object through the web API, you simply need to combine
116 a prefix (which indicates the HTTP server to use) with the cap (which
117 indicates which object inside that server to access). Since the default Tahoe
118 webport is 3456, the most common prefix is one that will use a local node
119 listening on this port:
121 http://127.0.0.1:3456/uri/ + $CAP
123 So, to access the directory named above (which happens to be the
124 publically-writeable sample directory on the Tahoe test grid, described at
125 http://allmydata.org/trac/tahoe/wiki/TestGrid), the URL would be:
127 http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/
129 (note that the colons in the directory-cap are url-encoded into "%3A"
132 Likewise, to access the file named above, use:
134 http://127.0.0.1:3456/uri/URI%3ACHK%3Aime6pvkaxuetdfah2p2f35pe54%3A4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a%3A3%3A10%3A202
136 In the rest of this document, we'll use "$DIRCAP" as shorthand for a read-cap
137 or write-cap that refers to a directory, and "$FILECAP" to abbreviate a cap
138 that refers to a file (whether mutable or immutable). So those URLs above can
141 http://127.0.0.1:3456/uri/$DIRCAP/
142 http://127.0.0.1:3456/uri/$FILECAP
144 The operation summaries below will abbreviate these further, by eliding the
145 server prefix. They will be displayed like this:
153 Tahoe directories contain named child entries, just like directories in a regular
154 local filesystem. These child entries, called "dirnodes", consist of a name,
155 metadata, a write slot, and a read slot. The write and read slots normally contain
156 a write-cap and read-cap referring to the same object, which can be either a file
157 or a subdirectory. The write slot may be empty (actually, both may be empty,
158 but that is unusual).
160 If you have a Tahoe URL that refers to a directory, and want to reference a
161 named child inside it, just append the child name to the URL. For example, if
162 our sample directory contains a file named "welcome.txt", we can refer to
165 http://127.0.0.1:3456/uri/$DIRCAP/welcome.txt
167 (or http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/welcome.txt)
169 Multiple levels of subdirectories can be handled this way:
171 http://127.0.0.1:3456/uri/$DIRCAP/tahoe-source/docs/webapi.txt
173 In this document, when we need to refer to a URL that references a file using
174 this child-of-some-directory format, we'll use the following string:
176 /uri/$DIRCAP/[SUBDIRS../]FILENAME
178 The "[SUBDIRS../]" part means that there are zero or more (optional)
179 subdirectory names in the middle of the URL. The "FILENAME" at the end means
180 that this whole URL refers to a file of some sort, rather than to a
183 When we need to refer specifically to a directory in this way, we'll write:
185 /uri/$DIRCAP/[SUBDIRS../]SUBDIR
188 Note that all components of pathnames in URLs are required to be UTF-8
189 encoded, so "resume.doc" (with an acute accent on both E's) would be accessed
192 http://127.0.0.1:3456/uri/$DIRCAP/r%C3%A9sum%C3%A9.doc
194 Also note that the filenames inside upload POST forms are interpreted using
195 whatever character set was provided in the conventional '_charset' field, and
196 defaults to UTF-8 if not otherwise specified. The JSON representation of each
197 directory contains native unicode strings. Tahoe directories are specified to
198 contain unicode filenames, and cannot contain binary strings that are not
199 representable as such.
201 All Tahoe operations that refer to existing files or directories must include
202 a suitable read- or write- cap in the URL: the webapi server won't add one
203 for you. If you don't know the cap, you can't access the file. This allows
204 the security properties of Tahoe caps to be extended across the webapi
207 == Slow Operations, Progress, and Cancelling ==
209 Certain operations can be expected to take a long time. The "t=deep-check",
210 described below, will recursively visit every file and directory reachable
211 from a given starting point, which can take minutes or even hours for
212 extremely large directory structures. A single long-running HTTP request is a
213 fragile thing: proxies, NAT boxes, browsers, and users may all grow impatient
214 with waiting and give up on the connection.
216 For this reason, long-running operations have an "operation handle", which
217 can be used to poll for status/progress messages while the operation
218 proceeds. This handle can also be used to cancel the operation. These handles
219 are created by the client, and passed in as a an "ophandle=" query argument
220 to the POST or PUT request which starts the operation. The following
221 operations can then be used to retrieve status:
223 GET /operations/$HANDLE?output=HTML (with or without t=status)
224 GET /operations/$HANDLE?output=JSON (same)
226 These two retrieve the current status of the given operation. Each operation
227 presents a different sort of information, but in general the page retrieved
230 * whether the operation is complete, or if it is still running
231 * how much of the operation is complete, and how much is left, if possible
233 Note that the final status output can be quite large: a deep-manifest of a
234 directory structure with 300k directories and 200k unique files is about
235 275MB of JSON, and might take two minutes to generate. For this reason, the
236 full status is not provided until the operation has completed.
238 The HTML form will include a meta-refresh tag, which will cause a regular
239 web browser to reload the status page about 60 seconds later. This tag will
240 be removed once the operation has completed.
242 There may be more status information available under
243 /operations/$HANDLE/$ETC : i.e., the handle forms the root of a URL space.
245 POST /operations/$HANDLE?t=cancel
247 This terminates the operation, and returns an HTML page explaining what was
248 cancelled. If the operation handle has already expired (see below), this
249 POST will return a 404, which indicates that the operation is no longer
250 running (either it was completed or terminated). The response body will be
251 the same as a GET /operations/$HANDLE on this operation handle, and the
252 handle will be expired immediately afterwards.
254 The operation handle will eventually expire, to avoid consuming an unbounded
255 amount of memory. The handle's time-to-live can be reset at any time, by
256 passing a retain-for= argument (with a count of seconds) to either the
257 initial POST that starts the operation, or the subsequent GET request which
258 asks about the operation. For example, if a 'GET
259 /operations/$HANDLE?output=JSON&retain-for=600' query is performed, the
260 handle will remain active for 600 seconds (10 minutes) after the GET was
263 In addition, if the GET includes a release-after-complete=True argument, and
264 the operation has completed, the operation handle will be released
267 If a retain-for= argument is not used, the default handle lifetimes are:
269 * handles will remain valid at least until their operation finishes
270 * uncollected handles for finished operations (i.e. handles for
271 operations that have finished but for which the GET page has not been
272 accessed since completion) will remain valid for four days, or for
273 the total time consumed by the operation, whichever is greater.
274 * collected handles (i.e. the GET page has been retrieved at least once
275 since the operation completed) will remain valid for one day.
277 Many "slow" operations can begin to use unacceptable amounts of memory when
278 operating on large directory structures. The memory usage increases when the
279 ophandle is polled, as the results must be copied into a JSON string, sent
280 over the wire, then parsed by a client. So, as an alternative, many "slow"
281 operations have streaming equivalents. These equivalents do not use operation
282 handles. Instead, they emit line-oriented status results immediately. Client
283 code can cancel the operation by simply closing the HTTP connection.
285 == Programmatic Operations ==
287 Now that we know how to build URLs that refer to files and directories in a
288 Tahoe virtual filesystem, what sorts of operations can we do with those URLs?
289 This section contains a catalog of GET, PUT, DELETE, and POST operations that
290 can be performed on these URLs. This set of operations are aimed at programs
291 that use HTTP to communicate with a Tahoe node. A later section describes
292 operations that are intended for web browsers.
294 === Reading A File ===
297 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME
299 This will retrieve the contents of the given file. The HTTP response body
300 will contain the sequence of bytes that make up the file.
302 To view files in a web browser, you may want more control over the
303 Content-Type and Content-Disposition headers. Please see the next section
304 "Browser Operations", for details on how to modify these URLs for that
307 === Writing/Uploading A File ===
310 PUT /uri/$DIRCAP/[SUBDIRS../]FILENAME
312 Upload a file, using the data from the HTTP request body, and add whatever
313 child links and subdirectories are necessary to make the file available at
314 the given location. Once this operation succeeds, a GET on the same URL will
315 retrieve the same contents that were just uploaded. This will create any
316 necessary intermediate subdirectories.
318 To use the /uri/$FILECAP form, $FILECAP must be a write-cap for a mutable file.
320 In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a
321 writeable mutable file, that file's contents will be overwritten in-place. If
322 it is a read-cap for a mutable file, an error will occur. If it is an
323 immutable file, the old file will be discarded, and a new one will be put in
326 When creating a new file, if "mutable=true" is in the query arguments, the
327 operation will create a mutable file instead of an immutable one.
329 This returns the file-cap of the resulting file. If a new file was created
330 by this method, the HTTP response code (as dictated by rfc2616) will be set
331 to 201 CREATED. If an existing file was replaced or modified, the response
334 Note that the 'curl -T localfile http://127.0.0.1:3456/uri/$DIRCAP/foo.txt'
335 command can be used to invoke this operation.
339 This uploads a file, and produces a file-cap for the contents, but does not
340 attach the file into the filesystem. No directories will be modified by
341 this operation. The file-cap is returned as the body of the HTTP response.
343 If "mutable=true" is in the query arguments, the operation will create a
344 mutable file, and return its write-cap in the HTTP respose. The default is
345 to create an immutable file, returning the read-cap as a response.
347 === Creating A New Directory ===
352 Create a new empty directory and return its write-cap as the HTTP response
353 body. This does not make the newly created directory visible from the
354 filesystem. The "PUT" operation is provided for backwards compatibility:
355 new code should use POST.
357 POST /uri?t=mkdir-with-children
359 Create a new directory, populated with a set of child nodes, and return its
360 write-cap as the HTTP response body. The new directory is not attached to
361 any other directory: the returned write-cap is the only reference to it.
363 Initial children are provided as the body of the POST form (this is more
364 efficient than doing separate mkdir and set_children operations). If the
365 body is empty, the new directory will be empty. If not empty, the body will
366 be interpreted as a UTF-8 JSON-encoded dictionary of children with which the
367 new directory should be populated, using the same format as would be
368 returned in the 'children' value of the t=json GET request, described below.
369 Each dictionary key should be a child name, and each value should be a list
370 of [TYPE, PROPDICT], where PROPDICT contains "rw_uri", "ro_uri", and
371 "metadata" keys (all others are ignored). For example, the PUT request body
375 "Fran\u00e7ais": [ "filenode", {
376 "ro_uri": "URI:CHK:...",
379 "ctime": 1202777696.7564139,
380 "mtime": 1202777696.7564139,
382 "linkcrtime": 1202777696.7564139,
383 "linkmotime": 1202777696.7564139
385 "subdir": [ "dirnode", {
386 "rw_uri": "URI:DIR2:...",
387 "ro_uri": "URI:DIR2-RO:...",
389 "ctime": 1202778102.7589991,
390 "mtime": 1202778111.2160511,
392 "linkcrtime": 1202777696.7564139,
393 "linkmotime": 1202777696.7564139
397 For forward-compatibility, a mutable directory can also contain caps in
398 a format that is unknown to the webapi server. When such caps are retrieved
399 from a mutable directory in a "ro_uri" field, they will be prefixed with
400 the string "ro.", indicating that they must not be decoded without
401 checking that they are read-only. The "ro." prefix must not be stripped
402 off without performing this check. (Future versions of the webapi server
403 will perform it where necessary.)
405 If both the "rw_uri" and "ro_uri" fields are present in a given PROPDICT,
406 and the webapi server recognizes the rw_uri as a write cap, then it will
407 reset the ro_uri to the corresponding read cap and discard the original
408 contents of ro_uri (in order to ensure that the two caps correspond to the
409 same object and that the ro_uri is in fact read-only). However this may not
410 happen for caps in a format unknown to the webapi server. Therefore, when
411 writing a directory the webapi client should ensure that the contents
412 of "rw_uri" and "ro_uri" for a given PROPDICT are a consistent
413 (write cap, read cap) pair if possible. If the webapi client only has
414 one cap and does not know whether it is a write cap or read cap, then
415 it is acceptable to set "rw_uri" to that cap and omit "ro_uri". The
416 client must not put a write cap into a "ro_uri" field.
418 The metadata may have a "no-write" field. If this is set to true in the
419 metadata of a link, it will not be possible to open that link for writing
420 via the SFTP frontend; see docs/frontends/FTP-and-SFTP.txt for details.
421 Also, if the "no-write" field is set to true in the metadata of a link to
422 a mutable child, it will cause the link to be diminished to read-only.
424 Note that the webapi-using client application must not provide the
425 "Content-Type: multipart/form-data" header that usually accompanies HTML
426 form submissions, since the body is not formatted this way. Doing so will
427 cause a server error as the lower-level code misparses the request body.
429 Child file names should each be expressed as a unicode string, then used as
430 keys of the dictionary. The dictionary should then be converted into JSON,
431 and the resulting string encoded into UTF-8. This UTF-8 bytestring should
432 then be used as the POST body.
434 POST /uri?t=mkdir-immutable
436 Like t=mkdir-with-children above, but the new directory will be
437 deep-immutable. This means that the directory itself is immutable, and that
438 it can only contain objects that are treated as being deep-immutable, like
439 immutable files, literal files, and deep-immutable directories.
441 For forward-compatibility, a deep-immutable directory can also contain caps
442 in a format that is unknown to the webapi server. When such caps are retrieved
443 from a deep-immutable directory in a "ro_uri" field, they will be prefixed
444 with the string "imm.", indicating that they must not be decoded without
445 checking that they are immutable. The "imm." prefix must not be stripped
446 off without performing this check. (Future versions of the webapi server
447 will perform it where necessary.)
449 The cap for each child may be given either in the "rw_uri" or "ro_uri"
450 field of the PROPDICT (not both). If a cap is given in the "rw_uri" field,
451 then the webapi server will check that it is an immutable read-cap of a
452 *known* format, and give an error if it is not. If a cap is given in the
453 "ro_uri" field, then the webapi server will still check whether known
454 caps are immutable, but for unknown caps it will simply assume that the
455 cap can be stored, as described above. Note that an attacker would be
456 able to store any cap in an immutable directory, so this check when
457 creating the directory is only to help non-malicious clients to avoid
458 accidentally giving away more authority than intended.
460 A non-empty request body is mandatory, since after the directory is created,
461 it will not be possible to add more children to it.
463 POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir
464 PUT /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir
466 Create new directories as necessary to make sure that the named target
467 ($DIRCAP/SUBDIRS../SUBDIR) is a directory. This will create additional
468 intermediate mutable directories as necessary. If the named target directory
469 already exists, this will make no changes to it.
471 If the final directory is created, it will be empty.
473 This operation will return an error if a blocking file is present at any of
474 the parent names, preventing the server from creating the necessary parent
475 directory; or if it would require changing an immutable directory.
477 The write-cap of the new directory will be returned as the HTTP response
480 POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-with-children
482 Like /uri?t=mkdir-with-children, but the final directory is created as a
483 child of an existing mutable directory. This will create additional
484 intermediate mutable directories as necessary. If the final directory is
485 created, it will be populated with initial children from the POST request
486 body, as described above.
488 This operation will return an error if a blocking file is present at any of
489 the parent names, preventing the server from creating the necessary parent
490 directory; or if it would require changing an immutable directory; or if
491 the immediate parent directory already has a a child named SUBDIR.
493 POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir-immutable
495 Like /uri?t=mkdir-immutable, but the final directory is created as a child
496 of an existing mutable directory. The final directory will be deep-immutable,
497 and will be populated with the children specified as a JSON dictionary in
498 the POST request body.
500 In Tahoe 1.6 this operation creates intermediate mutable directories if
501 necessary, but that behaviour should not be relied on; see ticket #920.
503 This operation will return an error if the parent directory is immutable,
504 or already has a child named SUBDIR.
506 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME
508 Create a new empty mutable directory and attach it to the given existing
509 directory. This will create additional intermediate directories as necessary.
511 This operation will return an error if a blocking file is present at any of
512 the parent names, preventing the server from creating the necessary parent
513 directory, or if it would require changing any immutable directory.
515 The URL of this operation points to the parent of the bottommost new directory,
516 whereas the /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir operation above has a URL
517 that points directly to the bottommost new directory.
519 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME
521 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME, but the new directory will
522 be populated with initial children via the POST request body. This command
523 will create additional intermediate mutable directories as necessary.
525 This operation will return an error if a blocking file is present at any of
526 the parent names, preventing the server from creating the necessary parent
527 directory; or if it would require changing an immutable directory; or if
528 the immediate parent directory already has a a child named NAME.
530 Note that the name= argument must be passed as a queryarg, because the POST
531 request body is used for the initial children JSON.
533 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-immutable&name=NAME
535 Like /uri/$DIRCAP/[SUBDIRS../]?t=mkdir-with-children&name=NAME, but the
536 final directory will be deep-immutable. The children are specified as a
537 JSON dictionary in the POST request body. Again, the name= argument must be
538 passed as a queryarg.
540 In Tahoe 1.6 this operation creates intermediate mutable directories if
541 necessary, but that behaviour should not be relied on; see ticket #920.
543 This operation will return an error if the parent directory is immutable,
544 or already has a child named NAME.
546 === Get Information About A File Or Directory (as JSON) ===
548 GET /uri/$FILECAP?t=json
549 GET /uri/$DIRCAP?t=json
550 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json
551 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json
553 This returns a machine-parseable JSON-encoded description of the given
554 object. The JSON always contains a list, and the first element of the list is
555 always a flag that indicates whether the referenced object is a file or a
556 directory. If it is a capability to a file, then the information includes
557 file size and URI, like this:
559 GET /uri/$FILECAP?t=json :
563 "verify_uri": verify_uri,
568 If it is a capability to a directory followed by a path from that directory
569 to a file, then the information also includes metadata from the link to the
570 file in the parent directory, like this:
572 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json :
576 "verify_uri": verify_uri,
580 "ctime": 1202777696.7564139,
581 "mtime": 1202777696.7564139,
583 "linkcrtime": 1202777696.7564139,
584 "linkmotime": 1202777696.7564139
587 If it is a directory, then it includes information about the children of
588 this directory, as a mapping from child name to a set of data about the
589 child (the same data that would appear in a corresponding GET?t=json of the
590 child itself). The child entries also include metadata about each child,
591 including link-creation- and link-change- timestamps. The output looks like
594 GET /uri/$DIRCAP?t=json :
595 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json :
598 "rw_uri": read_write_uri,
599 "ro_uri": read_only_uri,
600 "verify_uri": verify_uri,
603 "foo.txt": [ "filenode", {
607 "ctime": 1202777696.7564139,
608 "mtime": 1202777696.7564139,
610 "linkcrtime": 1202777696.7564139,
611 "linkmotime": 1202777696.7564139
613 "subdir": [ "dirnode", {
617 "ctime": 1202778102.7589991,
618 "mtime": 1202778111.2160511,
620 "linkcrtime": 1202777696.7564139,
621 "linkmotime": 1202777696.7564139
625 In the above example, note how 'children' is a dictionary in which the keys
626 are child names and the values depend upon whether the child is a file or a
627 directory. The value is mostly the same as the JSON representation of the
628 child object (except that directories do not recurse -- the "children"
629 entry of the child is omitted, and the directory view includes the metadata
630 that is stored on the directory edge).
632 The rw_uri field will be present in the information about a directory
633 if and only if you have read-write access to that directory. The verify_uri
634 field will be present if and only if the object has a verify-cap
635 (non-distributed LIT files do not have verify-caps).
637 If the cap is of an unknown format, then the file size and verify_uri will
640 GET /uri/$UNKNOWNCAP?t=json :
643 "ro_uri": unknown_read_uri
646 GET /uri/$DIRCAP/[SUBDIRS../]UNKNOWNCHILDNAME?t=json :
649 "rw_uri": unknown_write_uri,
650 "ro_uri": unknown_read_uri,
653 "ctime": 1202777696.7564139,
654 "mtime": 1202777696.7564139,
656 "linkcrtime": 1202777696.7564139,
657 "linkmotime": 1202777696.7564139
660 As in the case of file nodes, the metadata will only be present when the
661 capability is to a directory followed by a path. The "mutable" field is also
662 not always present; when it is absent, the mutability of the object is not
665 ==== About the metadata ====
667 The value of the 'mtime' key and of the 'tahoe':'linkmotime' is updated
668 whenever a link to a child is set. The value of the 'ctime' key and of the
669 'tahoe':'linkcrtime' key is updated whenever a link to a child is created --
670 i.e. when there was not previously a link under that name.
672 In Tahoe earlier than v1.4.0, only the 'mtime'/'ctime' keys were populated.
673 Starting in Tahoe v1.4.0, the 'linkmotime'/'linkcrtime' keys in the 'tahoe'
674 sub-dict are also populated. However, prior to v1.7.0, a bug caused the
675 'tahoe' sub-dict to be deleted by webapi requests in which new metadata
676 is specified, and not to be added to existing child links that lack it.
678 The reason we added the new values in Tahoe v1.4.0 is that there is a
679 "set_children" API (described below) which you can use to overwrite the
680 values of the 'mtime'/'ctime' pair, and this API is used by the "tahoe
681 backup" command (both in Tahoe v1.3.0 and in Tahoe v1.4.0) to set the
682 'mtime' and 'ctime' values when backing up files from a local filesystem
683 into the Tahoe filesystem. As of Tahoe v1.4.0, the set_children API cannot
684 be used to set anything under the 'tahoe' key of the metadata dict -- if
685 you include 'tahoe' keys in your 'metadata' arguments then it will silently
688 Therefore, if the 'tahoe' sub-dict is present, you can rely on the
689 'linkcrtime' and 'linkmotime' values therein to have the semantics described
690 above. (This is assuming that only official Tahoe clients have been used to
691 write those links, and that their system clocks were set to what you expected
692 -- there is nothing preventing someone from editing their Tahoe client or
693 writing their own Tahoe client which would overwrite those values however
694 they like, and there is nothing to constrain their system clock from taking
697 The meaning of the 'ctime'/'mtime' fields are slightly more complex.
699 The meaning of the 'mtime' field is: whenever the edge is updated (by an HTTP
700 PUT or POST, as is done by the "tahoe cp" command), then the mtime is set to
701 the current time on the clock of the updating client. Whenever the edge is
702 updated by "tahoe backup" then the mtime is instead set to the value which
703 the updating client read from its local filesystem for the "mtime" of the
704 local file in question, which means the last time the contents of that file
705 were changed. Note however, that if the edge in the Tahoe filesystem points
706 to a mutable file and the contents of that mutable file is changed then the
707 "mtime" value on that edge will *not* be updated, since the edge itself
708 wasn't updated -- only the mutable file was.
710 The meaning of the 'ctime' field is even more complex. Whenever a new edge is
711 created (by an HTTP PUT or POST, as is done by "tahoe cp") then the ctime is
712 set to the current time on the clock of the updating client. Whenever the
713 edge is created *or updated* by "tahoe backup" then the ctime is instead set
714 to the value which the updating client read from its local filesystem. On
715 Windows, it reads the timestamp of when the local file was created and puts
716 that into the "ctime", and on other platforms it reads the timestamp of the
717 most recent time that either the contents or the metadata of the local file
718 was changed and puts that into the ctime. Again, if the edge points to a
719 mutable file and the content of that mutable file is changed then the ctime
720 will not be updated in any case.
722 Therefore there are several ways that the 'ctime' field could be confusing:
724 1. You might be confused about whether it reflects the time of the creation
725 of a link in the Tahoe filesystem or a timestamp copied in from a local
728 2. You might be confused about whether it is a copy of the file creation time
729 (if "tahoe backup" was run on a Windows system) or of the last
730 contents-or-metadata change (if "tahoe backup" was run on a different
733 3. You might be confused by the fact that changing the contents of a mutable
734 file in Tahoe don't have any effect on any links pointing at that file in any
735 directories, although "tahoe backup" sets the link 'ctime'/'mtime' to reflect
736 timestamps about the local file corresponding to the Tahoe file to which the
739 4. Also, quite apart from Tahoe, you might be confused about the meaning of
740 the 'ctime' in UNIX local filesystems, which people sometimes think means
741 file creation time, but which actually means, in UNIX local filesystems, the
742 most recent time that the file contents or the file metadata (such as owner,
743 permission bits, extended attributes, etc.) has changed. Note that although
744 'ctime' does not mean file creation time in UNIX, it does mean link creation
745 time in Tahoe, unless the "tahoe backup" command has been used on that link,
746 in which case it means something about the local filesystem file which
747 corresponds to the Tahoe file which is pointed at by the link. It means
748 either file creation time of the local file (if "tahoe backup" was run on
749 Windows) or file-contents-or-metadata-update-time of the local file (if
750 "tahoe backup" was run on a different operating system).
752 === Attaching an existing File or Directory by its read- or write- cap ===
754 PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri
756 This attaches a child object (either a file or directory) to a specified
757 location in the virtual filesystem. The child object is referenced by its
758 read- or write- cap, as provided in the HTTP request body. This will create
759 intermediate directories as necessary.
761 This is similar to a UNIX hardlink: by referencing a previously-uploaded file
762 (or previously-created directory) instead of uploading/creating a new one,
763 you can create two references to the same object.
765 The read- or write- cap of the child is provided in the body of the HTTP
766 request, and this same cap is returned in the response body.
768 The default behavior is to overwrite any existing object at the same
769 location. To prevent this (and make the operation return an error instead
770 of overwriting), add a "replace=false" argument, as "?t=uri&replace=false".
771 With replace=false, this operation will return an HTTP 409 "Conflict" error
772 if there is already an object at the given location, rather than
773 overwriting the existing object. To allow the operation to overwrite a
774 file, but return an error when trying to overwrite a directory, use
775 "replace=only-files" (this behavior is closer to the traditional UNIX "mv"
776 command). Note that "true", "t", and "1" are all synonyms for "True", and
777 "false", "f", and "0" are synonyms for "False", and the parameter is
780 Note that this operation does not take its child cap in the form of
781 separate "rw_uri" and "ro_uri" fields. Therefore, it cannot accept a
782 child cap in a format unknown to the webapi server, unless its URI
783 starts with "ro." or "imm.". This restriction is necessary because the
784 server is not able to attenuate an unknown write cap to a read cap.
785 Unknown URIs starting with "ro." or "imm.", on the other hand, are
786 assumed to represent read caps. The client should not prefix a write
787 cap with "ro." or "imm." and pass it to this operation, since that
788 would result in granting the cap's write authority to holders of the
791 === Adding multiple files or directories to a parent directory at once ===
793 POST /uri/$DIRCAP/[SUBDIRS..]?t=set_children
794 POST /uri/$DIRCAP/[SUBDIRS..]?t=set-children (Tahoe >= v1.6)
796 This command adds multiple children to a directory in a single operation.
797 It reads the request body and interprets it as a JSON-encoded description
798 of the child names and read/write-caps that should be added.
800 The body should be a JSON-encoded dictionary, in the same format as the
801 "children" value returned by the "GET /uri/$DIRCAP?t=json" operation
802 described above. In this format, each key is a child names, and the
803 corresponding value is a tuple of (type, childinfo). "type" is ignored, and
804 "childinfo" is a dictionary that contains "rw_uri", "ro_uri", and
805 "metadata" keys. You can take the output of "GET /uri/$DIRCAP1?t=json" and
806 use it as the input to "POST /uri/$DIRCAP2?t=set_children" to make DIR2
807 look very much like DIR1 (except for any existing children of DIR2 that
808 were not overwritten, and any existing "tahoe" metadata keys as described
811 When the set_children request contains a child name that already exists in
812 the target directory, this command defaults to overwriting that child with
813 the new value (both child cap and metadata, but if the JSON data does not
814 contain a "metadata" key, the old child's metadata is preserved). The
815 command takes a boolean "overwrite=" query argument to control this
816 behavior. If you use "?t=set_children&overwrite=false", then an attempt to
817 replace an existing child will instead cause an error.
819 Any "tahoe" key in the new child's "metadata" value is ignored. Any
820 existing "tahoe" metadata is preserved. The metadata["tahoe"] value is
821 reserved for metadata generated by the tahoe node itself. The only two keys
822 currently placed here are "linkcrtime" and "linkmotime". For details, see
823 the section above entitled "Get Information About A File Or Directory (as
824 JSON)", in the "About the metadata" subsection.
826 Note that this command was introduced with the name "set_children", which
827 uses an underscore rather than a hyphen as other multi-word command names
828 do. The variant with a hyphen is now accepted, but clients that desire
829 backward compatibility should continue to use "set_children".
832 === Deleting a File or Directory ===
834 DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME
836 This removes the given name from its parent directory. CHILDNAME is the
837 name to be removed, and $DIRCAP/SUBDIRS.. indicates the directory that will
840 Note that this does not actually delete the file or directory that the name
841 points to from the tahoe grid -- it only removes the named reference from
842 this directory. If there are other names in this directory or in other
843 directories that point to the resource, then it will remain accessible
844 through those paths. Even if all names pointing to this object are removed
845 from their parent directories, then someone with possession of its read-cap
846 can continue to access the object through that cap.
848 The object will only become completely unreachable once 1: there are no
849 reachable directories that reference it, and 2: nobody is holding a read-
850 or write- cap to the object. (This behavior is very similar to the way
851 hardlinks and anonymous files work in traditional UNIX filesystems).
853 This operation will not modify more than a single directory. Intermediate
854 directories which were implicitly created by PUT or POST methods will *not*
855 be automatically removed by DELETE.
857 This method returns the file- or directory- cap of the object that was just
860 == Browser Operations ==
862 This section describes the HTTP operations that provide support for humans
863 running a web browser. Most of these operations use HTML forms that use POST
864 to drive the Tahoe node. This section is intended for HTML authors who want
865 to write web pages that contain forms and buttons which manipulate the Tahoe
868 Note that for all POST operations, the arguments listed can be provided
869 either as URL query arguments or as form body fields. URL query arguments are
870 separated from the main URL by "?", and from each other by "&". For example,
871 "POST /uri/$DIRCAP?t=upload&mutable=true". Form body fields are usually
872 specified by using <input type="hidden"> elements. For clarity, the
873 descriptions below display the most significant arguments as URL query args.
875 === Viewing A Directory (as HTML) ===
877 GET /uri/$DIRCAP/[SUBDIRS../]
879 This returns an HTML page, intended to be displayed to a human by a web
880 browser, which contains HREF links to all files and directories reachable
881 from this directory. These HREF links do not have a t= argument, meaning
882 that a human who follows them will get pages also meant for a human. It also
883 contains forms to upload new files, and to delete files and directories.
884 Those forms use POST methods to do their job.
886 === Viewing/Downloading a File ===
889 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME
891 This will retrieve the contents of the given file. The HTTP response body
892 will contain the sequence of bytes that make up the file.
894 If you want the HTTP response to include a useful Content-Type header,
895 either use the second form (which starts with a $DIRCAP), or add a
896 "filename=foo" query argument, like "GET /uri/$FILECAP?filename=foo.jpg".
897 The bare "GET /uri/$FILECAP" does not give the Tahoe node enough information
898 to determine a Content-Type (since Tahoe immutable files are merely
899 sequences of bytes, not typed+named file objects).
901 If the URL has both filename= and "save=true" in the query arguments, then
902 the server to add a "Content-Disposition: attachment" header, along with a
903 filename= parameter. When a user clicks on such a link, most browsers will
904 offer to let the user save the file instead of displaying it inline (indeed,
905 most browsers will refuse to display it inline). "true", "t", "1", and other
906 case-insensitive equivalents are all treated the same.
908 Character-set handling in URLs and HTTP headers is a dubious art[1]. For
909 maximum compatibility, Tahoe simply copies the bytes from the filename=
910 argument into the Content-Disposition header's filename= parameter, without
911 trying to interpret them in any particular way.
914 GET /named/$FILECAP/FILENAME
916 This is an alternate download form which makes it easier to get the correct
917 filename. The Tahoe server will provide the contents of the given file, with
918 a Content-Type header derived from the given filename. This form is used to
919 get browsers to use the "Save Link As" feature correctly, and also helps
920 command-line tools like "wget" and "curl" use the right filename. Note that
921 this form can *only* be used with file caps; it is an error to use a
922 directory cap after the /named/ prefix.
924 === Get Information About A File Or Directory (as HTML) ===
926 GET /uri/$FILECAP?t=info
927 GET /uri/$DIRCAP/?t=info
928 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR/?t=info
929 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=info
931 This returns a human-oriented HTML page with more detail about the selected
932 file or directory object. This page contains the following items:
937 raw contents (text/plain)
938 access caps (URIs): verify-cap, read-cap, write-cap (for mutable objects)
939 check/verify/repair form
940 deep-check/deep-size/deep-stats/manifest (for directories)
941 replace-conents form (for mutable files)
943 === Creating a Directory ===
947 This creates a new empty directory, but does not attach it to the virtual
950 If a "redirect_to_result=true" argument is provided, then the HTTP response
951 will cause the web browser to be redirected to a /uri/$DIRCAP page that
952 gives access to the newly-created directory. If you bookmark this page,
953 you'll be able to get back to the directory again in the future. This is the
954 recommended way to start working with a Tahoe server: create a new unlinked
955 directory (using redirect_to_result=true), then bookmark the resulting
956 /uri/$DIRCAP page. There is a "create directory" button on the Welcome page
957 to invoke this action.
959 If "redirect_to_result=true" is not provided (or is given a value of
960 "false"), then the HTTP response body will simply be the write-cap of the
963 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME
965 This creates a new empty directory as a child of the designated SUBDIR. This
966 will create additional intermediate directories as necessary.
968 If a "when_done=URL" argument is provided, the HTTP response will cause the
969 web browser to redirect to the given URL. This provides a convenient way to
970 return the browser to the directory that was just modified. Without a
971 when_done= argument, the HTTP response will simply contain the write-cap of
972 the directory that was just created.
975 === Uploading a File ===
979 This uploads a file, and produces a file-cap for the contents, but does not
980 attach the file into the filesystem. No directories will be modified by
983 The file must be provided as the "file" field of an HTML encoded form body,
984 produced in response to an HTML form like this:
985 <form action="/uri" method="POST" enctype="multipart/form-data">
986 <input type="hidden" name="t" value="upload" />
987 <input type="file" name="file" />
988 <input type="submit" value="Upload Unlinked" />
991 If a "when_done=URL" argument is provided, the response body will cause the
992 browser to redirect to the given URL. If the when_done= URL has the string
993 "%(uri)s" in it, that string will be replaced by a URL-escaped form of the
994 newly created file-cap. (Note that without this substitution, there is no
995 way to access the file that was just uploaded).
997 The default (in the absence of when_done=) is to return an HTML page that
998 describes the results of the upload. This page will contain information
999 about which storage servers were used for the upload, how long each
1000 operation took, etc.
1002 If a "mutable=true" argument is provided, the operation will create a
1003 mutable file, and the response body will contain the write-cap instead of
1004 the upload results page. The default is to create an immutable file,
1005 returning the upload results page as a response.
1008 POST /uri/$DIRCAP/[SUBDIRS../]?t=upload
1010 This uploads a file, and attaches it as a new child of the given directory,
1011 which must be mutable. The file must be provided as the "file" field of an
1012 HTML-encoded form body, produced in response to an HTML form like this:
1013 <form action="." method="POST" enctype="multipart/form-data">
1014 <input type="hidden" name="t" value="upload" />
1015 <input type="file" name="file" />
1016 <input type="submit" value="Upload" />
1019 A "name=" argument can be provided to specify the new child's name,
1020 otherwise it will be taken from the "filename" field of the upload form
1021 (most web browsers will copy the last component of the original file's
1022 pathname into this field). To avoid confusion, name= is not allowed to
1025 If there is already a child with that name, and it is a mutable file, then
1026 its contents are replaced with the data being uploaded. If it is not a
1027 mutable file, the default behavior is to remove the existing child before
1028 creating a new one. To prevent this (and make the operation return an error
1029 instead of overwriting the old child), add a "replace=false" argument, as
1030 "?t=upload&replace=false". With replace=false, this operation will return an
1031 HTTP 409 "Conflict" error if there is already an object at the given
1032 location, rather than overwriting the existing object. Note that "true",
1033 "t", and "1" are all synonyms for "True", and "false", "f", and "0" are
1034 synonyms for "False". the parameter is case-insensitive.
1036 This will create additional intermediate directories as necessary, although
1037 since it is expected to be triggered by a form that was retrieved by "GET
1038 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1041 If a "mutable=true" argument is provided, any new file that is created will
1042 be a mutable file instead of an immutable one. <input type="checkbox"
1043 name="mutable" /> will give the user a way to set this option.
1045 If a "when_done=URL" argument is provided, the HTTP response will cause the
1046 web browser to redirect to the given URL. This provides a convenient way to
1047 return the browser to the directory that was just modified. Without a
1048 when_done= argument, the HTTP response will simply contain the file-cap of
1049 the file that was just uploaded (a write-cap for mutable files, or a
1050 read-cap for immutable files).
1052 POST /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=upload
1054 This also uploads a file and attaches it as a new child of the given
1055 directory, which must be mutable. It is a slight variant of the previous
1056 operation, as the URL refers to the target file rather than the parent
1057 directory. It is otherwise identical: this accepts mutable= and when_done=
1060 POST /uri/$FILECAP?t=upload
1062 This modifies the contents of an existing mutable file in-place. An error is
1063 signalled if $FILECAP does not refer to a mutable file. It behaves just like
1064 the "PUT /uri/$FILECAP" form, but uses a POST for the benefit of HTML forms
1067 === Attaching An Existing File Or Directory (by URI) ===
1069 POST /uri/$DIRCAP/[SUBDIRS../]?t=uri&name=CHILDNAME&uri=CHILDCAP
1071 This attaches a given read- or write- cap "CHILDCAP" to the designated
1072 directory, with a specified child name. This behaves much like the PUT t=uri
1073 operation, and is a lot like a UNIX hardlink. It is subject to the same
1074 restrictions as that operation on the use of cap formats unknown to the
1077 This will create additional intermediate directories as necessary, although
1078 since it is expected to be triggered by a form that was retrieved by "GET
1079 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
1082 This accepts the same replace= argument as POST t=upload.
1084 === Deleting A Child ===
1086 POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME
1088 This instructs the node to remove a child object (file or subdirectory) from
1089 the given directory, which must be mutable. Note that the entire subtree is
1090 unlinked from the parent. Unlike deleting a subdirectory in a UNIX local
1091 filesystem, the subtree need not be empty; if it isn't, then other references
1092 into the subtree will see that the child subdirectories are not modified by
1093 this operation. Only the link from the given directory to its child is severed.
1095 === Renaming A Child ===
1097 POST /uri/$DIRCAP/[SUBDIRS../]?t=rename&from_name=OLD&to_name=NEW
1099 This instructs the node to rename a child of the given directory, which must
1100 be mutable. This has a similar effect to removing the child, then adding the
1101 same child-cap under the new name, except that it preserves metadata. This
1102 operation cannot move the child to a different directory.
1104 This operation will replace any existing child of the new name, making it
1105 behave like the UNIX "mv -f" command.
1107 === Other Utilities ===
1111 This causes a redirect to /uri/$CAP, and retains any additional query
1112 arguments (like filename= or save=). This is for the convenience of web
1113 forms which allow the user to paste in a read- or write- cap (obtained
1114 through some out-of-band channel, like IM or email).
1116 Note that this form merely redirects to the specific file or directory
1117 indicated by the $CAP: unlike the GET /uri/$DIRCAP form, you cannot
1118 traverse to children by appending additional path segments to the URL.
1120 GET /uri/$DIRCAP/[SUBDIRS../]?t=rename-form&name=$CHILDNAME
1122 This provides a useful facility to browser-based user interfaces. It
1123 returns a page containing a form targetting the "POST $DIRCAP t=rename"
1124 functionality described above, with the provided $CHILDNAME present in the
1125 'from_name' field of that form. I.e. this presents a form offering to
1126 rename $CHILDNAME, requesting the new name, and submitting POST rename.
1128 GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri
1130 This returns the file- or directory- cap for the specified object.
1132 GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=readonly-uri
1134 This returns a read-only file- or directory- cap for the specified object.
1135 If the object is an immutable file, this will return the same value as
1138 === Debugging and Testing Features ===
1140 These URLs are less-likely to be helpful to the casual Tahoe user, and are
1141 mainly intended for developers.
1145 This triggers the FileChecker to determine the current "health" of the
1146 given file or directory, by counting how many shares are available. The
1147 page that is returned will display the results. This can be used as a "show
1148 me detailed information about this file" page.
1150 If a verify=true argument is provided, the node will perform a more
1151 intensive check, downloading and verifying every single bit of every share.
1153 If an add-lease=true argument is provided, the node will also add (or
1154 renew) a lease to every share it encounters. Each lease will keep the share
1155 alive for a certain period of time (one month by default). Once the last
1156 lease expires or is explicitly cancelled, the storage server is allowed to
1159 If an output=JSON argument is provided, the response will be
1160 machine-readable JSON instead of human-oriented HTML. The data is a
1161 dictionary with the following keys:
1163 storage-index: a base32-encoded string with the objects's storage index,
1164 or an empty string for LIT files
1165 summary: a string, with a one-line summary of the stats of the file
1166 results: a dictionary that describes the state of the file. For LIT files,
1167 this dictionary has only the 'healthy' key, which will always be
1168 True. For distributed files, this dictionary has the following
1170 count-shares-good: the number of good shares that were found
1171 count-shares-needed: 'k', the number of shares required for recovery
1172 count-shares-expected: 'N', the number of total shares generated
1173 count-good-share-hosts: the number of distinct storage servers with
1174 good shares. If this number is less than
1175 count-shares-good, then some shares are doubled
1176 up, increasing the correlation of failures. This
1177 indicates that one or more shares should be
1178 moved to an otherwise unused server, if one is
1180 count-wrong-shares: for mutable files, the number of shares for
1181 versions other than the 'best' one (highest
1182 sequence number, highest roothash). These are
1184 count-recoverable-versions: for mutable files, the number of
1185 recoverable versions of the file. For
1186 a healthy file, this will equal 1.
1187 count-unrecoverable-versions: for mutable files, the number of
1188 unrecoverable versions of the file.
1189 For a healthy file, this will be 0.
1190 count-corrupt-shares: the number of shares with integrity failures
1191 list-corrupt-shares: a list of "share locators", one for each share
1192 that was found to be corrupt. Each share locator
1193 is a list of (serverid, storage_index, sharenum).
1194 needs-rebalancing: (bool) True if there are multiple shares on a single
1195 storage server, indicating a reduction in reliability
1196 that could be resolved by moving shares to new
1198 servers-responding: list of base32-encoded storage server identifiers,
1199 one for each server which responded to the share
1201 healthy: (bool) True if the file is completely healthy, False otherwise.
1202 Healthy files have at least N good shares. Overlapping shares
1203 (indicated by count-good-share-hosts < count-shares-good) do not
1204 currently cause a file to be marked unhealthy. If there are at
1205 least N good shares, then corrupt shares do not cause the file to
1206 be marked unhealthy, although the corrupt shares will be listed
1207 in the results (list-corrupt-shares) and should be manually
1208 removed to wasting time in subsequent downloads (as the
1209 downloader rediscovers the corruption and uses alternate shares).
1210 sharemap: dict mapping share identifier to list of serverids
1211 (base32-encoded strings). This indicates which servers are
1212 holding which shares. For immutable files, the shareid is
1213 an integer (the share number, from 0 to N-1). For
1214 immutable files, it is a string of the form
1215 'seq%d-%s-sh%d', containing the sequence number, the
1216 roothash, and the share number.
1218 POST $URL?t=start-deep-check (must add &ophandle=XYZ)
1220 This initiates a recursive walk of all files and directories reachable from
1221 the target, performing a check on each one just like t=check. The result
1222 page will contain a summary of the results, including details on any
1223 file/directory that was not fully healthy.
1225 t=start-deep-check can only be invoked on a directory. An error (400
1226 BAD_REQUEST) will be signalled if it is invoked on a file. The recursive
1227 walker will deal with loops safely.
1229 This accepts the same verify= and add-lease= arguments as t=check.
1231 Since this operation can take a long time (perhaps a second per object),
1232 the ophandle= argument is required (see "Slow Operations, Progress, and
1233 Cancelling" above). The response to this POST will be a redirect to the
1234 corresponding /operations/$HANDLE page (with output=HTML or output=JSON to
1235 match the output= argument given to the POST). The deep-check operation
1236 will continue to run in the background, and the /operations page should be
1237 used to find out when the operation is done.
1239 Detailed check results for non-healthy files and directories will be
1240 available under /operations/$HANDLE/$STORAGEINDEX, and the HTML status will
1241 contain links to these detailed results.
1243 The HTML /operations/$HANDLE page for incomplete operations will contain a
1244 meta-refresh tag, set to 60 seconds, so that a browser which uses
1245 deep-check will automatically poll until the operation has completed.
1247 The JSON page (/options/$HANDLE?output=JSON) will contain a
1248 machine-readable JSON dictionary with the following keys:
1250 finished: a boolean, True if the operation is complete, else False. Some
1251 of the remaining keys may not be present until the operation
1253 root-storage-index: a base32-encoded string with the storage index of the
1254 starting point of the deep-check operation
1255 count-objects-checked: count of how many objects were checked. Note that
1256 non-distributed objects (i.e. small immutable LIT
1257 files) are not checked, since for these objects,
1258 the data is contained entirely in the URI.
1259 count-objects-healthy: how many of those objects were completely healthy
1260 count-objects-unhealthy: how many were damaged in some way
1261 count-corrupt-shares: how many shares were found to have corruption,
1262 summed over all objects examined
1263 list-corrupt-shares: a list of "share identifiers", one for each share
1264 that was found to be corrupt. Each share identifier
1265 is a list of (serverid, storage_index, sharenum).
1266 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1267 each file that was not fully healthy. 'pathname' is
1268 a list of strings (which can be joined by "/"
1269 characters to turn it into a single string),
1270 relative to the directory on which deep-check was
1271 invoked. The 'check-results' field is the same as
1272 that returned by t=check&output=JSON, described
1274 stats: a dictionary with the same keys as the t=start-deep-stats command
1277 POST $URL?t=stream-deep-check
1279 This initiates a recursive walk of all files and directories reachable from
1280 the target, performing a check on each one just like t=check. For each
1281 unique object (duplicates are skipped), a single line of JSON is emitted to
1282 the HTTP response channel (or an error indication, see below). When the walk
1283 is complete, a final line of JSON is emitted which contains the accumulated
1284 file-size/count "deep-stats" data.
1286 This command takes the same arguments as t=start-deep-check.
1288 A CLI tool can split the response stream on newlines into "response units",
1289 and parse each response unit as JSON. Each such parsed unit will be a
1290 dictionary, and will contain at least the "type" key: a string, one of
1291 "file", "directory", or "stats".
1293 For all units that have a type of "file" or "directory", the dictionary will
1294 contain the following keys:
1296 "path": a list of strings, with the path that is traversed to reach the
1298 "cap": a write-cap URI for the file or directory, if available, else a
1300 "verifycap": a verify-cap URI for the file or directory
1301 "repaircap": an URI for the weakest cap that can still be used to repair
1303 "storage-index": a base32 storage index for the object
1304 "check-results": a copy of the dictionary which would be returned by
1305 t=check&output=json, with three top-level keys:
1306 "storage-index", "summary", and "results", and a variety
1307 of counts and sharemaps in the "results" value.
1309 Note that non-distributed files (i.e. LIT files) will have values of None
1310 for verifycap, repaircap, and storage-index, since these files can neither
1311 be verified nor repaired, and are not stored on the storage servers.
1312 Likewise the check-results dictionary will be limited: an empty string for
1313 storage-index, and a results dictionary with only the "healthy" key.
1315 The last unit in the stream will have a type of "stats", and will contain
1316 the keys described in the "start-deep-stats" operation, below.
1318 If any errors occur during the traversal (specifically if a directory is
1319 unrecoverable, such that further traversal is not possible), an error
1320 indication is written to the response body, instead of the usual line of
1321 JSON. This error indication line will begin with the string "ERROR:" (in all
1322 caps), and contain a summary of the error on the rest of the line. The
1323 remaining lines of the response body will be a python exception. The client
1324 application should look for the ERROR: and stop processing JSON as soon as
1325 it is seen. Note that neither a file being unrecoverable nor a directory
1326 merely being unhealthy will cause traversal to stop. The line just before
1327 the ERROR: will describe the directory that was untraversable, since the
1328 unit is emitted to the HTTP response body before the child is traversed.
1331 POST $URL?t=check&repair=true
1333 This performs a health check of the given file or directory, and if the
1334 checker determines that the object is not healthy (some shares are missing
1335 or corrupted), it will perform a "repair". During repair, any missing
1336 shares will be regenerated and uploaded to new servers.
1338 This accepts the same verify=true and add-lease= arguments as t=check. When
1339 an output=JSON argument is provided, the machine-readable JSON response
1340 will contain the following keys:
1342 storage-index: a base32-encoded string with the objects's storage index,
1343 or an empty string for LIT files
1344 repair-attempted: (bool) True if repair was attempted
1345 repair-successful: (bool) True if repair was attempted and the file was
1346 fully healthy afterwards. False if no repair was
1347 attempted, or if a repair attempt failed.
1348 pre-repair-results: a dictionary that describes the state of the file
1349 before any repair was performed. This contains exactly
1350 the same keys as the 'results' value of the t=check
1351 response, described above.
1352 post-repair-results: a dictionary that describes the state of the file
1353 after any repair was performed. If no repair was
1354 performed, post-repair-results and pre-repair-results
1355 will be the same. This contains exactly the same keys
1356 as the 'results' value of the t=check response,
1359 POST $URL?t=start-deep-check&repair=true (must add &ophandle=XYZ)
1361 This triggers a recursive walk of all files and directories, performing a
1362 t=check&repair=true on each one.
1364 Like t=start-deep-check without the repair= argument, this can only be
1365 invoked on a directory. An error (400 BAD_REQUEST) will be signalled if it
1366 is invoked on a file. The recursive walker will deal with loops safely.
1368 This accepts the same verify= and add-lease= arguments as
1369 t=start-deep-check. It uses the same ophandle= mechanism as
1370 start-deep-check. When an output=JSON argument is provided, the response
1371 will contain the following keys:
1373 finished: (bool) True if the operation has completed, else False
1374 root-storage-index: a base32-encoded string with the storage index of the
1375 starting point of the deep-check operation
1376 count-objects-checked: count of how many objects were checked
1378 count-objects-healthy-pre-repair: how many of those objects were completely
1379 healthy, before any repair
1380 count-objects-unhealthy-pre-repair: how many were damaged in some way
1381 count-objects-healthy-post-repair: how many of those objects were completely
1382 healthy, after any repair
1383 count-objects-unhealthy-post-repair: how many were damaged in some way
1385 count-repairs-attempted: repairs were attempted on this many objects.
1386 count-repairs-successful: how many repairs resulted in healthy objects
1387 count-repairs-unsuccessful: how many repairs resulted did not results in
1388 completely healthy objects
1389 count-corrupt-shares-pre-repair: how many shares were found to have
1390 corruption, summed over all objects
1391 examined, before any repair
1392 count-corrupt-shares-post-repair: how many shares were found to have
1393 corruption, summed over all objects
1394 examined, after any repair
1395 list-corrupt-shares: a list of "share identifiers", one for each share
1396 that was found to be corrupt (before any repair).
1397 Each share identifier is a list of (serverid,
1398 storage_index, sharenum).
1399 list-remaining-corrupt-shares: like list-corrupt-shares, but mutable shares
1400 that were successfully repaired are not
1401 included. These are shares that need
1402 manual processing. Since immutable shares
1403 cannot be modified by clients, all corruption
1404 in immutable shares will be listed here.
1405 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1406 each file that was not fully healthy. 'pathname' is
1407 relative to the directory on which deep-check was
1408 invoked. The 'check-results' field is the same as
1409 that returned by t=check&repair=true&output=JSON,
1411 stats: a dictionary with the same keys as the t=start-deep-stats command
1414 POST $URL?t=stream-deep-check&repair=true
1416 This triggers a recursive walk of all files and directories, performing a
1417 t=check&repair=true on each one. For each unique object (duplicates are
1418 skipped), a single line of JSON is emitted to the HTTP response channel (or
1419 an error indication). When the walk is complete, a final line of JSON is
1420 emitted which contains the accumulated file-size/count "deep-stats" data.
1422 This emits the same data as t=stream-deep-check (without the repair=true),
1423 except that the "check-results" field is replaced with a
1424 "check-and-repair-results" field, which contains the keys returned by
1425 t=check&repair=true&output=json (i.e. repair-attempted, repair-successful,
1426 pre-repair-results, and post-repair-results). The output does not contain
1427 the summary dictionary that is provied by t=start-deep-check&repair=true
1428 (the one with count-objects-checked and list-unhealthy-files), since the
1429 receiving client is expected to calculate those values itself from the
1430 stream of per-object check-and-repair-results.
1432 Note that the "ERROR:" indication will only be emitted if traversal stops,
1433 which will only occur if an unrecoverable directory is encountered. If a
1434 file or directory repair fails, the traversal will continue, and the repair
1435 failure will be indicated in the JSON data (in the "repair-successful" key).
1437 POST $DIRURL?t=start-manifest (must add &ophandle=XYZ)
1439 This operation generates a "manfest" of the given directory tree, mostly
1440 for debugging. This is a table of (path, filecap/dircap), for every object
1441 reachable from the starting directory. The path will be slash-joined, and
1442 the filecap/dircap will contain a link to the object in question. This page
1443 gives immediate access to every object in the virtual filesystem subtree.
1445 This operation uses the same ophandle= mechanism as deep-check. The
1446 corresponding /operations/$HANDLE page has three different forms. The
1447 default is output=HTML.
1449 If output=text is added to the query args, the results will be a text/plain
1450 list. The first line is special: it is either "finished: yes" or "finished:
1451 no"; if the operation is not finished, you must periodically reload the
1452 page until it completes. The rest of the results are a plaintext list, with
1453 one file/dir per line, slash-separated, with the filecap/dircap separated
1456 If output=JSON is added to the queryargs, then the results will be a
1457 JSON-formatted dictionary with six keys. Note that because large directory
1458 structures can result in very large JSON results, the full results will not
1459 be available until the operation is complete (i.e. until output["finished"]
1462 finished (bool): if False then you must reload the page until True
1463 origin_si (base32 str): the storage index of the starting point
1464 manifest: list of (path, cap) tuples, where path is a list of strings.
1465 verifycaps: list of (printable) verify cap strings
1466 storage-index: list of (base32) storage index strings
1467 stats: a dictionary with the same keys as the t=start-deep-stats command
1470 POST $DIRURL?t=start-deep-size (must add &ophandle=XYZ)
1472 This operation generates a number (in bytes) containing the sum of the
1473 filesize of all directories and immutable files reachable from the given
1474 directory. This is a rough lower bound of the total space consumed by this
1475 subtree. It does not include space consumed by mutable files, nor does it
1476 take expansion or encoding overhead into account. Later versions of the
1477 code may improve this estimate upwards.
1479 The /operations/$HANDLE status output consists of two lines of text:
1484 POST $DIRURL?t=start-deep-stats (must add &ophandle=XYZ)
1486 This operation performs a recursive walk of all files and directories
1487 reachable from the given directory, and generates a collection of
1488 statistics about those objects.
1490 The result (obtained from the /operations/$OPHANDLE page) is a
1491 JSON-serialized dictionary with the following keys (note that some of these
1492 keys may be missing until 'finished' is True):
1494 finished: (bool) True if the operation has finished, else False
1495 count-immutable-files: count of how many CHK files are in the set
1496 count-mutable-files: same, for mutable files (does not include directories)
1497 count-literal-files: same, for LIT files (data contained inside the URI)
1498 count-files: sum of the above three
1499 count-directories: count of directories
1500 count-unknown: count of unrecognized objects (perhaps from the future)
1501 size-immutable-files: total bytes for all CHK files in the set, =deep-size
1502 size-mutable-files (TODO): same, for current version of all mutable files
1503 size-literal-files: same, for LIT files
1504 size-directories: size of directories (includes size-literal-files)
1505 size-files-histogram: list of (minsize, maxsize, count) buckets,
1506 with a histogram of filesizes, 5dB/bucket,
1507 for both literal and immutable files
1508 largest-directory: number of children in the largest directory
1509 largest-immutable-file: number of bytes in the largest CHK file
1511 size-mutable-files is not implemented, because it would require extra
1512 queries to each mutable file to get their size. This may be implemented in
1515 Assuming no sharing, the basic space consumed by a single root directory is
1516 the sum of size-immutable-files, size-mutable-files, and size-directories.
1517 The actual disk space used by the shares is larger, because of the
1518 following sources of overhead:
1521 expansion due to erasure coding
1522 share management data (leases)
1523 backend (ext3) minimum block size
1525 POST $URL?t=stream-manifest
1527 This operation performs a recursive walk of all files and directories
1528 reachable from the given starting point. For each such unique object
1529 (duplicates are skipped), a single line of JSON is emitted to the HTTP
1530 response channel (or an error indication, see below). When the walk is
1531 complete, a final line of JSON is emitted which contains the accumulated
1532 file-size/count "deep-stats" data.
1534 A CLI tool can split the response stream on newlines into "response units",
1535 and parse each response unit as JSON. Each such parsed unit will be a
1536 dictionary, and will contain at least the "type" key: a string, one of
1537 "file", "directory", or "stats".
1539 For all units that have a type of "file" or "directory", the dictionary will
1540 contain the following keys:
1542 "path": a list of strings, with the path that is traversed to reach the
1544 "cap": a write-cap URI for the file or directory, if available, else a
1546 "verifycap": a verify-cap URI for the file or directory
1547 "repaircap": an URI for the weakest cap that can still be used to repair
1549 "storage-index": a base32 storage index for the object
1551 Note that non-distributed files (i.e. LIT files) will have values of None
1552 for verifycap, repaircap, and storage-index, since these files can neither
1553 be verified nor repaired, and are not stored on the storage servers.
1555 The last unit in the stream will have a type of "stats", and will contain
1556 the keys described in the "start-deep-stats" operation, below.
1558 If any errors occur during the traversal (specifically if a directory is
1559 unrecoverable, such that further traversal is not possible), an error
1560 indication is written to the response body, instead of the usual line of
1561 JSON. This error indication line will begin with the string "ERROR:" (in all
1562 caps), and contain a summary of the error on the rest of the line. The
1563 remaining lines of the response body will be a python exception. The client
1564 application should look for the ERROR: and stop processing JSON as soon as
1565 it is seen. The line just before the ERROR: will describe the directory that
1566 was untraversable, since the manifest entry is emitted to the HTTP response
1567 body before the child is traversed.
1569 == Other Useful Pages ==
1571 The portion of the web namespace that begins with "/uri" (and "/named") is
1572 dedicated to giving users (both humans and programs) access to the Tahoe
1573 virtual filesystem. The rest of the namespace provides status information
1574 about the state of the Tahoe node.
1576 GET / (the root page)
1578 This is the "Welcome Page", and contains a few distinct sections:
1580 Node information: library versions, local nodeid, services being provided.
1582 Filesystem Access Forms: create a new directory, view a file/directory by
1583 URI, upload a file (unlinked), download a file by
1586 Grid Status: introducer information, helper information, connected storage
1591 This page lists all active uploads and downloads, and contains a short list
1592 of recent upload/download operations. Each operation has a link to a page
1593 that describes file sizes, servers that were involved, and the time consumed
1594 in each phase of the operation.
1596 A GET of /status/?t=json will contain a machine-readable subset of the same
1597 data. It returns a JSON-encoded dictionary. The only key defined at this
1598 time is "active", with a value that is a list of operation dictionaries, one
1599 for each active operation. Once an operation is completed, it will no longer
1600 appear in data["active"] .
1602 Each op-dict contains a "type" key, one of "upload", "download",
1603 "mapupdate", "publish", or "retrieve" (the first two are for immutable
1604 files, while the latter three are for mutable files and directories).
1606 The "upload" op-dict will contain the following keys:
1608 type (string): "upload"
1609 storage-index-string (string): a base32-encoded storage index
1610 total-size (int): total size of the file
1611 status (string): current status of the operation
1612 progress-hash (float): 1.0 when the file has been hashed
1613 progress-ciphertext (float): 1.0 when the file has been encrypted.
1614 progress-encode-push (float): 1.0 when the file has been encoded and
1615 pushed to the storage servers. For helper
1616 uploads, the ciphertext value climbs to 1.0
1617 first, then encoding starts. For unassisted
1618 uploads, ciphertext and encode-push progress
1619 will climb at the same pace.
1621 The "download" op-dict will contain the following keys:
1623 type (string): "download"
1624 storage-index-string (string): a base32-encoded storage index
1625 total-size (int): total size of the file
1626 status (string): current status of the operation
1627 progress (float): 1.0 when the file has been fully downloaded
1629 Front-ends which want to report progress information are advised to simply
1630 average together all the progress-* indicators. A slightly more accurate
1631 value can be found by ignoring the progress-hash value (since the current
1632 implementation hashes synchronously, so clients will probably never see
1633 progress-hash!=1.0).
1637 This page provides a basic tool to predict the likely storage and bandwidth
1638 requirements of a large Tahoe grid. It provides forms to input things like
1639 total number of users, number of files per user, average file size, number
1640 of servers, expansion ratio, hard drive failure rate, etc. It then provides
1641 numbers like how many disks per server will be needed, how many read
1642 operations per second should be expected, and the likely MTBF for files in
1643 the grid. This information is very preliminary, and the model upon which it
1644 is based still needs a lot of work.
1648 If the node is running a helper (i.e. if [helper]enabled is set to True in
1649 tahoe.cfg), then this page will provide a list of all the helper operations
1650 currently in progress. If "?t=json" is added to the URL, it will return a
1651 JSON-formatted list of helper statistics, which can then be used to produce
1652 graphs to indicate how busy the helper is.
1656 This page provides "node statistics", which are collected from a variety of
1659 load_monitor: every second, the node schedules a timer for one second in
1660 the future, then measures how late the subsequent callback
1661 is. The "load_average" is this tardiness, measured in
1662 seconds, averaged over the last minute. It is an indication
1663 of a busy node, one which is doing more work than can be
1664 completed in a timely fashion. The "max_load" value is the
1665 highest value that has been seen in the last 60 seconds.
1667 cpu_monitor: every minute, the node uses time.clock() to measure how much
1668 CPU time it has used, and it uses this value to produce
1669 1min/5min/15min moving averages. These values range from 0%
1670 (0.0) to 100% (1.0), and indicate what fraction of the CPU
1671 has been used by the Tahoe node. Not all operating systems
1672 provide meaningful data to time.clock(): they may report 100%
1673 CPU usage at all times.
1675 uploader: this counts how many immutable files (and bytes) have been
1676 uploaded since the node was started
1678 downloader: this counts how many immutable files have been downloaded
1679 since the node was started
1681 publishes: this counts how many mutable files (including directories) have
1682 been modified since the node was started
1684 retrieves: this counts how many mutable files (including directories) have
1685 been read since the node was started
1687 There are other statistics that are tracked by the node. The "raw stats"
1688 section shows a formatted dump of all of them.
1690 By adding "?t=json" to the URL, the node will return a JSON-formatted
1691 dictionary of stats values, which can be used by other tools to produce
1692 graphs of node behavior. The misc/munin/ directory in the source
1693 distribution provides some tools to produce these graphs.
1695 GET / (introducer status)
1697 For Introducer nodes, the welcome page displays information about both
1698 clients and servers which are connected to the introducer. Servers make
1699 "service announcements", and these are listed in a table. Clients will
1700 subscribe to hear about service announcements, and these subscriptions are
1701 listed in a separate table. Both tables contain information about what
1702 version of Tahoe is being run by the remote node, their advertised and
1703 outbound IP addresses, their nodeid and nickname, and how long they have
1706 By adding "?t=json" to the URL, the node will return a JSON-formatted
1707 dictionary of stats values, which can be used to produce graphs of connected
1708 clients over time. This dictionary has the following keys:
1710 ["subscription_summary"] : a dictionary mapping service name (like
1711 "storage") to an integer with the number of
1712 clients that have subscribed to hear about that
1714 ["announcement_summary"] : a dictionary mapping service name to an integer
1715 with the number of servers which are announcing
1717 ["announcement_distinct_hosts"] : a dictionary mapping service name to an
1718 integer which represents the number of
1719 distinct hosts that are providing that
1720 service. If two servers have announced
1721 FURLs which use the same hostnames (but
1722 different ports and tubids), they are
1723 considered to be on the same host.
1726 == Static Files in /public_html ==
1728 The webapi server will take any request for a URL that starts with /static
1729 and serve it from a configurable directory which defaults to
1730 $BASEDIR/public_html . This is configured by setting the "[node]web.static"
1731 value in $BASEDIR/tahoe.cfg . If this is left at the default value of
1732 "public_html", then http://localhost:3456/static/subdir/foo.html will be
1733 served with the contents of the file $BASEDIR/public_html/subdir/foo.html .
1735 This can be useful to serve a javascript application which provides a
1736 prettier front-end to the rest of the Tahoe webapi.
1739 == Safety and security issues -- names vs. URIs ==
1741 Summary: use explicit file- and dir- caps whenever possible, to reduce the
1742 potential for surprises when the filesystem structure is changed.
1744 Tahoe provides a mutable filesystem, but the ways that the filesystem can
1745 change are limited. The only thing that can change is that the mapping from
1746 child names to child objects that each directory contains can be changed by
1747 adding a new child name pointing to an object, removing an existing child name,
1748 or changing an existing child name to point to a different object.
1750 Obviously if you query Tahoe for information about the filesystem and then act
1751 to change the filesystem (such as by getting a listing of the contents of a
1752 directory and then adding a file to the directory), then the filesystem might
1753 have been changed after you queried it and before you acted upon it. However,
1754 if you use the URI instead of the pathname of an object when you act upon the
1755 object, then the only change that can happen is if the object is a directory
1756 then the set of child names it has might be different. If, on the other hand,
1757 you act upon the object using its pathname, then a different object might be in
1758 that place, which can result in more kinds of surprises.
1760 For example, suppose you are writing code which recursively downloads the
1761 contents of a directory. The first thing your code does is fetch the listing
1762 of the contents of the directory. For each child that it fetched, if that
1763 child is a file then it downloads the file, and if that child is a directory
1764 then it recurses into that directory. Now, if the download and the recurse
1765 actions are performed using the child's name, then the results might be
1766 wrong, because for example a child name that pointed to a sub-directory when
1767 you listed the directory might have been changed to point to a file (in which
1768 case your attempt to recurse into it would result in an error and the file
1769 would be skipped), or a child name that pointed to a file when you listed the
1770 directory might now point to a sub-directory (in which case your attempt to
1771 download the child would result in a file containing HTML text describing the
1774 If your recursive algorithm uses the uri of the child instead of the name of
1775 the child, then those kinds of mistakes just can't happen. Note that both the
1776 child's name and the child's URI are included in the results of listing the
1777 parent directory, so it isn't any harder to use the URI for this purpose.
1779 The read and write caps in a given directory node are separate URIs, and
1780 can't be assumed to point to the same object even if they were retrieved in
1781 the same operation (although the webapi server attempts to ensure this
1782 in most cases). If you need to rely on that property, you should explicitly
1783 verify it. More generally, you should not make assumptions about the
1784 internal consistency of the contents of mutable directories. As a result
1785 of the signatures on mutable object versions, it is guaranteed that a given
1786 version was written in a single update, but -- as in the case of a file --
1787 the contents may have been chosen by a malicious writer in a way that is
1788 designed to confuse applications that rely on their consistency.
1790 In general, use names if you want "whatever object (whether file or
1791 directory) is found by following this name (or sequence of names) when my
1792 request reaches the server". Use URIs if you want "this particular object".
1794 == Concurrency Issues ==
1796 Tahoe uses both mutable and immutable files. Mutable files can be created
1797 explicitly by doing an upload with ?mutable=true added, or implicitly by
1798 creating a new directory (since a directory is just a special way to
1799 interpret a given mutable file).
1801 Mutable files suffer from the same consistency-vs-availability tradeoff that
1802 all distributed data storage systems face. It is not possible to
1803 simultaneously achieve perfect consistency and perfect availability in the
1804 face of network partitions (servers being unreachable or faulty).
1806 Tahoe tries to achieve a reasonable compromise, but there is a basic rule in
1807 place, known as the Prime Coordination Directive: "Don't Do That". What this
1808 means is that if write-access to a mutable file is available to several
1809 parties, then those parties are responsible for coordinating their activities
1810 to avoid multiple simultaneous updates. This could be achieved by having
1811 these parties talk to each other and using some sort of locking mechanism, or
1812 by serializing all changes through a single writer.
1814 The consequences of performing uncoordinated writes can vary. Some of the
1815 writers may lose their changes, as somebody else wins the race condition. In
1816 many cases the file will be left in an "unhealthy" state, meaning that there
1817 are not as many redundant shares as we would like (reducing the reliability
1818 of the file against server failures). In the worst case, the file can be left
1819 in such an unhealthy state that no version is recoverable, even the old ones.
1820 It is this small possibility of data loss that prompts us to issue the Prime
1821 Coordination Directive.
1823 Tahoe nodes implement internal serialization to make sure that a single Tahoe
1824 node cannot conflict with itself. For example, it is safe to issue two
1825 directory modification requests to a single tahoe node's webapi server at the
1826 same time, because the Tahoe node will internally delay one of them until
1827 after the other has finished being applied. (This feature was introduced in
1828 Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing
1829 web requests themselves).
1831 For more details, please see the "Consistency vs Availability" and "The Prime
1832 Coordination Directive" sections of mutable.txt, in the same directory as
1836 [1]: URLs and HTTP and UTF-8, Oh My
1838 HTTP does not provide a mechanism to specify the character set used to
1839 encode non-ascii names in URLs (rfc2396#2.1). We prefer the convention that
1840 the filename= argument shall be a URL-encoded UTF-8 encoded unicode object.
1841 For example, suppose we want to provoke the server into using a filename of
1842 "f i a n c e-acute e" (i.e. F I A N C U+00E9 E). The UTF-8 encoding of this
1843 is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\xC3\xA9e", as python's
1844 repr() function would show). To encode this into a URL, the non-printable
1845 characters must be escaped with the urlencode '%XX' mechansim, giving us
1846 "fianc%C3%A9e". Thus, the first line of the HTTP request will be "GET
1847 /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1". Not all browsers
1848 provide this: IE7 uses the Latin-1 encoding, which is fianc%E9e.
1850 The response header will need to indicate a non-ASCII filename. The actual
1851 mechanism to do this is not clear. For ASCII filenames, the response header
1854 Content-Disposition: attachment; filename="english.txt"
1856 If Tahoe were to enforce the utf-8 convention, it would need to decode the
1857 URL argument into a unicode string, and then encode it back into a sequence
1858 of bytes when creating the response header. One possibility would be to use
1859 unencoded utf-8. Developers suggest that IE7 might accept this:
1861 #1: Content-Disposition: attachment; filename="fianc\xC3\xA9e"
1862 (note, the last four bytes of that line, not including the newline, are
1863 0xC3 0xA9 0x65 0x22)
1865 RFC2231#4 (dated 1997): suggests that the following might work, and some
1866 developers (http://markmail.org/message/dsjyokgl7hv64ig3) have reported that
1867 it is supported by firefox (but not IE7):
1869 #2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
1871 My reading of RFC2616#19.5.1 (which defines Content-Disposition) says that
1872 the filename= parameter is defined to be wrapped in quotes (presumeably to
1873 allow spaces without breaking the parsing of subsequent parameters), which
1876 #3: Content-Disposition: attachment; filename*=utf-8''"fianc%C3%A9e"
1878 However this is contrary to the examples in the email thread listed above.
1880 Developers report that IE7 (when it is configured for UTF-8 URL encoding,
1881 which is not the default in asian countries), will accept:
1883 #4: Content-Disposition: attachment; filename=fianc%C3%A9e
1885 However, for maximum compatibility, Tahoe simply copies bytes from the URL
1886 into the response header, rather than enforcing the utf-8 convention. This
1887 means it does not try to decode the filename from the URL argument, nor does
1888 it encode the filename into the response header.