2 = The Tahoe REST-ful Web API =
4 1. Enabling the web-API port
5 2. Basic Concepts: GET, PUT, DELETE, POST
6 3. URLs, Machine-Oriented Interfaces
7 4. Browser Operations: Human-Oriented Interfaces
8 5. Welcome / Debug / Status pages
9 6. Safety and security issues -- names vs. URIs
13 == Enabling the web-API port ==
15 Every Tahoe node is capable of running a built-in HTTP server. To enable
16 this, just write a port number into a file named "webport" in the node's base
17 directory. For example, writing "8123" into $NODEDIR/webport will cause the
18 node to run a webserver on port 8123.
20 This string is actually a Twisted "strports" specification, meaning you can
21 get more control over the interface to which the server binds by supplying
22 additional arguments. For more details, see the documentation on
23 twisted.application.strports:
24 http://twistedmatrix.com/documents/current/api/twisted.application.strports.html
26 Writing "tcp:8123:interface=127.0.0.1" into $NODEDIR/webport does the same
27 but binds to the loopback interface, ensuring that only the programs on the
28 local host can connect. Using
29 "ssl:8123:privateKey=mykey.pem:certKey=cert.pem" runs an SSL server.
31 This webport can be set when the node is created by passing a --webport
32 option to the 'tahoe create-client' command. By default, the node listens on
33 port 8123, on the loopback (127.0.0.1) interface.
37 As described in architecture.txt, each file and directory in a Tahoe virtual
38 filesystem is referenced by an identifier that combines the designation of
39 the object with the authority to do something with it (such as read or modify
40 the contents). This identifier is called a "read-cap" or "write-cap",
41 depending upon whether it enables read-only or read-write access. These
42 "caps" are also referred to as URIs.
44 The Tahoe web-based API is "REST-ful", meaning it implements the concepts of
45 "REpresentational State Transfer": the original scheme by which the World
46 Wide Web was intended to work. Each object (file or directory) is referenced
47 by a URL that includes the read- or write- cap. HTTP methods (GET, PUT, and
48 DELETE) are used to manipulate these objects. You can think of the URL as a
49 noun, and the method as a verb.
51 In REST, the GET method is used to retrieve information about an object, or
52 to retrieve some representation of the object itself. When the object is a
53 file, the basic GET method will simply return the contents of that file.
54 Other variations (generally implemented by adding query parameters to the
55 URL) will return information about the object, such as metadata. GET
56 operations are required to have no side-effects.
58 PUT is used to upload new objects into the filesystem, or to replace an
59 existing object. DELETE it used to delete objects from the filesystem. Both
60 PUT and DELETE are required to be idempotent: performing the same operation
61 multiple times must have the same side-effects as only performing it once.
63 POST is used for more complicated actions that cannot be expressed as a GET,
64 PUT, or DELETE. POST operations can be thought of as a method call: sending
65 some message to the object referenced by the URL. In Tahoe, POST is also used
66 for operations that must be triggered by an HTML form (including upload and
67 delete), because otherwise a regular web browser has no way to accomplish
70 Tahoe's web API is designed for two different consumers. The first is a
71 program that needs to manipulate the virtual file system. Such programs are
72 expected to use the RESTful interface described above. The second is a human
73 using a standard web browser to work with the filesystem. This user is given
74 a series of HTML pages with links to download files, and forms that use POST
75 actions to upload, rename, and delete files.
79 Tahoe uses a variety of read- and write- caps to identify files and
80 directories. The most common of these is the "immutable file read-cap", which
81 is used for most uploaded files. These read-caps look like the following:
83 URI:CHK:ime6pvkaxuetdfah2p2f35pe54:4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a:3:10:202
85 The next most common is a "directory write-cap", which provides both read and
86 write access to a directory, and look like this:
88 URI:DIR2:djrdkfawoqihigoett4g6auz6a:jx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq
90 There are also "directory read-caps", which start with "URI:DIR2-RO:", and
91 give read-only access to a directory. Finally there are also mutable file
92 read- and write- caps, which start with "URI:SSK", and give access to mutable
95 (later versions of Tahoe will make these strings shorter, and will remove the
96 unfortunate colons, which must be escaped when these caps are embedded in
99 To refer to any Tahoe object through the web API, you simply need to combine
100 a prefix (which indicates the HTTP server to use) with the cap (which
101 indicates which object inside that server to access). Since the default Tahoe
102 webport is 8123, the most common prefix is one that will use a local node
103 listening on this port:
105 http://127.0.0.1:8123/uri/ + $CAP
107 So, to access the directory named above (which happens to be the
108 publically-writable sample directory on the Tahoe test grid, described at
109 http://allmydata.org/trac/tahoe/wiki/TestGrid), the URL would be:
111 http://127.0.0.1:8123/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/
113 (note that the colons in the directory-cap are url-encoded into "%3A"
116 Likewise, to access the file named above, use:
118 http://127.0.0.1:8123/uri/URI%3ACHK%3Aime6pvkaxuetdfah2p2f35pe54%3A4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a%3A3%3A10%3A202
120 In the rest of this document, we'll use "$DIRCAP" as shorthand for a read-cap
121 or write-cap that refers to a directory, and "$FILECAP" to abbreviate a cap
122 that refers to a file (whether mutable or immutable). So those URLs above can
125 http://127.0.0.1:8123/uri/$DIRCAP/
126 http://127.0.0.1:8123/uri/$FILECAP
128 The operation summaries below will abbreviate these further, by eliding the
129 server prefix. They will be displayed like this:
137 Tahoe directories contain named children, just like directories in a regular
138 local filesystem. These children can be either files or subdirectories.
140 If you have a Tahoe URL that refers to a directory, and want to reference a
141 named child inside it, just append the child name to the URL. For example, if
142 our sample directory contains a file named "welcome.txt", we can refer to
145 http://127.0.0.1:8123/uri/$DIRCAP/welcome.txt
147 (or http://127.0.0.1:8123/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/welcome.txt)
149 Multiple levels of subdirectories can be handled this way:
151 http://127.0.0.1:8123/uri/$DIRCAP/tahoe-source/docs/webapi.txt
153 In this document, when we need to refer to a URL that references a file using
154 this child-of-some-directory format, we'll use the following string:
156 /uri/$DIRCAP/[SUBDIRS../]FILENAME
158 The "[SUBDIRS../]" part means that there are zero or more (optional)
159 subdirectory names in the middle of the URL. The "FILENAME" at the end means
160 that this whole URL refers to a file of some sort, rather than to a
163 When we need to refer specifically to a directory in this way, we'll write:
165 /uri/$DIRCAP/[SUBDIRS../]SUBDIR
168 Note that all components of pathnames in URLs are required to be UTF-8
169 encoded, so "resume.doc" (with an acute accent on both E's) would be accessed
172 http://127.0.0.1:8123/uri/$DIRCAP/r%C3%A9sum%C3%A9.doc
174 Also note that the filenames inside upload POST forms are interpreted using
175 whatever character set was provided in the conventional '_charset' field, and
176 defaults to UTF-8 if not otherwise specified. The JSON representation of each
177 directory contains native unicode strings. Tahoe directories are specified to
178 contain unicode filenames, and cannot contain binary strings that are not
179 representable as such.
181 All Tahoe operations that refer to existing files or directories must include
182 a suitable read- or write- cap in the URL: the webapi server won't add one
183 for you. If you don't know the cap, you can't access the file. This allows
184 the security properties of Tahoe caps to be extended across the webapi
187 == Programmatic Operations ==
189 Now that we know how to build URLs that refer to files and directories in a
190 Tahoe virtual filesystem, what sorts of operations can we do with those URLs?
191 This section contains a catalog of GET, PUT, DELETE, and POST operations that
192 can be performed on these URLs. This set of operations are aimed at programs
193 that use HTTP to communicate with a Tahoe node. The next section describes
194 operations that are intended for web browsers.
196 === Reading A File ===
199 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME
201 This will retrieve the contents of the given file. The HTTP response body
202 will contain the sequence of bytes that make up the file.
204 To view files in a web browser, you may want more control over the
205 Content-Type and Content-Disposition headers. Please see the next section
206 "Browser Operations", for details on how to modify these URLs for that
209 === Writing/Uploading A File ===
212 PUT /uri/$DIRCAP/[SUBDIRS../]FILENAME
214 Upload a file, using the data from the HTTP request body, and add whatever
215 child links and subdirectories are necessary to make the file available at
216 the given location. Once this operation succeeds, a GET on the same URL will
217 retrieve the same contents that were just uploaded. This will create any
218 necessary intermediate subdirectories.
220 To use the /uri/$FILECAP form, $FILECAP be a write-cap for a mutable file.
222 In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a
223 writable mutable file, that files contents will be overwritten in-place. If
224 it is a read-cap for a mutable file, an error will occur. If it is an
225 immutable file, the old file will be discarded, and a new one will be put in
228 When creating a new file, if "mutable=true" is in the query arguments, the
229 operation will create a mutable file instead of an immutable one.
231 This returns the file-cap of the resulting file. If a new file was created
232 by this method, the HTTP response code (as dictated by rfc2616) will be set
233 to 201 CREATED. If an existing file was replaced or modified, the response
236 Note that the 'curl -T localfile http://127.0.0.1:8123/uri/$DIRCAP/foo.txt'
237 command can be used to invoke this operation.
241 This uploads a file, and produces a file-cap for the contents, but does not
242 attach the file into the virtual drive. No directories will be modified by
243 this operation. The file-cap is returned as the body of the HTTP response.
245 If "mutable=true" is in the query arguments, the operation will create a
246 mutable file, and return its write-cap in the HTTP respose. The default is
247 to create an immutable file, returning the read-cap as a response.
249 === Creating A New Directory ===
254 Create a new empty directory and return its write-cap as the HTTP response
255 body. This does not make the newly created directory visible from the
256 virtual drive. The "PUT" operation is provided for backwards compatibility:
257 new code should use POST.
259 POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir
260 PUT /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir
262 Create new directories as necessary to make sure that the named target
263 ($DIRCAP/SUBDIRS../SUBDIR) is a directory. This will create additional
264 intermediate directories as necessary. If the named target directory already
265 exists, this will make no changes to it.
267 This will return an error if a blocking file is present at any of the parent
268 names, preventing the server from creating the necessary parent directory.
270 The write-cap of the new directory will be returned as the HTTP response
273 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME
275 Create a new empty directory and attach it to the given existing directory.
276 This will create additional intermediate directories as necessary.
278 The URL of this form points to the parent of the bottom-most new directory,
279 whereas the previous form has a URL that points directly to the bottom-most
282 === Get Information About A File Or Directory (as JSON) ===
284 GET /uri/$FILECAP?t=json
285 GET /uri/$DIRCAP?t=json
286 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json
287 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json
289 This returns a machine-parseable JSON-encoded description of the given
290 object. The JSON always contains a list, and the first element of the list
291 is always a flag that indicates whether the referenced object is a file or a
292 directory. If it is a file, then the information includes file size and URI,
295 GET /uri/$FILECAP?t=json :
296 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json :
298 [ "filenode", { "ro_uri": file_uri,
301 "metadata": {"ctime": 1202777696.7564139,
302 "mtime": 1202777696.7564139
306 If it is a directory, then it includes information about the children of
307 this directory, as a mapping from child name to a set of data about the
308 child (the same data that would appear in a corresponding GET?t=json of the
309 child itself). The child entries also include metadata about each child,
310 including creation- and modification- timestamps. The output looks like
313 GET /uri/$DIRCAP?t=json :
314 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json :
316 [ "dirnode", { "rw_uri": read_write_uri,
317 "ro_uri": read_only_uri,
320 "foo.txt": [ "filenode", { "ro_uri": uri,
323 "ctime": 1202777696.7564139,
324 "mtime": 1202777696.7564139
327 "subdir": [ "dirnode", { "rw_uri": rwuri,
330 "ctime": 1202778102.7589991,
331 "mtime": 1202778111.2160511,
336 In the above example, note how 'children' is a dictionary in which the keys
337 are child names and the values depend upon whether the child is a file or a
338 directory. The value is mostly the same as the JSON representation of the
339 child object (except that directories do not recurse -- the "children"
340 entry of the child is omitted, and the directory view includes the metadata
341 that is stored on the directory edge).
343 Then the rw_uri field will be present in the information about a directory
344 if and only if you have read-write access to that directory,
347 === Attaching an existing File or Directory by its read- or write- cap ===
349 PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri
351 This attaches a child object (either a file or directory) to a specified
352 location in the virtual filesystem. The child object is referenced by its
353 read- or write- cap, as provided in the HTTP request body. This will create
354 intermediate directories as necessary.
356 This is similar to a UNIX hardlink: by referencing a previously-uploaded
357 file (or previously-created directory) instead of uploading/creating a new
358 one, you can create two references to the same object.
360 The read- or write- cap of the child is provided in the body of the HTTP
361 request, and this same cap is returned in the response body.
363 The default behavior is to overwrite any existing object at the same
364 location. To prevent this (and make the operation return an error instead of
365 overwriting), add a "replace=false" argument, as "?t=uri&replace=false".
366 With replace=false, this operation will return an HTTP 409 "Conflict" error
367 if there is already an object at the given location, rather than overwriting
368 the existing object. Note that "true", "t", and "1" are all synonyms for
369 "True", and "false", "f", and "0" are synonyms for "False". the parameter is
372 === Deleting a File or Directory ===
374 DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME
376 This removes the given name from its parent directory. CHILDNAME is the
377 name to be removed, and $DIRCAP/SUBDIRS.. indicates the directory that will
380 Note that this does not actually delete the file or directory that the name
381 points to from the tahoe grid -- it only removes the named reference from
382 this directory. If there are other names in this directory or in other
383 directories that point to the resource, then it will remain accessible
384 through those paths. Even if all names pointing to this object are removed
385 from their parent directories, then someone with possession of its read-cap
386 can continue to access the object through that cap.
388 The object will only become completely unreachable once 1: there are no
389 reachable directories that reference it, and 2: nobody is holding a read-
390 or write- cap to the object. (This behavior is very similar to the way
391 hardlinks and anonymous files work in traditional unix filesystems).
393 This operation will not modify more than a single directory. Intermediate
394 directories which were implicitly created by PUT or POST methods will *not*
395 be automatically removed by DELETE.
397 This method returns the file- or directory- cap of the object that was just
400 == Browser Operations ==
402 This section describes the HTTP operations that provide support for humans
403 running a web browser. Most of these operations use HTML forms that use POST
404 to drive the Tahoe node.
406 Note that for all POST operations, the arguments listed can be provided
407 either as URL query arguments or as form body fields. URL query arguments are
408 separated from the main URL by "?", and from each other by "&". For example,
409 "POST /uri/$DIRCAP?t=upload&mutable=true". Form body fields are usually
410 specified by using <input type="hidden"> elements. For clarity, the
411 descriptions below display the most significant arguments as URL query args.
413 === Viewing A Directory (as HTML) ===
415 GET /uri/$DIRCAP/[SUBDIRS../]
417 This returns an HTML page, intended to be displayed to a human by a web
418 browser, which contains HREF links to all files and directories reachable
419 from this directory. These HREF links do not have a t= argument, meaning
420 that a human who follows them will get pages also meant for a human. It also
421 contains forms to upload new files, and to delete files and directories.
422 Those forms use POST methods to do their job.
424 === Viewing/Downloading a File ===
427 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME
429 This will retrieve the contents of the given file. The HTTP response body
430 will contain the sequence of bytes that make up the file.
432 If you want the HTTP response to include a useful Content-Type header,
433 either use the second form (which starts with a $DIRCAP), or add a
434 "filename=foo" query argument, like "GET /uri/$FILECAP?filename=foo.jpg".
435 The bare "GET /uri/$FILECAP" does not give the Tahoe node enough information
436 to determine a Content-Type (since Tahoe immutable files are merely
437 sequences of bytes, not typed+named file objects).
439 If the URL has both filename= and "save=true" in the query arguments, then
440 the server to add a "Content-Disposition: attachment" header, along with a
441 filename= parameter. When a user clicks on such a link, most browsers will
442 offer to let the user save the file instead of displaying it inline (indeed,
443 most browsers will refuse to display it inline). "true", "t", "1", and other
444 case-insensitive equivalents are all treated the same.
446 Character-set handling in URLs and HTTP headers is a dubious art[1]. For
447 maximum compatibility, Tahoe simply copies the bytes from the filename=
448 argument into the Content-Disposition header's filename= parameter, without
449 trying to interpret them in any particular way.
452 GET /named/$FILECAP/FILENAME
454 This is an alternate download form which makes it easier to get the correct
455 filename. The Tahoe server will provide the contents of the given file, with
456 a Content-Type header derived from the given filename. This form is used to
457 get browsers to use the "Save Link As" feature correctly, and also helps
458 command-line tools like "wget" and "curl" use the right filename. Note that
459 this form can *only* be used with file caps; it is an error to use a
460 directory cap after the /named/ prefix.
462 === Creating a Directory ===
466 This creates a new directory, but does not attach it to the virtual
469 If a "redirect_to_result=true" argument is provided, then the HTTP response
470 will cause the web browser to be redirected to a /uri/$DIRCAP page that
471 gives access to the newly-created directory. If you bookmark this page,
472 you'll be able to get back to the directory again in the future. This is the
473 recommended way to start working with a Tahoe server: create a new unlinked
474 directory (using redirect_to_result=true), then bookmark the resulting
475 /uri/$DIRCAP page. There is a "Create Directory" button on the Welcome page
476 to invoke this action.
478 If "redirect_to_result=true" is not provided (or is given a value of
479 "false"), then the HTTP response body will simply be the write-cap of the
482 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME
484 This creates a new directory as a child of the designated SUBDIR. This will
485 create additional intermediate directories as necessary.
487 If a "when_done=URL" argument is provided, the HTTP response will cause the
488 web browser to redirect to the given URL. This provides a convenient way to
489 return the browser to the directory that was just modified. Without a
490 when_done= argument, the HTTP response will simply contain the write-cap of
491 the directory that was just created.
494 === Uploading a File ===
498 This uploads a file, and produces a file-cap for the contents, but does not
499 attach the file into the virtual drive. No directories will be modified by
502 The file must be provided as the "file" field of an HTML encoded form body,
503 produced in response to an HTML form like this:
504 <form action="/uri" method="POST" enctype="multipart/form-data">
505 <input type="hidden" name="t" value="upload" />
506 <input type="file" name="file" />
507 <input type="submit" value="Upload Unlinked" />
510 If a "when_done=URL" argument is provided, the response body will cause the
511 browser to redirect to the given URL. If the when_done= URL has the string
512 "%(uri)s" in it, that string will be replaced by a URL-escaped form of the
513 newly created file-cap. (Note that without this substitution, there is no
514 way to access the file that was just uploaded).
516 The default (in the absence of when_done=) is to return an HTML page that
517 describes the results of the upload. This page will contain information
518 about which storage servers were used for the upload, how long each
521 If a "mutable=true" argument is provided, the operation will create a
522 mutable file, and the response body will contain the write-cap instead of
523 the upload results page. The default is to create an immutable file,
524 returning the upload results page as a response.
527 POST /uri/$DIRCAP/[SUBDIRS../]?t=upload
529 This uploads a file, and attaches it as a new child of the given directory.
530 The file must be provided as the "file" field of an HTML encoded form body,
531 produced in response to an HTML form like this:
532 <form action="." method="POST" enctype="multipart/form-data">
533 <input type="hidden" name="t" value="upload" />
534 <input type="file" name="file" />
535 <input type="submit" value="Upload" />
538 A "name=" argument can be provided to specify the new child's name,
539 otherwise it will be taken from the "filename" field of the upload form
540 (most web browsers will copy the last component of the original file's
541 pathname into this field). To avoid confusion, name= is not allowed to
544 If there is already a child with that name, and it is a mutable file, then
545 its contents are replaced with the data being uploaded. If it is not a
546 mutable file, the default behavior is to remove the existing child before
547 creating a new one. To prevent this (and make the operation return an error
548 instead of overwriting the old child), add a "replace=false" argument, as
549 "?t=upload&replace=false". With replace=false, this operation will return an
550 HTTP 409 "Conflict" error if there is already an object at the given
551 location, rather than overwriting the existing object. Note that "true",
552 "t", and "1" are all synonyms for "True", and "false", "f", and "0" are
553 synonyms for "False". the parameter is case-insensitive.
555 This will create additional intermediate directories as necessary, although
556 since it is expected to be triggered by a form that was retrieved by "GET
557 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
560 If a "mutable=true" argument is provided, any new file that is created will
561 be a mutable file instead of an immutable one. <input type="checkbox"
562 name="mutable" /> will give the user a way to set this option.
564 If a "when_done=URL" argument is provided, the HTTP response will cause the
565 web browser to redirect to the given URL. This provides a convenient way to
566 return the browser to the directory that was just modified. Without a
567 when_done= argument, the HTTP response will simply contain the file-cap of
568 the file that was just uploaded (a write-cap for mutable files, or a
569 read-cap for immutable files).
571 POST /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=upload
573 This also uploads a file and attaches it as a new child of the given
574 directory. It is a slight variant of the previous operation, as the URL
575 refers to the target file rather than the parent directory. It is otherwise
576 identical: this accepts mutable= and when_done= arguments too.
578 POST /uri/$FILECAP?t=upload
580 === Attaching An Existing File Or Directory (by URI) ===
582 POST /uri/$DIRCAP/[SUBDIRS../]?t=uri&name=CHILDNAME&uri=CHILDCAP
584 This attaches a given read- or write- cap "CHILDCAP" to the designated
585 directory, with a specified child name. This behaves much like the PUT t=uri
586 operation, and is a lot like a UNIX hardlink.
588 This will create additional intermediate directories as necessary, although
589 since it is expected to be triggered by a form that was retrieved by "GET
590 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
593 === Deleting A Child ===
595 POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME
597 This instructs the node to delete a child object (file or subdirectory) from
598 the given directory. Note that the entire subtree is removed. This is
599 somewhat like "rm -rf" (from the point of view of the parent), but other
600 references into the subtree will see that the child subdirectories are not
601 modified by this operation. Only the link from the given directory to its
604 === Renaming A Child ===
606 POST /uri/$DIRCAP/[SUBDIRS../]?t=rename&from_name=OLD&to_name=NEW
608 This instructs the node to rename a child of the given directory. This is
609 exactly the same as removing the child, then adding the same child-cap under
610 the new name. This operation cannot move the child to a different directory.
612 This operation will replace any existing child of the new name, making it
613 behave like the UNIX "mv -f" command.
615 === Other Utilities ===
619 This causes a redirect to /uri/$CAP, and retains any additional query
620 arguments (like filename= or save=). This is for the convenience of web
621 forms which allow the user to paste in a read- or write- cap (obtained
622 through some out-of-band channel, like IM or email).
624 Note that this form merely redirects to the specific file or directory
625 indicated by the $CAP: unlike the GET /uri/$DIRCAP form, you cannot
626 traverse to children by appending additional path segments to the URL.
628 GET /uri/$DIRCAP/[SUBDIRS../]?t=rename-form&name=$CHILDNAME
630 This provides a useful facility to browser-based user interfaces. It
631 returns a page containing a form targetting the "POST $DIRCAP t=rename"
632 functionality described above, with the provided $CHILDNAME present in the
633 'from_name' field of that form. I.e. this presents a form offering to
634 rename $CHILDNAME, requesting the new name, and submitting POST rename.
636 GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri
638 This returns the file- or directory- cap for the specified object.
640 GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=readonly-uri
642 This returns a read-only file- or directory- cap for the specified object.
643 If the object is an immutable file, this will return the same value as
646 === Debugging and Testing Features ===
648 These URLs are less-likely to be helpful to the casual Tahoe user, and are
649 mainly intended for developers.
653 This triggers the FileChecker to determine the current "health" of the
654 given file or directory, by counting how many shares are available. The
655 page that is returned will display the results. This can be used as a "show
656 me detailed information about this file" page.
658 If a when_done=url argument is provided, the return value will be a redirect
659 to that URL instead of the checker results.
661 If a return_to=url argument is provided, the returned page will include a
662 link to the given URL entitled "Return to the parent directory".
664 If a verify=true argument is provided, the node will perform a more
665 intensive check, downloading and verifying every single bit of every share.
667 POST $URL?t=deep-check
669 This triggers a recursive walk of all files and directories reachable from
670 the target, performing a check on each one just like t=check. The result
671 page will contain a summary of the results, including details on any
672 file/directory that was not fully healthy.
674 t=deep-check is most useful to invoke on a directory. If invoked on a file,
675 it will just check that single object. The recursive walker will deal with
678 This accepts the same verify=, when_done=, and return_to= arguments as
681 Be aware that this can take a long time: perhaps a second per object.
683 GET $DIRURL?t=manifest
685 Return an HTML-formatted manifest of the given directory, for debugging.
687 GET $DIRURL?t=deep-size
689 Return a number (in bytes) containing the sum of the filesize of all
690 immutable files reachable from the given directory. This is a rough lower
691 bound of the total space consumed by this subtree. It does not include
692 space consumed by directories or immutable files, nor does it take
693 expansion or encoding overhead into account. Later versions of the code may
694 improve this estimate upwards.
696 GET $DIRURL?t=deep-stats
698 Return a JSON-encoded dictionary that lists interesting statistics about
699 the set of all files and directories reachable from the given directory:
701 count-immutable-files: count of how many CHK files are in the set
702 count-mutable-files: same, for mutable files (does not include directories)
703 count-literal-files: same, for LIT files (data contained inside the URI)
704 count-files: sum of the above three
705 count-directories: count of directories
706 size-immutable-files: total bytes for all CHK files in the set, =deep-size
707 size-mutable-files (TODO): same, for current version of all mutable files
708 size-literal-files: same, for LIT files
709 size-directories: size of directories (includes size-literal-files)
710 size-files-histogram: list of (minsize, maxsize, count) buckets,
711 with a histogram of filesizes, 5dB/bucket,
712 for both literal and immutable files
713 largest-directory: number of children in the largest directory
714 largest-immutable-file: number of bytes in the largest CHK file
716 size-mutable-files is not implemented, because it would require extra
717 queries to each mutable file to get their size. This may be implemented in
720 Assuming no sharing, the basic space consumed by a single root directory is
721 the sum of size-immutable-files, size-mutable-files, and size-directories.
722 The actual disk space used by the shares is larger, because of the
723 following sources of overhead:
726 expansion due to erasure coding
727 share management data (leases)
728 backend (ext3) minimum block size
730 == Other Useful Pages ==
732 The portion of the web namespace that begins with "/uri" (and "/named") is
733 dedicated to giving users (both humans and programs) access to the Tahoe
734 virtual filesystem. The rest of the namespace provides status information
735 about the state of the Tahoe node.
737 GET / (the root page)
739 This is the "Welcome Page", and contains a few distinct sections:
741 Node information: library versions, local nodeid, services being provided.
743 Filesystem Access Forms: create a new directory, view a file/directory by
744 URI, upload a file (unlinked), download a file by
747 Grid Status: introducer information, helper information, connected storage
752 This page lists all active uploads and downloads, and contains a short list
753 of recent upload/download operations. Each operation has a link to a page
754 that describes file sizes, servers that were involved, and the time consumed
755 in each phase of the operation.
759 This page provides a basic tool to predict the likely storage and bandwidth
760 requirements of a large Tahoe grid. It provides forms to input things like
761 total number of users, number of files per user, average file size, number
762 of servers, expansion ratio, hard drive failure rate, etc. It then provides
763 numbers like how many disks per server will be needed, how many read
764 operations per second should be expected, and the likely MTBF for files in
765 the grid. This information is very preliminary, and the model upon which it
766 is based still needs a lot of work.
770 If the node is running a helper (i.e. if "$BASEDIR/run_helper" is
771 non-empty), then this page will provide a list of all the helper operations
772 currently in progress. If "?t=json" is added to the URL, it will return a
773 JSON-formatted list of helper statistics, which can then be used to produce
774 graphs to indicate how busy the helper is.
778 This page provides "node statistics", which are collected from a variety of
781 load_monitor: every second, the node schedules a timer for one second in
782 the future, then measures how late the subsequent callback
783 is. The "load_average" is this tardiness, measured in
784 seconds, averaged over the last minute. It is an indication
785 of a busy node, one which is doing more work than can be
786 completed in a timely fashion. The "max_load" value is the
787 highest value that has been seen in the last 60 seconds.
789 cpu_monitor: every minute, the node uses time.clock() to measure how much
790 CPU time it has used, and it uses this value to produce
791 1min/5min/15min moving averages. These values range from 0%
792 (0.0) to 100% (1.0), and indicate what fraction of the CPU
793 has been used by the Tahoe node. Not all operating systems
794 provide meaningful data to time.clock(): they may report 100%
795 CPU usage at all times.
797 uploader: this counts how many immutable files (and bytes) have been
798 uploaded since the node was started
800 downloader: this counts how many immutable files have been downloaded
801 since the node was started
803 publishes: this counts how many mutable files (including directories) have
804 been modified since the node was started
806 retrieves: this counts how many mutable files (including directories) have
807 been read since the node was started
809 There are other statistics that are tracked by the node. The "raw stats"
810 section shows a formatted dump of all of them.
812 By adding "?t=json" to the URL, the node will return a JSON-formatted
813 dictionary of stats values, which can be used by other tools to produce
814 graphs of node behavior. The misc/munin/ directory in the source
815 distribution provides some tools to produce these graphs.
817 GET / (introducer status)
819 For Introducer nodes, the welcome page displays information about both
820 clients and servers which are connected to the introducer. Servers make
821 "service announcements", and these are listed in a table. Clients will
822 subscribe to hear about service announcements, and these subscriptions are
823 listed in a separate table. Both tables contain information about what
824 version of Tahoe is being run by the remote node, their advertised and
825 outbound IP addresses, their nodeid and nickname, and how long they have
828 By adding "?t=json" to the URL, the node will return a JSON-formatted
829 dictionary of stats values, which can be used to produce graphs of connected
833 == safety and security issues -- names vs. URIs ==
835 Summary: use explicit file- and dir- caps whenever possible, to reduce the
836 potential for surprises when the virtual drive is changed while you aren't
839 The vdrive provides a mutable filesystem, but the ways that the filesystem
840 can change are limited. The only thing that can change is that the mapping
841 from child names to child objects that each directory contains can be changed
842 by adding a new child name pointing to an object, removing an existing child
843 name, or changing an existing child name to point to a different object.
845 Obviously if you query tahoe for information about the filesystem and then
846 act upon the filesystem (such as by getting a listing of the contents of a
847 directory and then adding a file to the directory), then the filesystem might
848 have been changed after you queried it and before you acted upon it.
849 However, if you use the URI instead of the pathname of an object when you act
850 upon the object, then the only change that can happen is when the object is a
851 directory then the set of child names it has might be different. If, on the
852 other hand, you act upon the object using its pathname, then a different
853 object might be in that place, which can result in more kinds of surprises.
855 For example, suppose you are writing code which recursively downloads the
856 contents of a directory. The first thing your code does is fetch the listing
857 of the contents of the directory. For each child that it fetched, if that
858 child is a file then it downloads the file, and if that child is a directory
859 then it recurses into that directory. Now, if the download and the recurse
860 actions are performed using the child's name, then the results might be
861 wrong, because for example a child name that pointed to a sub-directory when
862 you listed the directory might have been changed to point to a file (in which
863 case your attempt to recurse into it would result in an error and the file
864 would be skipped), or a child name that pointed to a file when you listed the
865 directory might now point to a sub-directory (in which case your attempt to
866 download the child would result in a file containing HTML text describing the
869 If your recursive algorithm uses the uri of the child instead of the name of
870 the child, then those kinds of mistakes just can't happen. Note that both the
871 child's name and the child's URI are included in the results of listing the
872 parent directory, so it isn't any harder to use the URI for this purpose.
874 In general, use names if you want "whatever object (whether file or
875 directory) is found by following this name (or sequence of names) when my
876 request reaches the server". Use URIs if you want "this particular object".
878 == Concurrency Issues ==
880 Tahoe uses both mutable and immutable files. Mutable files can be created
881 explicitly by doing an upload with ?mutable=true added, or implicitly by
882 creating a new directory (since a directory is just a special way to
883 interpret a given mutable file).
885 Mutable files suffer from the same consistency-vs-availability tradeoff that
886 all distributed data storage systems face. It is not possible to
887 simultaneously achieve perfect consistency and perfect availability in the
888 face of network partitions (servers being unreachable or faulty).
890 Tahoe tries to achieve a reasonable compromise, but there is a basic rule in
891 place, known as the Prime Coordination Directive: "Don't Do That". What this
892 means is that if write-access to a mutable file is available to several
893 parties, then those parties are responsible for coordinating their activities
894 to avoid multiple simultaneous updates. This could be achieved by having
895 these parties talk to each other and using some sort of locking mechanism, or
896 by serializing all changes through a single writer.
898 The consequences of performing uncoordinated writes can vary. Some of the
899 writers may lose their changes, as somebody else wins the race condition. In
900 many cases the file will be left in an "unhealthy" state, meaning that there
901 are not as many redundant shares as we would like (reducing the reliability
902 of the file against server failures). In the worst case, the file can be left
903 in such an unhealthy state that no version is recoverable, even the old ones.
904 It is this small possibility of data loss that prompts us to issue the Prime
905 Coordination Directive.
907 Tahoe nodes implement internal serialization to make sure that a single Tahoe
908 node cannot conflict with itself. For example, it is safe to issue two
909 directory modification requests to a single tahoe node's webapi server at the
910 same time, because the Tahoe node will internally delay one of them until
911 after the other has finished being applied. (This feature was introduced in
912 Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing
913 web requests themselves).
915 For more details, please see the "Consistency vs Availability" and "The Prime
916 Coordination Directive" sections of mutable.txt, in the same directory as
920 [1]: URLs and HTTP and UTF-8, Oh My
922 HTTP does not provide a mechanism to specify the character set used to
923 encode non-ascii names in URLs (rfc2396#2.1). We prefer the convention that
924 the filename= argument shall be a URL-encoded UTF-8 encoded unicode object.
925 For example, suppose we want to provoke the server into using a filename of
926 "f i a n c e-acute e" (i.e. F I A N C U+00E9 E). The UTF-8 encoding of this
927 is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\xC3\xA9e", as python's
928 repr() function would show). To encode this into a URL, the non-printable
929 characters must be escaped with the urlencode '%XX' mechansim, giving us
930 "fianc%C3%A9e". Thus, the first line of the HTTP request will be "GET
931 /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1". Not all browsers
932 provide this: IE7 uses the Latin-1 encoding, which is fianc%E9e.
934 The response header will need to indicate a non-ASCII filename. The actual
935 mechanism to do this is not clear. For ASCII filenames, the response header
938 Content-Disposition: attachment; filename="english.txt"
940 If Tahoe were to enforce the utf-8 convention, it would need to decode the
941 URL argument into a unicode string, and then encode it back into a sequence
942 of bytes when creating the response header. One possibility would be to use
943 unencoded utf-8. Developers suggest that IE7 might accept this:
945 #1: Content-Disposition: attachment; filename="fianc\xC3\xA9e"
946 (note, the last four bytes of that line, not including the newline, are
949 RFC2231#4 (dated 1997): suggests that the following might work, and some
950 developers (http://markmail.org/message/dsjyokgl7hv64ig3) have reported that
951 it is supported by firefox (but not IE7):
953 #2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
955 My reading of RFC2616#19.5.1 (which defines Content-Disposition) says that
956 the filename= parameter is defined to be wrapped in quotes (presumeably to
957 allow spaces without breaking the parsing of subsequent parameters), which
960 #3: Content-Disposition: attachment; filename*=utf-8''"fianc%C3%A9e"
962 However this is contrary to the examples in the email thread listed above.
964 Developers report that IE7 (when it is configured for UTF-8 URL encoding,
965 which is not the default in asian countries), will accept:
967 #4: Content-Disposition: attachment; filename=fianc%C3%A9e
969 However, for maximum compatibility, Tahoe simply copies bytes from the URL
970 into the response header, rather than enforcing the utf-8 convention. This
971 means it does not try to decode the filename from the URL argument, nor does
972 it encode the filename into the response header.