2 = The Tahoe REST-ful Web API =
4 1. Enabling the web-API port
5 2. Basic Concepts: GET, PUT, DELETE, POST
6 3. URLs, Machine-Oriented Interfaces
7 4. Browser Operations: Human-Oriented Interfaces
8 5. Welcome / Debug / Status pages
9 6. Static Files in /public_html
10 7. Safety and security issues -- names vs. URIs
14 == Enabling the web-API port ==
16 Every Tahoe node is capable of running a built-in HTTP server. To enable
17 this, just write a port number into the "[node]web.port" line of your node's
18 tahoe.cfg file. For example, writing "web.port = 3456" into the "[node]"
19 section of $NODEDIR/tahoe.cfg will cause the node to run a webserver on port
22 This string is actually a Twisted "strports" specification, meaning you can
23 get more control over the interface to which the server binds by supplying
24 additional arguments. For more details, see the documentation on
25 twisted.application.strports:
26 http://twistedmatrix.com/documents/current/api/twisted.application.strports.html
28 Writing "tcp:3456:interface=127.0.0.1" into the web.port line does the same
29 but binds to the loopback interface, ensuring that only the programs on the
30 local host can connect. Using
31 "ssl:3456:privateKey=mykey.pem:certKey=cert.pem" runs an SSL server.
33 This webport can be set when the node is created by passing a --webport
34 option to the 'tahoe create-client' command. By default, the node listens on
35 port 3456, on the loopback (127.0.0.1) interface.
39 As described in architecture.txt, each file and directory in a Tahoe virtual
40 filesystem is referenced by an identifier that combines the designation of
41 the object with the authority to do something with it (such as read or modify
42 the contents). This identifier is called a "read-cap" or "write-cap",
43 depending upon whether it enables read-only or read-write access. These
44 "caps" are also referred to as URIs.
46 The Tahoe web-based API is "REST-ful", meaning it implements the concepts of
47 "REpresentational State Transfer": the original scheme by which the World
48 Wide Web was intended to work. Each object (file or directory) is referenced
49 by a URL that includes the read- or write- cap. HTTP methods (GET, PUT, and
50 DELETE) are used to manipulate these objects. You can think of the URL as a
51 noun, and the method as a verb.
53 In REST, the GET method is used to retrieve information about an object, or
54 to retrieve some representation of the object itself. When the object is a
55 file, the basic GET method will simply return the contents of that file.
56 Other variations (generally implemented by adding query parameters to the
57 URL) will return information about the object, such as metadata. GET
58 operations are required to have no side-effects.
60 PUT is used to upload new objects into the filesystem, or to replace an
61 existing object. DELETE it used to delete objects from the filesystem. Both
62 PUT and DELETE are required to be idempotent: performing the same operation
63 multiple times must have the same side-effects as only performing it once.
65 POST is used for more complicated actions that cannot be expressed as a GET,
66 PUT, or DELETE. POST operations can be thought of as a method call: sending
67 some message to the object referenced by the URL. In Tahoe, POST is also used
68 for operations that must be triggered by an HTML form (including upload and
69 delete), because otherwise a regular web browser has no way to accomplish
70 these tasks. In general, everything that can be done with a PUT or DELETE can
71 also be done with a POST.
73 Tahoe's web API is designed for two different consumers. The first is a
74 program that needs to manipulate the virtual file system. Such programs are
75 expected to use the RESTful interface described above. The second is a human
76 using a standard web browser to work with the filesystem. This user is given
77 a series of HTML pages with links to download files, and forms that use POST
78 actions to upload, rename, and delete files.
82 Tahoe uses a variety of read- and write- caps to identify files and
83 directories. The most common of these is the "immutable file read-cap", which
84 is used for most uploaded files. These read-caps look like the following:
86 URI:CHK:ime6pvkaxuetdfah2p2f35pe54:4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a:3:10:202
88 The next most common is a "directory write-cap", which provides both read and
89 write access to a directory, and look like this:
91 URI:DIR2:djrdkfawoqihigoett4g6auz6a:jx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq
93 There are also "directory read-caps", which start with "URI:DIR2-RO:", and
94 give read-only access to a directory. Finally there are also mutable file
95 read- and write- caps, which start with "URI:SSK", and give access to mutable
98 (later versions of Tahoe will make these strings shorter, and will remove the
99 unfortunate colons, which must be escaped when these caps are embedded in
102 To refer to any Tahoe object through the web API, you simply need to combine
103 a prefix (which indicates the HTTP server to use) with the cap (which
104 indicates which object inside that server to access). Since the default Tahoe
105 webport is 3456, the most common prefix is one that will use a local node
106 listening on this port:
108 http://127.0.0.1:3456/uri/ + $CAP
110 So, to access the directory named above (which happens to be the
111 publically-writable sample directory on the Tahoe test grid, described at
112 http://allmydata.org/trac/tahoe/wiki/TestGrid), the URL would be:
114 http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/
116 (note that the colons in the directory-cap are url-encoded into "%3A"
119 Likewise, to access the file named above, use:
121 http://127.0.0.1:3456/uri/URI%3ACHK%3Aime6pvkaxuetdfah2p2f35pe54%3A4btz54xk3tew6nd4y2ojpxj4m6wxjqqlwnztgre6gnjgtucd5r4a%3A3%3A10%3A202
123 In the rest of this document, we'll use "$DIRCAP" as shorthand for a read-cap
124 or write-cap that refers to a directory, and "$FILECAP" to abbreviate a cap
125 that refers to a file (whether mutable or immutable). So those URLs above can
128 http://127.0.0.1:3456/uri/$DIRCAP/
129 http://127.0.0.1:3456/uri/$FILECAP
131 The operation summaries below will abbreviate these further, by eliding the
132 server prefix. They will be displayed like this:
140 Tahoe directories contain named children, just like directories in a regular
141 local filesystem. These children can be either files or subdirectories.
143 If you have a Tahoe URL that refers to a directory, and want to reference a
144 named child inside it, just append the child name to the URL. For example, if
145 our sample directory contains a file named "welcome.txt", we can refer to
148 http://127.0.0.1:3456/uri/$DIRCAP/welcome.txt
150 (or http://127.0.0.1:3456/uri/URI%3ADIR2%3Adjrdkfawoqihigoett4g6auz6a%3Ajx5mplfpwexnoqff7y5e4zjus4lidm76dcuarpct7cckorh2dpgq/welcome.txt)
152 Multiple levels of subdirectories can be handled this way:
154 http://127.0.0.1:3456/uri/$DIRCAP/tahoe-source/docs/webapi.txt
156 In this document, when we need to refer to a URL that references a file using
157 this child-of-some-directory format, we'll use the following string:
159 /uri/$DIRCAP/[SUBDIRS../]FILENAME
161 The "[SUBDIRS../]" part means that there are zero or more (optional)
162 subdirectory names in the middle of the URL. The "FILENAME" at the end means
163 that this whole URL refers to a file of some sort, rather than to a
166 When we need to refer specifically to a directory in this way, we'll write:
168 /uri/$DIRCAP/[SUBDIRS../]SUBDIR
171 Note that all components of pathnames in URLs are required to be UTF-8
172 encoded, so "resume.doc" (with an acute accent on both E's) would be accessed
175 http://127.0.0.1:3456/uri/$DIRCAP/r%C3%A9sum%C3%A9.doc
177 Also note that the filenames inside upload POST forms are interpreted using
178 whatever character set was provided in the conventional '_charset' field, and
179 defaults to UTF-8 if not otherwise specified. The JSON representation of each
180 directory contains native unicode strings. Tahoe directories are specified to
181 contain unicode filenames, and cannot contain binary strings that are not
182 representable as such.
184 All Tahoe operations that refer to existing files or directories must include
185 a suitable read- or write- cap in the URL: the wapi server won't add one
186 for you. If you don't know the cap, you can't access the file. This allows
187 the security properties of Tahoe caps to be extended across the wapi
190 == Slow Operations, Progress, and Cancelling ==
192 Certain operations can be expected to take a long time. The "t=deep-check",
193 described below, will recursively visit every file and directory reachable
194 from a given starting point, which can take minutes or even hours for
195 extremely large directory structures. A single long-running HTTP request is a
196 fragile thing: proxies, NAT boxes, browsers, and users may all grow impatient
197 with waiting and give up on the connection.
199 For this reason, long-running operations have an "operation handle", which
200 can be used to poll for status/progress messages while the operation
201 proceeds. This handle can also be used to cancel the operation. These handles
202 are created by the client, and passed in as a an "ophandle=" query argument
203 to the POST or PUT request which starts the operation. The following
204 operations can then be used to retrieve status:
206 GET /operations/$HANDLE?output=HTML (with or without t=status)
207 GET /operations/$HANDLE?output=JSON (same)
209 These two retrieve the current status of the given operation. Each operation
210 presents a different sort of information, but in general the page retrieved
213 * whether the operation is complete, or if it is still running
214 * how much of the operation is complete, and how much is left, if possible
216 Note that the final status output can be quite large: a deep-manifest of a
217 directory structure with 300k directories and 200k unique files is about
218 275MB of JSON, and might take two minutes to generate. For this reason, the
219 full status is not provided until the operation has completed.
221 The HTML form will include a meta-refresh tag, which will cause a regular
222 web browser to reload the status page about 60 seconds later. This tag will
223 be removed once the operation has completed.
225 There may be more status information available under
226 /operations/$HANDLE/$ETC : i.e., the handle forms the root of a URL space.
228 POST /operations/$HANDLE?t=cancel
230 This terminates the operation, and returns an HTML page explaining what was
231 cancelled. If the operation handle has already expired (see below), this
232 POST will return a 404, which indicates that the operation is no longer
233 running (either it was completed or terminated). The response body will be
234 the same as a GET /operations/$HANDLE on this operation handle, and the
235 handle will be expired immediately afterwards.
237 The operation handle will eventually expire, to avoid consuming an unbounded
238 amount of memory. The handle's time-to-live can be reset at any time, by
239 passing a retain-for= argument (with a count of seconds) to either the
240 initial POST that starts the operation, or the subsequent GET request which
241 asks about the operation. For example, if a 'GET
242 /operations/$HANDLE?output=JSON&retain-for=600' query is performed, the
243 handle will remain active for 600 seconds (10 minutes) after the GET was
246 In addition, if the GET includes a release-after-complete=True argument, and
247 the operation has completed, the operation handle will be released
250 If a retain-for= argument is not used, the default handle lifetimes are:
252 * handles will remain valid at least until their operation finishes
253 * uncollected handles for finished operations (i.e. handles for operations
254 which have finished but for which the GET page has not been accessed since
255 completion) will remain valid for one hour, or for the total time consumed
256 by the operation, whichever is greater.
257 * collected handles (i.e. the GET page has been retrieved at least once
258 since the operation completed) will remain valid for ten minutes.
260 Many "slow" operations can begin to use unacceptable amounts of memory when
261 operation on large directory structures. The memory usage increases when the
262 ophandle is polled, as the results must be copied into a JSON string, sent
263 over the wire, then parsed by a client. So, as an alternative, many "slow"
264 operations have streaming equivalents. These equivalents do not use operation
265 handles. Instead, they emit line-oriented status results immediately. Client
266 code can cancel the operation by simply closing the HTTP connection.
268 == Programmatic Operations ==
270 Now that we know how to build URLs that refer to files and directories in a
271 Tahoe virtual filesystem, what sorts of operations can we do with those URLs?
272 This section contains a catalog of GET, PUT, DELETE, and POST operations that
273 can be performed on these URLs. This set of operations are aimed at programs
274 that use HTTP to communicate with a Tahoe node. The next section describes
275 operations that are intended for web browsers.
277 === Reading A File ===
280 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME
282 This will retrieve the contents of the given file. The HTTP response body
283 will contain the sequence of bytes that make up the file.
285 To view files in a web browser, you may want more control over the
286 Content-Type and Content-Disposition headers. Please see the next section
287 "Browser Operations", for details on how to modify these URLs for that
290 === Writing/Uploading A File ===
293 PUT /uri/$DIRCAP/[SUBDIRS../]FILENAME
295 Upload a file, using the data from the HTTP request body, and add whatever
296 child links and subdirectories are necessary to make the file available at
297 the given location. Once this operation succeeds, a GET on the same URL will
298 retrieve the same contents that were just uploaded. This will create any
299 necessary intermediate subdirectories.
301 To use the /uri/$FILECAP form, $FILECAP be a write-cap for a mutable file.
303 In the /uri/$DIRCAP/[SUBDIRS../]FILENAME form, if the target file is a
304 writable mutable file, that files contents will be overwritten in-place. If
305 it is a read-cap for a mutable file, an error will occur. If it is an
306 immutable file, the old file will be discarded, and a new one will be put in
309 When creating a new file, if "mutable=true" is in the query arguments, the
310 operation will create a mutable file instead of an immutable one.
312 This returns the file-cap of the resulting file. If a new file was created
313 by this method, the HTTP response code (as dictated by rfc2616) will be set
314 to 201 CREATED. If an existing file was replaced or modified, the response
317 Note that the 'curl -T localfile http://127.0.0.1:3456/uri/$DIRCAP/foo.txt'
318 command can be used to invoke this operation.
322 This uploads a file, and produces a file-cap for the contents, but does not
323 attach the file into the virtual drive. No directories will be modified by
324 this operation. The file-cap is returned as the body of the HTTP response.
326 If "mutable=true" is in the query arguments, the operation will create a
327 mutable file, and return its write-cap in the HTTP respose. The default is
328 to create an immutable file, returning the read-cap as a response.
330 === Creating A New Directory ===
335 Create a new empty directory and return its write-cap as the HTTP response
336 body. This does not make the newly created directory visible from the
337 virtual drive. The "PUT" operation is provided for backwards compatibility:
338 new code should use POST.
340 POST /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir
341 PUT /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=mkdir
343 Create new directories as necessary to make sure that the named target
344 ($DIRCAP/SUBDIRS../SUBDIR) is a directory. This will create additional
345 intermediate directories as necessary. If the named target directory already
346 exists, this will make no changes to it.
348 This will return an error if a blocking file is present at any of the parent
349 names, preventing the server from creating the necessary parent directory.
351 The write-cap of the new directory will be returned as the HTTP response
354 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=NAME
356 Create a new empty directory and attach it to the given existing directory.
357 This will create additional intermediate directories as necessary.
359 The URL of this form points to the parent of the bottom-most new directory,
360 whereas the previous form has a URL that points directly to the bottom-most
363 === Get Information About A File Or Directory (as JSON) ===
365 GET /uri/$FILECAP?t=json
366 GET /uri/$DIRCAP?t=json
367 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json
368 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json
370 This returns a machine-parseable JSON-encoded description of the given
371 object. The JSON always contains a list, and the first element of the list
372 is always a flag that indicates whether the referenced object is a file or a
373 directory. If it is a file, then the information includes file size and URI,
376 GET /uri/$FILECAP?t=json :
377 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=json :
379 [ "filenode", { "ro_uri": file_uri,
380 "verify_uri": verify_uri,
383 "metadata": {"ctime": 1202777696.7564139,
384 "mtime": 1202777696.7564139
388 If it is a directory, then it includes information about the children of
389 this directory, as a mapping from child name to a set of data about the
390 child (the same data that would appear in a corresponding GET?t=json of the
391 child itself). The child entries also include metadata about each child,
392 including creation- and modification- timestamps. The output looks like
395 GET /uri/$DIRCAP?t=json :
396 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR?t=json :
398 [ "dirnode", { "rw_uri": read_write_uri,
399 "ro_uri": read_only_uri,
400 "verify_uri": verify_uri,
403 "foo.txt": [ "filenode", { "ro_uri": uri,
406 "ctime": 1202777696.7564139,
407 "mtime": 1202777696.7564139
410 "subdir": [ "dirnode", { "rw_uri": rwuri,
413 "ctime": 1202778102.7589991,
414 "mtime": 1202778111.2160511,
419 In the above example, note how 'children' is a dictionary in which the keys
420 are child names and the values depend upon whether the child is a file or a
421 directory. The value is mostly the same as the JSON representation of the
422 child object (except that directories do not recurse -- the "children"
423 entry of the child is omitted, and the directory view includes the metadata
424 that is stored on the directory edge).
426 Then the rw_uri field will be present in the information about a directory
427 if and only if you have read-write access to that directory. The verify_uri
428 field will be presend if and only if the object has a verify-cap
429 (non-distributed LIT files do not have verify-caps).
432 === Attaching an existing File or Directory by its read- or write- cap ===
434 PUT /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri
436 This attaches a child object (either a file or directory) to a specified
437 location in the virtual filesystem. The child object is referenced by its
438 read- or write- cap, as provided in the HTTP request body. This will create
439 intermediate directories as necessary.
441 This is similar to a UNIX hardlink: by referencing a previously-uploaded
442 file (or previously-created directory) instead of uploading/creating a new
443 one, you can create two references to the same object.
445 The read- or write- cap of the child is provided in the body of the HTTP
446 request, and this same cap is returned in the response body.
448 The default behavior is to overwrite any existing object at the same
449 location. To prevent this (and make the operation return an error instead of
450 overwriting), add a "replace=false" argument, as "?t=uri&replace=false".
451 With replace=false, this operation will return an HTTP 409 "Conflict" error
452 if there is already an object at the given location, rather than overwriting
453 the existing object. Note that "true", "t", and "1" are all synonyms for
454 "True", and "false", "f", and "0" are synonyms for "False". the parameter is
457 === Deleting a File or Directory ===
459 DELETE /uri/$DIRCAP/[SUBDIRS../]CHILDNAME
461 This removes the given name from its parent directory. CHILDNAME is the
462 name to be removed, and $DIRCAP/SUBDIRS.. indicates the directory that will
465 Note that this does not actually delete the file or directory that the name
466 points to from the tahoe grid -- it only removes the named reference from
467 this directory. If there are other names in this directory or in other
468 directories that point to the resource, then it will remain accessible
469 through those paths. Even if all names pointing to this object are removed
470 from their parent directories, then someone with possession of its read-cap
471 can continue to access the object through that cap.
473 The object will only become completely unreachable once 1: there are no
474 reachable directories that reference it, and 2: nobody is holding a read-
475 or write- cap to the object. (This behavior is very similar to the way
476 hardlinks and anonymous files work in traditional unix filesystems).
478 This operation will not modify more than a single directory. Intermediate
479 directories which were implicitly created by PUT or POST methods will *not*
480 be automatically removed by DELETE.
482 This method returns the file- or directory- cap of the object that was just
485 == Browser Operations ==
487 This section describes the HTTP operations that provide support for humans
488 running a web browser. Most of these operations use HTML forms that use POST
489 to drive the Tahoe node.
491 Note that for all POST operations, the arguments listed can be provided
492 either as URL query arguments or as form body fields. URL query arguments are
493 separated from the main URL by "?", and from each other by "&". For example,
494 "POST /uri/$DIRCAP?t=upload&mutable=true". Form body fields are usually
495 specified by using <input type="hidden"> elements. For clarity, the
496 descriptions below display the most significant arguments as URL query args.
498 === Viewing A Directory (as HTML) ===
500 GET /uri/$DIRCAP/[SUBDIRS../]
502 This returns an HTML page, intended to be displayed to a human by a web
503 browser, which contains HREF links to all files and directories reachable
504 from this directory. These HREF links do not have a t= argument, meaning
505 that a human who follows them will get pages also meant for a human. It also
506 contains forms to upload new files, and to delete files and directories.
507 Those forms use POST methods to do their job.
509 === Viewing/Downloading a File ===
512 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME
514 This will retrieve the contents of the given file. The HTTP response body
515 will contain the sequence of bytes that make up the file.
517 If you want the HTTP response to include a useful Content-Type header,
518 either use the second form (which starts with a $DIRCAP), or add a
519 "filename=foo" query argument, like "GET /uri/$FILECAP?filename=foo.jpg".
520 The bare "GET /uri/$FILECAP" does not give the Tahoe node enough information
521 to determine a Content-Type (since Tahoe immutable files are merely
522 sequences of bytes, not typed+named file objects).
524 If the URL has both filename= and "save=true" in the query arguments, then
525 the server to add a "Content-Disposition: attachment" header, along with a
526 filename= parameter. When a user clicks on such a link, most browsers will
527 offer to let the user save the file instead of displaying it inline (indeed,
528 most browsers will refuse to display it inline). "true", "t", "1", and other
529 case-insensitive equivalents are all treated the same.
531 Character-set handling in URLs and HTTP headers is a dubious art[1]. For
532 maximum compatibility, Tahoe simply copies the bytes from the filename=
533 argument into the Content-Disposition header's filename= parameter, without
534 trying to interpret them in any particular way.
537 GET /named/$FILECAP/FILENAME
539 This is an alternate download form which makes it easier to get the correct
540 filename. The Tahoe server will provide the contents of the given file, with
541 a Content-Type header derived from the given filename. This form is used to
542 get browsers to use the "Save Link As" feature correctly, and also helps
543 command-line tools like "wget" and "curl" use the right filename. Note that
544 this form can *only* be used with file caps; it is an error to use a
545 directory cap after the /named/ prefix.
547 === Get Information About A File Or Directory (as HTML) ===
549 GET /uri/$FILECAP?t=info
550 GET /uri/$DIRCAP/?t=info
551 GET /uri/$DIRCAP/[SUBDIRS../]SUBDIR/?t=info
552 GET /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=info
554 This returns a human-oriented HTML page with more detail about the selected
555 file or directory object. This page contains the following items:
560 raw contents (text/plain)
561 access caps (URIs): verify-cap, read-cap, write-cap (for mutable objects)
562 check/verify/repair form
563 deep-check/deep-size/deep-stats/manifest (for directories)
564 replace-conents form (for mutable files)
566 === Creating a Directory ===
570 This creates a new directory, but does not attach it to the virtual
573 If a "redirect_to_result=true" argument is provided, then the HTTP response
574 will cause the web browser to be redirected to a /uri/$DIRCAP page that
575 gives access to the newly-created directory. If you bookmark this page,
576 you'll be able to get back to the directory again in the future. This is the
577 recommended way to start working with a Tahoe server: create a new unlinked
578 directory (using redirect_to_result=true), then bookmark the resulting
579 /uri/$DIRCAP page. There is a "Create Directory" button on the Welcome page
580 to invoke this action.
582 If "redirect_to_result=true" is not provided (or is given a value of
583 "false"), then the HTTP response body will simply be the write-cap of the
586 POST /uri/$DIRCAP/[SUBDIRS../]?t=mkdir&name=CHILDNAME
588 This creates a new directory as a child of the designated SUBDIR. This will
589 create additional intermediate directories as necessary.
591 If a "when_done=URL" argument is provided, the HTTP response will cause the
592 web browser to redirect to the given URL. This provides a convenient way to
593 return the browser to the directory that was just modified. Without a
594 when_done= argument, the HTTP response will simply contain the write-cap of
595 the directory that was just created.
598 === Uploading a File ===
602 This uploads a file, and produces a file-cap for the contents, but does not
603 attach the file into the virtual drive. No directories will be modified by
606 The file must be provided as the "file" field of an HTML encoded form body,
607 produced in response to an HTML form like this:
608 <form action="/uri" method="POST" enctype="multipart/form-data">
609 <input type="hidden" name="t" value="upload" />
610 <input type="file" name="file" />
611 <input type="submit" value="Upload Unlinked" />
614 If a "when_done=URL" argument is provided, the response body will cause the
615 browser to redirect to the given URL. If the when_done= URL has the string
616 "%(uri)s" in it, that string will be replaced by a URL-escaped form of the
617 newly created file-cap. (Note that without this substitution, there is no
618 way to access the file that was just uploaded).
620 The default (in the absence of when_done=) is to return an HTML page that
621 describes the results of the upload. This page will contain information
622 about which storage servers were used for the upload, how long each
625 If a "mutable=true" argument is provided, the operation will create a
626 mutable file, and the response body will contain the write-cap instead of
627 the upload results page. The default is to create an immutable file,
628 returning the upload results page as a response.
631 POST /uri/$DIRCAP/[SUBDIRS../]?t=upload
633 This uploads a file, and attaches it as a new child of the given directory.
634 The file must be provided as the "file" field of an HTML encoded form body,
635 produced in response to an HTML form like this:
636 <form action="." method="POST" enctype="multipart/form-data">
637 <input type="hidden" name="t" value="upload" />
638 <input type="file" name="file" />
639 <input type="submit" value="Upload" />
642 A "name=" argument can be provided to specify the new child's name,
643 otherwise it will be taken from the "filename" field of the upload form
644 (most web browsers will copy the last component of the original file's
645 pathname into this field). To avoid confusion, name= is not allowed to
648 If there is already a child with that name, and it is a mutable file, then
649 its contents are replaced with the data being uploaded. If it is not a
650 mutable file, the default behavior is to remove the existing child before
651 creating a new one. To prevent this (and make the operation return an error
652 instead of overwriting the old child), add a "replace=false" argument, as
653 "?t=upload&replace=false". With replace=false, this operation will return an
654 HTTP 409 "Conflict" error if there is already an object at the given
655 location, rather than overwriting the existing object. Note that "true",
656 "t", and "1" are all synonyms for "True", and "false", "f", and "0" are
657 synonyms for "False". the parameter is case-insensitive.
659 This will create additional intermediate directories as necessary, although
660 since it is expected to be triggered by a form that was retrieved by "GET
661 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
664 If a "mutable=true" argument is provided, any new file that is created will
665 be a mutable file instead of an immutable one. <input type="checkbox"
666 name="mutable" /> will give the user a way to set this option.
668 If a "when_done=URL" argument is provided, the HTTP response will cause the
669 web browser to redirect to the given URL. This provides a convenient way to
670 return the browser to the directory that was just modified. Without a
671 when_done= argument, the HTTP response will simply contain the file-cap of
672 the file that was just uploaded (a write-cap for mutable files, or a
673 read-cap for immutable files).
675 POST /uri/$DIRCAP/[SUBDIRS../]FILENAME?t=upload
677 This also uploads a file and attaches it as a new child of the given
678 directory. It is a slight variant of the previous operation, as the URL
679 refers to the target file rather than the parent directory. It is otherwise
680 identical: this accepts mutable= and when_done= arguments too.
682 POST /uri/$FILECAP?t=upload
684 This modifies the contents of an existing mutable file in-place. An error is
685 signalled if $FILECAP does not refer to a mutable file. It behaves just like
686 the "PUT /uri/$FILECAP" form, but uses a POST for the benefit of HTML forms
689 === Attaching An Existing File Or Directory (by URI) ===
691 POST /uri/$DIRCAP/[SUBDIRS../]?t=uri&name=CHILDNAME&uri=CHILDCAP
693 This attaches a given read- or write- cap "CHILDCAP" to the designated
694 directory, with a specified child name. This behaves much like the PUT t=uri
695 operation, and is a lot like a UNIX hardlink.
697 This will create additional intermediate directories as necessary, although
698 since it is expected to be triggered by a form that was retrieved by "GET
699 /uri/$DIRCAP/[SUBDIRS../]", it is likely that the parent directory will
702 === Deleting A Child ===
704 POST /uri/$DIRCAP/[SUBDIRS../]?t=delete&name=CHILDNAME
706 This instructs the node to delete a child object (file or subdirectory) from
707 the given directory. Note that the entire subtree is removed. This is
708 somewhat like "rm -rf" (from the point of view of the parent), but other
709 references into the subtree will see that the child subdirectories are not
710 modified by this operation. Only the link from the given directory to its
713 === Renaming A Child ===
715 POST /uri/$DIRCAP/[SUBDIRS../]?t=rename&from_name=OLD&to_name=NEW
717 This instructs the node to rename a child of the given directory. This is
718 exactly the same as removing the child, then adding the same child-cap under
719 the new name. This operation cannot move the child to a different directory.
721 This operation will replace any existing child of the new name, making it
722 behave like the UNIX "mv -f" command.
724 === Other Utilities ===
728 This causes a redirect to /uri/$CAP, and retains any additional query
729 arguments (like filename= or save=). This is for the convenience of web
730 forms which allow the user to paste in a read- or write- cap (obtained
731 through some out-of-band channel, like IM or email).
733 Note that this form merely redirects to the specific file or directory
734 indicated by the $CAP: unlike the GET /uri/$DIRCAP form, you cannot
735 traverse to children by appending additional path segments to the URL.
737 GET /uri/$DIRCAP/[SUBDIRS../]?t=rename-form&name=$CHILDNAME
739 This provides a useful facility to browser-based user interfaces. It
740 returns a page containing a form targetting the "POST $DIRCAP t=rename"
741 functionality described above, with the provided $CHILDNAME present in the
742 'from_name' field of that form. I.e. this presents a form offering to
743 rename $CHILDNAME, requesting the new name, and submitting POST rename.
745 GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=uri
747 This returns the file- or directory- cap for the specified object.
749 GET /uri/$DIRCAP/[SUBDIRS../]CHILDNAME?t=readonly-uri
751 This returns a read-only file- or directory- cap for the specified object.
752 If the object is an immutable file, this will return the same value as
755 === Debugging and Testing Features ===
757 These URLs are less-likely to be helpful to the casual Tahoe user, and are
758 mainly intended for developers.
762 This triggers the FileChecker to determine the current "health" of the
763 given file or directory, by counting how many shares are available. The
764 page that is returned will display the results. This can be used as a "show
765 me detailed information about this file" page.
767 If a verify=true argument is provided, the node will perform a more
768 intensive check, downloading and verifying every single bit of every share.
770 If an output=JSON argument is provided, the response will be
771 machine-readable JSON instead of human-oriented HTML. The data is a
772 dictionary with the following keys:
774 storage-index: a base32-encoded string with the objects's storage index,
775 or an empty string for LIT files
776 summary: a string, with a one-line summary of the stats of the file
777 results: a dictionary that describes the state of the file. For LIT files,
778 this dictionary has only the 'healthy' key, which will always be
779 True. For distributed files, this dictionary has the following
781 count-shares-good: the number of good shares that were found
782 count-shares-needed: 'k', the number of shares required for recovery
783 count-shares-expected: 'N', the number of total shares generated
784 count-good-share-hosts: the number of distinct storage servers with
785 good shares. If this number is less than
786 count-shares-good, then some shares are doubled
787 up, increasing the correlation of failures. This
788 indicates that one or more shares should be
789 moved to an otherwise unused server, if one is
791 count-wrong-shares: for mutable files, the number of shares for
792 versions other than the 'best' one (highest
793 sequence number, highest roothash). These are
795 count-recoverable-versions: for mutable files, the number of
796 recoverable versions of the file. For
797 a healthy file, this will equal 1.
798 count-unrecoverable-versions: for mutable files, the number of
799 unrecoverable versions of the file.
800 For a healthy file, this will be 0.
801 count-corrupt-shares: the number of shares with integrity failures
802 list-corrupt-shares: a list of "share locators", one for each share
803 that was found to be corrupt. Each share locator
804 is a list of (serverid, storage_index, sharenum).
805 needs-rebalancing: (bool) True if there are multiple shares on a single
806 storage server, indicating a reduction in reliability
807 that could be resolved by moving shares to new
809 servers-responding: list of base32-encoded storage server identifiers,
810 one for each server which responded to the share
812 healthy: (bool) True if the file is completely healthy, False otherwise.
813 Healthy files have at least N good shares. Overlapping shares
814 (indicated by count-good-share-hosts < count-shares-good) do not
815 currently cause a file to be marked unhealthy. If there are at
816 least N good shares, then corrupt shares do not cause the file to
817 be marked unhealthy, although the corrupt shares will be listed
818 in the results (list-corrupt-shares) and should be manually
819 removed to wasting time in subsequent downloads (as the
820 downloader rediscovers the corruption and uses alternate shares).
821 sharemap: dict mapping share identifier to list of serverids
822 (base32-encoded strings). This indicates which servers are
823 holding which shares. For immutable files, the shareid is
824 an integer (the share number, from 0 to N-1). For
825 immutable files, it is a string of the form
826 'seq%d-%s-sh%d', containing the sequence number, the
827 roothash, and the share number.
829 POST $URL?t=start-deep-check (must add &ophandle=XYZ)
831 This initiates a recursive walk of all files and directories reachable from
832 the target, performing a check on each one just like t=check. The result
833 page will contain a summary of the results, including details on any
834 file/directory that was not fully healthy.
836 t=start-deep-check can only be invoked on a directory. An error (400
837 BAD_REQUEST) will be signalled if it is invoked on a file. The recursive
838 walker will deal with loops safely.
840 This accepts the same verify= argument as t=check.
842 Since this operation can take a long time (perhaps a second per object),
843 the ophandle= argument is required (see "Slow Operations, Progress, and
844 Cancelling" above). The response to this POST will be a redirect to the
845 corresponding /operations/$HANDLE page (with output=HTML or output=JSON to
846 match the output= argument given to the POST). The deep-check operation
847 will continue to run in the background, and the /operations page should be
848 used to find out when the operation is done.
850 Detailed check results for non-healthy files and directories will be
851 available under /operations/$HANDLE/$STORAGEINDEX, and the HTML status will
852 contain links to these detailed results.
854 The HTML /operations/$HANDLE page for incomplete operations will contain a
855 meta-refresh tag, set to 60 seconds, so that a browser which uses
856 deep-check will automatically poll until the operation has completed.
858 The JSON page (/options/$HANDLE?output=JSON) will contain a
859 machine-readable JSON dictionary with the following keys:
861 finished: a boolean, True if the operation is complete, else False. Some
862 of the remaining keys may not be present until the operation
864 root-storage-index: a base32-encoded string with the storage index of the
865 starting point of the deep-check operation
866 count-objects-checked: count of how many objects were checked. Note that
867 non-distributed objects (i.e. small immutable LIT
868 files) are not checked, since for these objects,
869 the data is contained entirely in the URI.
870 count-objects-healthy: how many of those objects were completely healthy
871 count-objects-unhealthy: how many were damaged in some way
872 count-corrupt-shares: how many shares were found to have corruption,
873 summed over all objects examined
874 list-corrupt-shares: a list of "share identifiers", one for each share
875 that was found to be corrupt. Each share identifier
876 is a list of (serverid, storage_index, sharenum).
877 list-unhealthy-files: a list of (pathname, check-results) tuples, for
878 each file that was not fully healthy. 'pathname' is
879 a list of strings (which can be joined by "/"
880 characters to turn it into a single string),
881 relative to the directory on which deep-check was
882 invoked. The 'check-results' field is the same as
883 that returned by t=check&output=JSON, described
885 stats: a dictionary with the same keys as the t=start-deep-stats command
888 POST $URL?t=stream-deep-check
890 This initiates a recursive walk of all files and directories reachable from
891 the target, performing a check on each one just like t=check. For each
892 unique object (duplicates are skipped), a single line of JSON is emitted to
893 the HTTP response channel. When the walk is complete, a final line of JSON
894 is emitted which contains the accumulated file-size/count "deep-stats" data.
896 This command takes the same arguments as t=start-deep-check.
898 A CLI tool can split the response stream on newlines into "response units",
899 and parse each response unit as JSON. Each such parsed unit will be a
900 dictionary, and will contain at least the "type" key: a string, one of
901 "file", "directory", or "stats".
903 For all units that have a type of "file" or "directory", the dictionary will
904 contain the following keys:
906 "path": a list of strings, with the path that is traversed to reach the
908 "cap": a writecap for the file or directory, if available, else a readcap
909 "verifycap": a verifycap for the file or directory
910 "repaircap": the weakest cap which can still be used to repair the object
911 "storage-index": a base32 storage index for the object
912 "check-results": a copy of the dictionary which would be returned by
913 t=check&output=json, with three top-level keys:
914 "storage-index", "summary", and "results", and a variety
915 of counts and sharemaps in the "results" value.
917 Note that non-distributed files (i.e. LIT files) will have values of None
918 for verifycap, repaircap, and storage-index, since these files can neither
919 be verified nor repaired, and are not stored on the storage servers.
920 Likewise the check-results dictionary will be limited: an empty string for
921 storage-index, and a results dictionary with only the "healthy" key.
923 The last unit in the stream will have a type of "stats", and will contain
924 the keys described in the "start-deep-stats" operation, below.
927 POST $URL?t=check&repair=true
929 This performs a health check of the given file or directory, and if the
930 checker determines that the object is not healthy (some shares are missing
931 or corrupted), it will perform a "repair". During repair, any missing
932 shares will be regenerated and uploaded to new servers.
934 This accepts the same verify=true argument as t=check. When an output=JSON
935 argument is provided, the machine-readable JSON response will contain the
938 storage-index: a base32-encoded string with the objects's storage index,
939 or an empty string for LIT files
940 repair-attempted: (bool) True if repair was attempted
941 repair-successful: (bool) True if repair was attempted and the file was
942 fully healthy afterwards. False if no repair was
943 attempted, or if a repair attempt failed.
944 pre-repair-results: a dictionary that describes the state of the file
945 before any repair was performed. This contains exactly
946 the same keys as the 'results' value of the t=check
947 response, described above.
948 post-repair-results: a dictionary that describes the state of the file
949 after any repair was performed. If no repair was
950 performed, post-repair-results and pre-repair-results
951 will be the same. This contains exactly the same keys
952 as the 'results' value of the t=check response,
955 POST $URL?t=start-deep-check&repair=true (must add &ophandle=XYZ)
957 This triggers a recursive walk of all files and directories, performing a
958 t=check&repair=true on each one.
960 Like t=start-deep-check without the repair= argument, this can only be
961 invoked on a directory. An error (400 BAD_REQUEST) will be signalled if it
962 is invoked on a file. The recursive walker will deal with loops safely.
964 This accepts the same verify=true argument as t=start-deep-check. It uses
965 the same ophandle= mechanism as start-deep-check. When an output=JSON
966 argument is provided, the response will contain the following keys:
968 finished: (bool) True if the operation has completed, else False
969 root-storage-index: a base32-encoded string with the storage index of the
970 starting point of the deep-check operation
971 count-objects-checked: count of how many objects were checked
973 count-objects-healthy-pre-repair: how many of those objects were completely
974 healthy, before any repair
975 count-objects-unhealthy-pre-repair: how many were damaged in some way
976 count-objects-healthy-post-repair: how many of those objects were completely
977 healthy, after any repair
978 count-objects-unhealthy-post-repair: how many were damaged in some way
980 count-repairs-attempted: repairs were attempted on this many objects.
981 count-repairs-successful: how many repairs resulted in healthy objects
982 count-repairs-unsuccessful: how many repairs resulted did not results in
983 completely healthy objects
984 count-corrupt-shares-pre-repair: how many shares were found to have
985 corruption, summed over all objects
986 examined, before any repair
987 count-corrupt-shares-post-repair: how many shares were found to have
988 corruption, summed over all objects
989 examined, after any repair
990 list-corrupt-shares: a list of "share identifiers", one for each share
991 that was found to be corrupt (before any repair).
992 Each share identifier is a list of (serverid,
993 storage_index, sharenum).
994 list-remaining-corrupt-shares: like list-corrupt-shares, but mutable shares
995 that were successfully repaired are not
996 included. These are shares that need
997 manual processing. Since immutable shares
998 cannot be modified by clients, all corruption
999 in immutable shares will be listed here.
1000 list-unhealthy-files: a list of (pathname, check-results) tuples, for
1001 each file that was not fully healthy. 'pathname' is
1002 relative to the directory on which deep-check was
1003 invoked. The 'check-results' field is the same as
1004 that returned by t=check&repair=true&output=JSON,
1006 stats: a dictionary with the same keys as the t=start-deep-stats command
1009 POST $DIRURL?t=start-manifest (must add &ophandle=XYZ)
1011 This operation generates a "manfest" of the given directory tree, mostly
1012 for debugging. This is a table of (path, filecap/dircap), for every object
1013 reachable from the starting directory. The path will be slash-joined, and
1014 the filecap/dircap will contain a link to the object in question. This page
1015 gives immediate access to every object in the virtual filesystem subtree.
1017 This operation uses the same ophandle= mechanism as deep-check. The
1018 corresponding /operations/$HANDLE page has three different forms. The
1019 default is output=HTML.
1021 If output=text is added to the query args, the results will be a text/plain
1022 list. The first line is special: it is either "finished: yes" or "finished:
1023 no"; if the operation is not finished, you must periodically reload the
1024 page until it completes. The rest of the results are a plaintext list, with
1025 one file/dir per line, slash-separated, with the filecap/dircap separated
1028 If output=JSON is added to the queryargs, then the results will be a
1029 JSON-formatted dictionary with six keys. Note that because large directory
1030 structures can result in very large JSON results, the full results will not
1031 be available until the operation is complete (i.e. until output["finished"]
1034 finished (bool): if False then you must reload the page until True
1035 origin_si (base32 str): the storage index of the starting point
1036 manifest: list of (path, cap) tuples, where path is a list of strings.
1037 verifycaps: list of (printable) verify cap strings
1038 storage-index: list of (base32) storage index strings
1039 stats: a dictionary with the same keys as the t=start-deep-stats command
1042 POST $DIRURL?t=start-deep-size (must add &ophandle=XYZ)
1044 This operation generates a number (in bytes) containing the sum of the
1045 filesize of all directories and immutable files reachable from the given
1046 directory. This is a rough lower bound of the total space consumed by this
1047 subtree. It does not include space consumed by mutable files, nor does it
1048 take expansion or encoding overhead into account. Later versions of the
1049 code may improve this estimate upwards.
1051 The /operations/$HANDLE status output consists of two lines of text:
1056 POST $DIRURL?t=start-deep-stats (must add &ophandle=XYZ)
1058 This operation performs a recursive walk of all files and directories
1059 reachable from the given directory, and generates a collection of
1060 statistics about those objects.
1062 The result (obtained from the /operations/$OPHANDLE page) is a
1063 JSON-serialized dictionary with the following keys (note that some of these
1064 keys may be missing until 'finished' is True):
1066 finished: (bool) True if the operation has finished, else False
1067 count-immutable-files: count of how many CHK files are in the set
1068 count-mutable-files: same, for mutable files (does not include directories)
1069 count-literal-files: same, for LIT files (data contained inside the URI)
1070 count-files: sum of the above three
1071 count-directories: count of directories
1072 size-immutable-files: total bytes for all CHK files in the set, =deep-size
1073 size-mutable-files (TODO): same, for current version of all mutable files
1074 size-literal-files: same, for LIT files
1075 size-directories: size of directories (includes size-literal-files)
1076 size-files-histogram: list of (minsize, maxsize, count) buckets,
1077 with a histogram of filesizes, 5dB/bucket,
1078 for both literal and immutable files
1079 largest-directory: number of children in the largest directory
1080 largest-immutable-file: number of bytes in the largest CHK file
1082 size-mutable-files is not implemented, because it would require extra
1083 queries to each mutable file to get their size. This may be implemented in
1086 Assuming no sharing, the basic space consumed by a single root directory is
1087 the sum of size-immutable-files, size-mutable-files, and size-directories.
1088 The actual disk space used by the shares is larger, because of the
1089 following sources of overhead:
1092 expansion due to erasure coding
1093 share management data (leases)
1094 backend (ext3) minimum block size
1096 POST $URL?t=stream-manifest
1098 This operation performs a recursive walk of all files and directories
1099 reachable from the given starting point. For each such unique object
1100 (duplicates are skipped), a single line of JSON is emitted to the HTTP
1101 response channel. When the walk is complete, a final line of JSON is emitted
1102 which contains the accumulated file-size/count "deep-stats" data.
1104 A CLI tool can split the response stream on newlines into "response units",
1105 and parse each response unit as JSON. Each such parsed unit will be a
1106 dictionary, and will contain at least the "type" key: a string, one of
1107 "file", "directory", or "stats".
1109 For all units that have a type of "file" or "directory", the dictionary will
1110 contain the following keys:
1112 "path": a list of strings, with the path that is traversed to reach the
1114 "cap": a writecap for the file or directory, if available, else a readcap
1115 "verifycap": a verifycap for the file or directory
1116 "repaircap": the weakest cap which can still be used to repair the object
1117 "storage-index": a base32 storage index for the object
1119 Note that non-distributed files (i.e. LIT files) will have values of None
1120 for verifycap, repaircap, and storage-index, since these files can neither
1121 be verified nor repaired, and are not stored on the storage servers.
1123 The last unit in the stream will have a type of "stats", and will contain
1124 the keys described in the "start-deep-stats" operation, below.
1127 == Other Useful Pages ==
1129 The portion of the web namespace that begins with "/uri" (and "/named") is
1130 dedicated to giving users (both humans and programs) access to the Tahoe
1131 virtual filesystem. The rest of the namespace provides status information
1132 about the state of the Tahoe node.
1134 GET / (the root page)
1136 This is the "Welcome Page", and contains a few distinct sections:
1138 Node information: library versions, local nodeid, services being provided.
1140 Filesystem Access Forms: create a new directory, view a file/directory by
1141 URI, upload a file (unlinked), download a file by
1144 Grid Status: introducer information, helper information, connected storage
1149 This page lists all active uploads and downloads, and contains a short list
1150 of recent upload/download operations. Each operation has a link to a page
1151 that describes file sizes, servers that were involved, and the time consumed
1152 in each phase of the operation.
1154 A GET of /status/?t=json will contain a machine-readable subset of the same
1155 data. It returns a JSON-encoded dictionary. The only key defined at this
1156 time is "active", with a value that is a list of operation dictionaries, one
1157 for each active operation. Once an operation is completed, it will no longer
1158 appear in data["active"] .
1160 Each op-dict contains a "type" key, one of "upload", "download",
1161 "mapupdate", "publish", or "retrieve" (the first two are for immutable
1162 files, while the latter three are for mutable files and directories).
1164 The "upload" op-dict will contain the following keys:
1166 type (string): "upload"
1167 storage-index-string (string): a base32-encoded storage index
1168 total-size (int): total size of the file
1169 status (string): current status of the operation
1170 progress-hash (float): 1.0 when the file has been hashed
1171 progress-ciphertext (float): 1.0 when the file has been encrypted.
1172 progress-encode-push (float): 1.0 when the file has been encoded and
1173 pushed to the storage servers. For helper
1174 uploads, the ciphertext value climbs to 1.0
1175 first, then encoding starts. For unassisted
1176 uploads, ciphertext and encode-push progress
1177 will climb at the same pace.
1179 The "download" op-dict will contain the following keys:
1181 type (string): "download"
1182 storage-index-string (string): a base32-encoded storage index
1183 total-size (int): total size of the file
1184 status (string): current status of the operation
1185 progress (float): 1.0 when the file has been fully downloaded
1187 Front-ends which want to report progress information are advised to simply
1188 average together all the progress-* indicators. A slightly more accurate
1189 value can be found by ignoring the progress-hash value (since the current
1190 implementation hashes synchronously, so clients will probably never see
1191 progress-hash!=1.0).
1195 This page provides a basic tool to predict the likely storage and bandwidth
1196 requirements of a large Tahoe grid. It provides forms to input things like
1197 total number of users, number of files per user, average file size, number
1198 of servers, expansion ratio, hard drive failure rate, etc. It then provides
1199 numbers like how many disks per server will be needed, how many read
1200 operations per second should be expected, and the likely MTBF for files in
1201 the grid. This information is very preliminary, and the model upon which it
1202 is based still needs a lot of work.
1206 If the node is running a helper (i.e. if [helper]enabled is set to True in
1207 tahoe.cfg), then this page will provide a list of all the helper operations
1208 currently in progress. If "?t=json" is added to the URL, it will return a
1209 JSON-formatted list of helper statistics, which can then be used to produce
1210 graphs to indicate how busy the helper is.
1214 This page provides "node statistics", which are collected from a variety of
1217 load_monitor: every second, the node schedules a timer for one second in
1218 the future, then measures how late the subsequent callback
1219 is. The "load_average" is this tardiness, measured in
1220 seconds, averaged over the last minute. It is an indication
1221 of a busy node, one which is doing more work than can be
1222 completed in a timely fashion. The "max_load" value is the
1223 highest value that has been seen in the last 60 seconds.
1225 cpu_monitor: every minute, the node uses time.clock() to measure how much
1226 CPU time it has used, and it uses this value to produce
1227 1min/5min/15min moving averages. These values range from 0%
1228 (0.0) to 100% (1.0), and indicate what fraction of the CPU
1229 has been used by the Tahoe node. Not all operating systems
1230 provide meaningful data to time.clock(): they may report 100%
1231 CPU usage at all times.
1233 uploader: this counts how many immutable files (and bytes) have been
1234 uploaded since the node was started
1236 downloader: this counts how many immutable files have been downloaded
1237 since the node was started
1239 publishes: this counts how many mutable files (including directories) have
1240 been modified since the node was started
1242 retrieves: this counts how many mutable files (including directories) have
1243 been read since the node was started
1245 There are other statistics that are tracked by the node. The "raw stats"
1246 section shows a formatted dump of all of them.
1248 By adding "?t=json" to the URL, the node will return a JSON-formatted
1249 dictionary of stats values, which can be used by other tools to produce
1250 graphs of node behavior. The misc/munin/ directory in the source
1251 distribution provides some tools to produce these graphs.
1253 GET / (introducer status)
1255 For Introducer nodes, the welcome page displays information about both
1256 clients and servers which are connected to the introducer. Servers make
1257 "service announcements", and these are listed in a table. Clients will
1258 subscribe to hear about service announcements, and these subscriptions are
1259 listed in a separate table. Both tables contain information about what
1260 version of Tahoe is being run by the remote node, their advertised and
1261 outbound IP addresses, their nodeid and nickname, and how long they have
1264 By adding "?t=json" to the URL, the node will return a JSON-formatted
1265 dictionary of stats values, which can be used to produce graphs of connected
1266 clients over time. This dictionary has the following keys:
1268 ["subscription_summary"] : a dictionary mapping service name (like
1269 "storage") to an integer with the number of
1270 clients that have subscribed to hear about that
1272 ["announcement_summary"] : a dictionary mapping service name to an integer
1273 with the number of servers which are announcing
1275 ["announcement_distinct_hosts"] : a dictionary mapping service name to an
1276 integer which represents the number of
1277 distinct hosts that are providing that
1278 service. If two servers have announced
1279 FURLs which use the same hostnames (but
1280 different ports and tubids), they are
1281 considered to be on the same host.
1284 == Static Files in /public_html ==
1286 The wapi server will take any request for a URL that starts with /static
1287 and serve it from a configurable directory which defaults to
1288 $BASEDIR/public_html . This is configured by setting the "[node]web.static"
1289 value in $BASEDIR/tahoe.cfg . If this is left at the default value of
1290 "public_html", then http://localhost:3456/static/subdir/foo.html will be
1291 served with the contents of the file $BASEDIR/public_html/subdir/foo.html .
1293 This can be useful to serve a javascript application which provides a
1294 prettier front-end to the rest of the Tahoe wapi.
1297 == safety and security issues -- names vs. URIs ==
1299 Summary: use explicit file- and dir- caps whenever possible, to reduce the
1300 potential for surprises when the virtual drive is changed while you aren't
1303 The vdrive provides a mutable filesystem, but the ways that the filesystem
1304 can change are limited. The only thing that can change is that the mapping
1305 from child names to child objects that each directory contains can be changed
1306 by adding a new child name pointing to an object, removing an existing child
1307 name, or changing an existing child name to point to a different object.
1309 Obviously if you query tahoe for information about the filesystem and then
1310 act upon the filesystem (such as by getting a listing of the contents of a
1311 directory and then adding a file to the directory), then the filesystem might
1312 have been changed after you queried it and before you acted upon it.
1313 However, if you use the URI instead of the pathname of an object when you act
1314 upon the object, then the only change that can happen is when the object is a
1315 directory then the set of child names it has might be different. If, on the
1316 other hand, you act upon the object using its pathname, then a different
1317 object might be in that place, which can result in more kinds of surprises.
1319 For example, suppose you are writing code which recursively downloads the
1320 contents of a directory. The first thing your code does is fetch the listing
1321 of the contents of the directory. For each child that it fetched, if that
1322 child is a file then it downloads the file, and if that child is a directory
1323 then it recurses into that directory. Now, if the download and the recurse
1324 actions are performed using the child's name, then the results might be
1325 wrong, because for example a child name that pointed to a sub-directory when
1326 you listed the directory might have been changed to point to a file (in which
1327 case your attempt to recurse into it would result in an error and the file
1328 would be skipped), or a child name that pointed to a file when you listed the
1329 directory might now point to a sub-directory (in which case your attempt to
1330 download the child would result in a file containing HTML text describing the
1333 If your recursive algorithm uses the uri of the child instead of the name of
1334 the child, then those kinds of mistakes just can't happen. Note that both the
1335 child's name and the child's URI are included in the results of listing the
1336 parent directory, so it isn't any harder to use the URI for this purpose.
1338 In general, use names if you want "whatever object (whether file or
1339 directory) is found by following this name (or sequence of names) when my
1340 request reaches the server". Use URIs if you want "this particular object".
1342 == Concurrency Issues ==
1344 Tahoe uses both mutable and immutable files. Mutable files can be created
1345 explicitly by doing an upload with ?mutable=true added, or implicitly by
1346 creating a new directory (since a directory is just a special way to
1347 interpret a given mutable file).
1349 Mutable files suffer from the same consistency-vs-availability tradeoff that
1350 all distributed data storage systems face. It is not possible to
1351 simultaneously achieve perfect consistency and perfect availability in the
1352 face of network partitions (servers being unreachable or faulty).
1354 Tahoe tries to achieve a reasonable compromise, but there is a basic rule in
1355 place, known as the Prime Coordination Directive: "Don't Do That". What this
1356 means is that if write-access to a mutable file is available to several
1357 parties, then those parties are responsible for coordinating their activities
1358 to avoid multiple simultaneous updates. This could be achieved by having
1359 these parties talk to each other and using some sort of locking mechanism, or
1360 by serializing all changes through a single writer.
1362 The consequences of performing uncoordinated writes can vary. Some of the
1363 writers may lose their changes, as somebody else wins the race condition. In
1364 many cases the file will be left in an "unhealthy" state, meaning that there
1365 are not as many redundant shares as we would like (reducing the reliability
1366 of the file against server failures). In the worst case, the file can be left
1367 in such an unhealthy state that no version is recoverable, even the old ones.
1368 It is this small possibility of data loss that prompts us to issue the Prime
1369 Coordination Directive.
1371 Tahoe nodes implement internal serialization to make sure that a single Tahoe
1372 node cannot conflict with itself. For example, it is safe to issue two
1373 directory modification requests to a single tahoe node's wapi server at the
1374 same time, because the Tahoe node will internally delay one of them until
1375 after the other has finished being applied. (This feature was introduced in
1376 Tahoe-1.1; back with Tahoe-1.0 the web client was responsible for serializing
1377 web requests themselves).
1379 For more details, please see the "Consistency vs Availability" and "The Prime
1380 Coordination Directive" sections of mutable.txt, in the same directory as
1384 [1]: URLs and HTTP and UTF-8, Oh My
1386 HTTP does not provide a mechanism to specify the character set used to
1387 encode non-ascii names in URLs (rfc2396#2.1). We prefer the convention that
1388 the filename= argument shall be a URL-encoded UTF-8 encoded unicode object.
1389 For example, suppose we want to provoke the server into using a filename of
1390 "f i a n c e-acute e" (i.e. F I A N C U+00E9 E). The UTF-8 encoding of this
1391 is 0x66 0x69 0x61 0x6e 0x63 0xc3 0xa9 0x65 (or "fianc\xC3\xA9e", as python's
1392 repr() function would show). To encode this into a URL, the non-printable
1393 characters must be escaped with the urlencode '%XX' mechansim, giving us
1394 "fianc%C3%A9e". Thus, the first line of the HTTP request will be "GET
1395 /uri/CAP...?save=true&filename=fianc%C3%A9e HTTP/1.1". Not all browsers
1396 provide this: IE7 uses the Latin-1 encoding, which is fianc%E9e.
1398 The response header will need to indicate a non-ASCII filename. The actual
1399 mechanism to do this is not clear. For ASCII filenames, the response header
1402 Content-Disposition: attachment; filename="english.txt"
1404 If Tahoe were to enforce the utf-8 convention, it would need to decode the
1405 URL argument into a unicode string, and then encode it back into a sequence
1406 of bytes when creating the response header. One possibility would be to use
1407 unencoded utf-8. Developers suggest that IE7 might accept this:
1409 #1: Content-Disposition: attachment; filename="fianc\xC3\xA9e"
1410 (note, the last four bytes of that line, not including the newline, are
1411 0xC3 0xA9 0x65 0x22)
1413 RFC2231#4 (dated 1997): suggests that the following might work, and some
1414 developers (http://markmail.org/message/dsjyokgl7hv64ig3) have reported that
1415 it is supported by firefox (but not IE7):
1417 #2: Content-Disposition: attachment; filename*=utf-8''fianc%C3%A9e
1419 My reading of RFC2616#19.5.1 (which defines Content-Disposition) says that
1420 the filename= parameter is defined to be wrapped in quotes (presumeably to
1421 allow spaces without breaking the parsing of subsequent parameters), which
1424 #3: Content-Disposition: attachment; filename*=utf-8''"fianc%C3%A9e"
1426 However this is contrary to the examples in the email thread listed above.
1428 Developers report that IE7 (when it is configured for UTF-8 URL encoding,
1429 which is not the default in asian countries), will accept:
1431 #4: Content-Disposition: attachment; filename=fianc%C3%A9e
1433 However, for maximum compatibility, Tahoe simply copies bytes from the URL
1434 into the response header, rather than enforcing the utf-8 convention. This
1435 means it does not try to decode the filename from the URL argument, nor does
1436 it encode the filename into the response header.
projects / tahoe-lafs / tahoe-lafs.git / blob