[tahoe-lafs/tahoe-lafs.git] / README

Welcome to the Allmydata-Tahoe project.  This project implements a secure,
distributed, fault-tolerant storage grid.  All of the source code is available 
under a Free Software licence.

The basic idea is that the data in this storage grid is spread over all
participating nodes, using an algorithm that can recover the data even if
some of the nodes are not available.

The interface to the storage grid allows you to store and fetch files, either
by self-authenticating cryptographic identifier or by filename and path.

See the web site for all kinds of information, news, and community
discussion:

http://allmydata.org


GETTING PRECOMPILED BINARIES FOR DEBIAN-LIKE SYSTEMS:

Pre-compiled binaries are available for Debian or Ubuntu.  Please see the
following web page for instructions:

http://allmydata.org/trac/tahoe/wiki/DownloadDebianPackages


BUILDING ON WINDOWS:

If you are building on Windows, then the easy way is to install cygwin and
use cygwin version of Python and the cygwin versions of all dependencies
(which will happen naturally if you follow this README -- note that you
cannot use Windows-native versions of any of the dependencies -- they all
have to be cygwin versions).

The second-easiest way is to install cygwin and use cygwin development tools
such as bash, GNU make, gcc, etc., but install the Windows-native version of
Python and the Windows-native versions of all of the dependencies.  If you
create a distutils config file (as per
http://docs.python.org/inst/config-syntax.html ) and put "compiler=mingw32"
in it, then you can follow the rest of this README and the dependencies will
all be automatically built (by the cygwin gcc compiler) as Windows-native
libraries.

The third-easiest way is to use a Microsoft compiler or some other compiler.
This README does not explain how to do that.  You are on your own for now,
but please feel free to contribute a document which explains how to build all
these dependencies using your favorite compiler.


DEPENDENCIES:

If you aren't getting a pre-compiled binary, then you'll have to ensure that
the following packages are installed before you install Tahoe.

There are two kinds of dependencies, "manual dependencies" and
"easy_install-able dependencies".  The latter kind are normally automatically
satisfied for you when you install Tahoe, but if something goes wrong, please
see the EASY_INSTALLABLE DEPENDENCIES section below.

All of the manual dependencies can probably be installed through your
standard package management tool if you are running on a modern Unix
operating system.  For example, on an debian-like system, you can do "sudo
apt-get install gcc make python-dev python-twisted python-pyopenssl".

The Manual Dependencies:

 + a C compiler (language)

 + GNU make (build tool)

 + Python 2.4 or newer (tested against 2.4, and 2.5.1 ), including
   development headers (language)

   http://python.org/

 + Twisted Python (tested against 2.2.0, 2.4.0, and 2.5.0) (network and
   operating system integration library)

   http://twistedmatrix.com/

   You need the following subpackages, which are included in the default
   Twisted distribution:

   * core (the standard Twisted package)
   * web, trial, conch

   Twisted requires zope.interface, a copy of which is included in the
   Twisted distribution. Note that Twisted does *not* require the entire Zope
   distribution, merely the much smaller zope.interface component.

 + Python PyOpenSSL (0.6 or later) (secure transport layer)

   http://pyopenssl.sourceforge.net

   To install PyOpenSSL on Windows-native, download this:
   http://allmydata.org/source/pyOpenSSL-0.6.win32-py2.5.exe

   or for Python 2.4, this:

   http://allmydata.org/source/pyOpenSSL-0.6.win32-py2.4.exe

   To install PyOpenSSL on Windows-cygwin, install the OpenSSL development
   libraries with the cygwin package management tool, then get the pyOpenSSL
   source code, cd into it, and run "python ./setup.py install".

 + OpenSSL, including development headers (cryptography library); not
   required on native Windows (required on cygwin)

   http://openssl.org
  
   The Windows-native pyOpenSSL package comes with OpenSSL, which is why you
   don't need to install OpenSSL separately on Windows-native.

 + the pywin32 package (210 or later); required only on native Windows (not
   required on cygwin)

   http://sourceforge.net/projects/pywin32/


GETTING THE SOURCE CODE:

You need the source code if you are going to install The Debian Way, The
Setuptools Way, or The Running-In-Place Way (see below).  You do not need the
source code if you are getting precompiled binaries for Debian or Ubuntu (see
above), or if you are going to install The easy_install Way (see below).

The code is available via darcs by running the following command:

darcs get http://allmydata.org/source/tahoe/trunk tahoe

This will create a directory named "tahoe" in the current working directory
and put a copy of the latest source code into it.  Later, if you want to get
any new changes, then cd into that directory and run the command "darcs
pull".

Tarballs of sources are available at:

http://allmydata.org/source/tahoe/


INSTALLING:

There are four ways to do it: The easy_install Way, The Setuptools Way, The
Running-In-Place Way, and The Debian Way.  Choose one:

 The easy_install Way:

  You don't need to download the source code first.

  Tahoe is registered with the Python Package Index (PyPI), so the
  'easy_install' tool can download and install it for you. Just type
  'easy_install allmydata-tahoe' from any shell. That will download the most
  recent Tahoe source tarball, unpack it in a temporary directory, install it
  to the standard location, then download and install any easy_install-able
  dependencies that you need (setuptools, zfec, foolscap, simplejson, and
  nevow).  (This will work only if you already have the dependencies listed
  in the MANUAL DEPENDENCIES section, above.)

 The Setuptools Way:

  Get the source code (see above).

  Run 'python setup.py install'. This will compile and install the Tahoe code
  to the standard location for your operating system (on unix, that is
  somewhere inside /usr/lib/). It will also acquire and install the
  easy_install-able dependencies (setuptools, zfec, foolscap, simplejson, and
  nevow) to the same place.

  (To install it to a non-standard location, see
  http://allmydata.org/trac/tahoe/wiki/SetuptoolsAndGNUStow .)

 The Running-In-Place Way:

  You can use Tahoe without installing it.  The steps are these:

  1. Get the source code (see above).

  2. Run "make build-deps" to install the easy_install-able dependencies
     (setuptools, zfec, foolscap, simplejson, and nevow) into a local
     subdirectory of the Tahoe source distribution.

  3. Build Tahoe by running "make".

  4. Once you've built it then you can execute "./bin/allmydata-tahoe". (When
     the allmydata-tahoe script is in a Tahoe source distribution, it adds
     the necessary directory to the Python "sys.path".  It also looks for any
     dependencies that you installed by "make build-deps" and includes them
     in the sys.path.)  See the RUNNING section, below.

 The Debian Way:

  The Debian Way is to build .deb files which you can then install with
  "dpkg".

  This requires certain debian packages (build-essential, fakeroot,
  devscripts, debhelper, cdbs) to be installed first, since they are used to
  construct the Tahoe .deb files. A full list of these required packages can
  be found in the "Build-Depends" line in the misc/DIST/debian/control in the
  top-level tahoe directory (replacing the word DIST with etch, dapper, edgy,
  or feisty as appropriate).

  Get the source code (see above).

  If you're running on a debian system, run 'make deb-etch', 'make deb-sid',
  'make deb-edgy', or 'make deb-feisty' from within the tahoe top-level
  directory to construct a debian package named 'allmydata-tahoe' which you
  can then install with dpkg.


TESTING THAT IT IS PROPERLY INSTALLED

 'make check-deps' checks that all of the required Python package
 dependencies are installed.

 'make test' runs the unit test suites.  (This can take a long time on
 slow computers.  There are a lot of tests and some of them do a lot of
 public-key cryptography.)

 Executing the allmydata-tahoe script from the "bin" subdirectory will work
 only if Tahoe itself is installed, either because it is installed into the
 local subdirectory (as per "The Running-In-Place Way") or because it is
 installed into your system (as per the other three ways of installing).


RUNNING:

 Run the "allmydata-tahoe" executable.

 If you installed "The Running-In-Place Way", then it is in your source tree,
 in the "bin" subdirectory thereof.  If you installed in one of the other
 three ways, then it has been installed into your operating system's
 filesystem, perhaps in "/usr/bin" on Unix, or in "C:\Python25\Scripts" on
 Window.

 The "allmydata-tahoe" utility is used to create, start, and stop nodes.
 Each node lives in a separate base directory, inside of which you can add
 files to configure and control the node.  Nodes also read and write files
 within that directory.

 A grid consists of a single central 'introducer and vdrive' node and one or
 more 'client' nodes.  If you are joining an existing grid, the
 introducer-and-vdrive node will already be running, and you'll just need to
 create a client node.  If you're creating a brand new grid, you'll need to
 create both an introducer-and-vdrive and a client (and then invite other
 people to create their own client nodes and join your grid).

 The introducer (-and-vdrive) node is constructed by running 'allmydata-tahoe
 create-introducer --basedir $HERE'.  Once constructed, you can start the
 introducer by running 'allmydata-tahoe start --basedir $HERE' (or, if you
 are already in the introducer's base directory, just type 'allmydata-tahoe
 start').  Inside that base directory, there will be a pair of files
 'introducer.furl' and 'vdrive.furl'.  Make a copy of these, as they'll be
 needed on the client nodes.

 To construct a client node, pick a new working directory for it, then run
 'allmydata-tahoe create-client --basedir $HERE'.  Copy the two .furl files
 from the introducer into this new directory, then run 'allmydata-tahoe start
 --basedir $HERE'.  After that, the client node should be off and running.
 The first thing it will do is connect to the introducer and introduce itself
 to all other nodes on the grid.  You can follow its progress by looking at
 the $HERE/logs/twistd.log file.

 To actually use the client, enable the web interface by writing a port
 number (like "8123") into a file named $HERE/webport and then restarting the
 node with 'allmydata-tahoe restart --basedir $HERE'. This will prompt the
 client node to run a webserver on the desired port, through which you can
 view, upload, download, and delete files. This 'webport' file is actually a
 "strports specification", defined in
 http://twistedmatrix.com/documents/current/api/twisted.application.strports.html
 , so you can have it only listen on a local interface by writing
 "tcp:8123:interface=127.0.0.1" to this file, or make it use SSL by writing
 "ssl:8123:privateKey=mykey.pem:certKey=cert.pem" instead.

 A client node directory can also be created without installing the code
 first.  Just use 'make create-client', and a new directory named 'CLIENTDIR'
 will be created inside the top of the source tree.  Copy the relevant .furl
 files in, set the webport, then start the node by using 'make start-client'.
 To stop it again, use 'make stop-client'.  Similar makefile targets exist
 for making and running an introducer node.

 If you are behind a firewall and you can configure your firewall to forward
 TCP connections on a port to the computer running your Tahoe node, then you
 can configure the Tahoe node to announce itself as being available on that
 IP address and port.  The way to do this is to create a file named
 $HERE/advertised_ip_addresses, in which you can put IP addresses and port
 numbers in "dotted-quad:port" form, e.g. "209.97.232.113:1345".  You can put
 multiple IP-address-and-port-number entries into this file, on separate
 lines.

 There is a public grid available for testing.  See
 http://allmydata.org/trac/tahoe/wiki/TestGrid for the necessary .furl files.


LICENCE:

 This program is free software; you can redistribute it and/or modify it
 under the terms of the GNU General Public License as published by the Free
 Software Foundation; either version 2 of the License, or (at your option)
 any later version, with the added permission that, if you become obligated
 to release a derived work under this licence (as per section 2.b), you may
 delay the fulfillment of this obligation for up to 12 months.  If you are
 obligated to release code under section 2.b of this licence, you are
 obligated to release it under these same terms, including the 12-month grace
 period clause.  See the COPYING file for details.


EASY_INSTALLABLE DEPENDENCIES

The following Python packages are required, but normally they are
automatically installed as a side-effect of installing Tahoe.

 + Python setuptools (build and distribution tool)

   http://peak.telecommunity.com/DevCenter/EasyInstall#installation-instructions

   The Tahoe install process will automatically download and install
   setuptools if it is not present.  However, if an old, incompatible version
   of setuptools is present (< v0.6c6 on Cygwin, or < v0.6a9 on other
   platforms), then the install will fail.

   If the install fails due to your current version of setuptools being
   incompatible, please either upgrade or uninstall your version of
   setuptools and re-run the install.

 + zfec (erasure coding library)

   http://cheeseshop.python.org/pypi/zfec

   zfec is packaged in a setuptools-compatible way and included in the Python
   Package Index (PyPI), so it will be automatically installed when you
   install Tahoe (see INSTALLING).  It can be manually installed by running
   "easy_install zfec".

 + Python foolscap (secure remote object library)

   http://cheeseshop.python.org/pypi/foolscap

   foolscape is packaged in a setuptools-compatible way and included in the
   Python Package Index (PyPI), so it will be automatically installed when
   you install Tahoe (see INSTALLING).  It can be manually installed by
   running "easy_install foolscap".

 + Python simplejson (JSON parser)

   http://cheeseshop.python.org/pypi/simplejson

   simplejson is packaged in a setuptools-compatible way and included in the
   Python Package Index (PyPI), so it will be automatically installed when
   you install Tahoe (see INSTALLING).  It can be manually installed by
   running "easy_install simplejson".

 + Python Nevow (0.6.0 or later) (web presentation language)

   http://divmod.org/trac/wiki/DivmodNevow

   Note that the current version of Nevow (0.9.18) requires Twisted 2.4.0 or
   later.

   Nevow is packaged in a setuptools-compatible way and included in the
   Python Package Index (PyPI), so it will be automatically installed when
   you install Tahoe (see INSTALLING).  It can be manually installed by
   running "easy_install nevow".