From: Zooko O'Whielacronx Date: Thu, 5 Jun 2008 20:55:05 +0000 (-0700) Subject: setup and docs: various improvements to setup and docs X-Git-Tag: allmydata-tahoe-1.1.0~47 X-Git-Url: https://git.rkrishnan.org/copyable.html?a=commitdiff_plain;h=970edc5c65aeecaabc51f5b0803e8740698147dd;p=tahoe-lafs%2Ftahoe-lafs.git setup and docs: various improvements to setup and docs Remove docs/install-details.html and README.win32 for now (see #282). Remove checks for pywin32 and pyopenssl in Makefile -- that is (or will be) automated by setuptools. Remove twisted from setup_requires. This causes the problem in which Nevow doesn't declare its dependency on Twisted (#440) to yield a clear ImportError mentioning Twisted and to fail repeatedly, rather than yielding a weird ImportError and working on the second identical attempt. Fix Makefile to set PATH so that trial and twistd can be found by "make test" after Twisted was installed into support/ during "make" --- diff --git a/Makefile b/Makefile index 27df4eeb..f8251dfc 100644 --- a/Makefile +++ b/Makefile @@ -22,7 +22,6 @@ ifeq ($(PLAT),win32) SUPPORT = $(shell cygpath -w $(shell pwd))\support SUPPORTLIB := $(SUPPORT)\Lib\site-packages SRCPATH := $(shell cygpath -w $(shell pwd)/src) - CHECK_PYWIN32_DEP := check-pywin32-dep INNOSETUP := $(shell cygpath -au "$(PROGRAMFILES)/Inno Setup 5/Compil32.exe") else PYVER=$(shell $(PYTHON) misc/pyver.py) @@ -48,8 +47,8 @@ else PP=PYTHONPATH="$(SUPPORTLIB)" endif -TRIALCMD = $(shell $(PP) $(PYTHON) misc/find_trial.py) -TRIAL=PYTHONUNBUFFERED=1 $(TRIALCMD) --rterrors $(REACTOROPT) +TRIALCMD = $(shell PATH=${PATH}:${PWD}/support/bin $(PP) $(PYTHON) misc/find_trial.py) +TRIAL=PATH=${PATH}:${PWD}/support/bin PYTHONUNBUFFERED=1 $(TRIALCMD) --rterrors $(REACTOROPT) .PHONY: make-version build @@ -113,7 +112,7 @@ endif # TESTING -.PHONY: check-all-deps $(CHECK_PYWIN32_DEP) signal-error-deps signal-error-pywin32-dep test test-figleaf figleaf-output +.PHONY: signal-error-deps test test-figleaf figleaf-output signal-error-deps: @@ -124,37 +123,11 @@ signal-error-deps: @echo exit 1 -signal-error-pywin32-dep: - @echo - @echo - @echo "ERROR: the pywin32 dependency is not in place. Please see docs/install.html for help on installing dependencies." - @echo - @echo - exit 1 - -signal-error-pyopenssl-dep: - @echo - @echo - @echo "ERROR: the pyOpenSSL dependency is not in place (note that pyOpenSSL required OpenSSL). Please see docs/install.html for help on installing dependencies." - @echo - @echo - exit 1 - check-auto-deps: @$(PP) $(PYTHON) -c 'import _auto_deps ; _auto_deps.require_auto_deps()' || $(MAKE) signal-error-deps -check-all-deps: check-deps check-auto-deps - -check-pywin32-dep: - @$(PP) $(PYTHON) -c 'import win32process' || $(MAKE) signal-error-pywin32-dep - -check-pyopenssl-dep: - @$(PP) $(PYTHON) -c 'import OpenSSL' || $(MAKE) signal-error-pyopenssl-dep - -check-deps: $(CHECK_PYWIN32_DEP) check-pyopenssl-dep - .checked-deps: - $(MAKE) check-all-deps + $(MAKE) check-auto-deps touch .checked-deps # you can use 'make test TEST=allmydata.test.test_introducer' to run just diff --git a/README b/README index 464ba59b..7c4b3dab 100644 --- a/README +++ b/README @@ -1,6 +1,8 @@ -Welcome to the Tahoe project, a secure, decentralized, fault-tolerant +Welcome to the Tahoe project [1], a secure, decentralized, fault-tolerant filesystem. All of the source code is available under a Free Software, Open -Source licence. +Source licence (or two). Please see docs/about.html for an introduction, docs/install.html for install instructions, docs/running.html for usage instructions. + +[1] http://allmydata.org diff --git a/README.win32 b/README.win32 deleted file mode 100644 index 1b421f0b..00000000 --- a/README.win32 +++ /dev/null @@ -1,91 +0,0 @@ -BUILDING ALLMYDATA.ORG TAHOE ON WINDOWS - -There are three ways to do it: - -OPTION 1: ALL CYGWIN ALL THE TIME - -If you are building on Windows, then the easy way is to install cygwin -and use the cygwin version of Python and the cygwin versions of all -dependencies (which will happen naturally if you follow the main -README file -- note that you cannot use Windows-native versions of any -of the dependencies -- they all have to be the cygwin versions). So -if you are taking this approach then you don't need to read the rest -of this README.win32 file at all. - -OPTION 2: CYGWIN TOOLS TO BUILD WINDOWS-NATIVE LIBRARIES - -The second-easiest way is to install cygwin and use cygwin development -tools such as bash, GNU make, gcc, etc., but install the -Windows-native version of Python and the Windows-native versions of -all of the dependencies. If you create a distutils config file (as -per http://docs.python.org/inst/config-syntax.html ) and put "[build]" -then "compiler=mingw32" in it, then you can follow the rest of the -main README file and the dependencies will all be automatically built -(by the cygwin gcc compiler) as Windows-native libraries. This -README.win32 file contains some extra notes about how to take this -approach. - -Note that in this case, any library dependencies you install must be -built without linking against the cygwin.dll library. You can cause -cygwin to build native libraries without linking against cygwin.dll -by passing the '-mno-cygwin' flag to the compiler. For example, to -build cryptopp this way you can use the command - CXX='g++ -mno-cygwin' make - -Note that cryptopp's cryptest.exe tool may fail to build. To build -that you might need to add the ISMINGW option to cryptopp's make e.g. - CXX='g++ -mno-cygwin' make ISMINGW=1 - -OPTION 3: OTHER BUILD TOOLS - -The third-easiest way is to use a Microsoft compiler or some other -compiler. Our README files do not currently explain how to do that. -You are on your own for now, but please feel free to contribute a -document which explains how to build all these dependencies using your -favorite compiler.XXX MikeB: the previous paragraph is false -- please -fix it! --Zooko - - -Okay, here are some notes about following "OPTION 2: CYGWIN TOOLS TO BUILD -WINDOWS-NATIVE LIBRARIES" approach: - - -EXTRA MANUAL DEPENDENCIES - -In addition to the dependencies listed in the main README file, you -also need the following: - - + the pywin32 package (210 or later) - - http://sourceforge.net/projects/pywin32/ - - -NOTES ABOUT BUILDING OPENSSL - -In order to compile the tahoe source you need to have libeay32.dll version -0.9.8.5 and ssleay32.dll version 0.9.8.5 or newer installed. If you find that -you have the wrong version of either of these dlls, you can download and -compile openssl from http://openssl.org. - -You will need to have perl installed to compile openssl. One place where you -can find a version of perl is http://www.activestate.com/products/activeperl. - -If you want to compile openssl using Visual Studio, you may find adding your -Visual Studio bin, include and lib directories to your %PATH% environment -variable helpful. For example, if you're compiling with VS2005: - - "C:\Program Files\Microsoft Visual Studio\VC98\Bin;C:\Program Files\ - Microsoft Visual Studio 8\VC\include;C:\Program Files\Microsoft Visual - Studio 8\VC\PlatformSDK\Include;C:\Program Files\Microsoft Visual Studio - 8\VC\lib;C:\Program Files\Microsoft Visual Studio 8\VC\PlatformSDK\Lib" - - -NOTES ABOUT INSTALLING PYOPENSSL - -To install PyOpenSSL on Windows-native, download this: - -http://allmydata.org/source/pyOpenSSL-0.6.win32-py2.5.exe - -or for Python 2.4, this: - -http://allmydata.org/source/pyOpenSSL-0.6.win32-py2.4.exe diff --git a/_auto_deps.py b/_auto_deps.py index 9cbbba6a..c5125230 100644 --- a/_auto_deps.py +++ b/_auto_deps.py @@ -1,5 +1,5 @@ install_requires=["zfec >= 1.1.0", - "foolscap >= 0.2.5", + "foolscap[secure_connections] >= 0.2.8", "simplejson >= 1.4", # pycryptopp < 0.5 had a bug which, using a Microsoft @@ -9,7 +9,7 @@ install_requires=["zfec >= 1.1.0", "pycryptopp >= 0.5", "nevow >= 0.6.0", "zope.interface", - "twisted >= 2.4.0", + "Twisted >= 2.4.0", # we require 0.6c8 to build, but can handle older versions # to run diff --git a/docs/install-details.html b/docs/install-details.html deleted file mode 100644 index 42d3c05d..00000000 --- a/docs/install-details.html +++ /dev/null @@ -1,369 +0,0 @@ - - - - Tahoe Install Details - - - - - - - - -
  • pywin32 - >= v210 -- required only for Windows (and not for Cygwin)
    • - -
    -DEPENDENCIES:
-
-If you aren't getting a pre-compiled binary, then you'll have to ensure that
-the following packages are installed before you install Tahoe.
-
-There are two kinds of dependencies, "manual dependencies" and
-"easy_install-able dependencies".  The latter kind are normally automatically
-satisfied for you when you install Tahoe, but if something goes wrong, please
-see the EASY_INSTALLABLE DEPENDENCIES section below.
-
-All of the manual dependencies can probably be installed through your
-standard package management tool if you are running on a modern Unix
-operating system.  For example, on an debian-like system, you can do "sudo
-apt-get install build-essential python-dev python-twisted python-pyopenssl".
-
-The Manual Dependencies:
-
- + a C compiler (language)
-
- + GNU make (build tool)
-
- + Python 2.4 or newer (tested against 2.4.4, and 2.5.1 -- note that 2.4.1 is
-   known not to work due to a bug in its base-32 encoder), including
-   development headers i.e. "Python.h" (language)
-
-   http://python.org/
-
- + Twisted Python (tested against 2.2.0, 2.4.0, and 2.5.0) (network and
-   operating system integration library)
-
-   http://twistedmatrix.com/
-
-   Installing from the Twisted source tarball works on all known supported
-   platforms, including cygwin.
-
-   You need the following subpackages, which are included in the default
-   Twisted distribution:
-
-   * core (the standard Twisted package)
-   * web, trial, conch
-
-   Twisted requires that you manually install zope.interface, a copy of which
-   is included in the Twisted distribution. Note that Twisted does *not*
-   require the entire Zope distribution, merely the much smaller
-   zope.interface component.
-
- + OpenSSL, including development headers (cryptography library)
-
-   http://openssl.org
-  
- + Crypto++, including development headers (cryptography library)
-
-   http://cryptopp.com
-
- + Python PyOpenSSL (0.6 or later) (secure transport layer)
-
-   http://pyopenssl.sourceforge.net
-
-   To install PyOpenSSL on cygwin, install the OpenSSL development libraries
-   with the cygwin package management tool, then get the pyOpenSSL source
-   code, cd into it, and run "python ./setup.py install".
-
-
-
-xxx
-
-
-GETTING THE SOURCE CODE:
-
-You need the source code if you are going to install The Debian Way, The
-Setuptools Way, or The Running-In-Place Way (see below).  You do not need the
-source code if you are getting precompiled binaries for Debian or Ubuntu (see
-above), or if you are going to install The easy_install Way (see below).
-
-The code is available via darcs by running the following command:
-
-darcs get http://allmydata.org/source/tahoe/trunk tahoe
-
-This will create a directory named "tahoe" in the current working directory
-and put a copy of the latest source code into it.  Later, if you want to get
-any new changes, then cd into that directory and run the command "darcs
-pull".
-
-Tarballs of sources are available at:
-
-http://allmydata.org/source/tahoe/
-
-
-INSTALLING:
-
-There are four ways to do it: The easy_install Way, The Setuptools Way, The
-Running-In-Place Way, and The Debian Way.  Choose one.  If you're not sure,
-choose the easy_install way.
-
- The easy_install Way:
-
-  You don't need to download the source code first.  You do need to have the
-  "easy_install" tool installed first:
-
-  http://peak.telecommunity.com/DevCenter/EasyInstall#installing-easy-install
-
-  The 'easy_install' tool can download and install tahoe for you. Just type
-  'easy_install allmydata-tahoe' from any shell. That will download the most
-  recent Tahoe source tarball, unpack it in a temporary directory, install it
-  to the standard location, then download and install any easy_install-able
-  dependencies that you need (setuptools, zfec, foolscap, simplejson, nevow,
-  and pycryptopp).  (This will work only if you have already installed the
-  dependencies listed in the MANUAL DEPENDENCIES section, above.)
-
-  The end result will be that the Tahoe code is installed to the standard
-  location for libraries on your operating system (on unix, that is somewhere
-  inside /usr/lib/), and the "tahoe" executable will be installed to the
-  standard location for executables on operating system.
-
- The Setuptools Way:
-
-  Get the source code (see above).
-
-  Run 'python setup.py install'. This will compile and install the Tahoe code
-  to the standard location for your operating system (on unix, that is
-  somewhere inside /usr/lib/). It will also acquire and install the
-  easy_install-able dependencies (setuptools, zfec, foolscap, simplejson,
-  nevow, and pycryptopp) to the same place.  (This will work only if you have
-  already installed the dependencies listed in the MANUAL DEPENDENCIES
-  section, above.)
-
-  (To install it to a non-standard location, see
-  http://allmydata.org/trac/tahoe/wiki/SetuptoolsAndGNUStow .)
-
-  The end result will be that the Tahoe code is installed to the standard
-  location for libraries on your operating system (on unix, that is somewhere
-  inside /usr/lib/), and the "tahoe" executable will be installed to the
-  standard location for executables on operating system.
-
- The Running-In-Place Way:
-
-  You can use Tahoe without installing it.  The steps are these:
-
-  1. Get the source code (see above).
-
-  2. Run "make build-auto-deps" to install the easy_install-able dependencies
-     (setuptools, zfec, foolscap, simplejson, nevow, and pycryptopp) into a
-     local subdirectory of the Tahoe source distribution. (Note that when the
-     dependent libraries are updated, you should use "make clean" before
-     "make build-auto-deps" to make sure you will get the newest versions).
-
-  3. Build Tahoe by running "make".
-
-  4. Once you've built it then you can execute "./bin/tahoe". (When the tahoe
-     script is in a Tahoe source distribution, it adds the necessary
-     directory to the Python "sys.path". It also looks for any dependencies
-     that you installed by "make build-auto-deps" and includes them in the
-     sys.path.) See the RUNNING section, below.
-
- The Debian Way:
-
-  The Debian Way is to build .deb files which you can then install with
-  "dpkg".
-
-  This requires certain debian packages (build-essential, fakeroot,
-  devscripts, debhelper, cdbs) to be installed first, since they are used to
-  construct the Tahoe .deb files. A full list of these required packages can
-  be found in the "Build-Depends" line in the misc/DIST/debian/control in the
-  top-level tahoe directory (replacing the word DIST with etch, dapper, edgy,
-  or feisty as appropriate).
-
-  Get the source code (see above).
-
-  If you're running on a debian system, run 'make deb-etch', 'make deb-sid',
-  'make deb-edgy', or 'make deb-feisty' from within the tahoe top-level
-  directory to construct a debian package named 'allmydata-tahoe' which you
-  can then install with dpkg.
-
-
-TESTING THAT IT IS PROPERLY INSTALLED
-
- If you have gotten the source code, then you can run 'make check-deps'
- checks that all of the required Python package dependencies are installed.
- You can run 'make test' runs the unit test suites.  (This can take a long
- time on slow computers.  There are a lot of tests and some of them do a lot
- of public-key cryptography.)
-
- Executing the tahoe script from the "bin" subdirectory will work only if
- Tahoe itself is installed, either because it is installed into the local
- subdirectory (as per "The Running-In-Place Way") or because it is installed
- into your system (as per the other three ways of installing).
-
-
-RUNNING:
-
- Run the "tahoe" executable.
-
- If you installed "The Running-In-Place Way", then it is in your source tree,
- in the "bin" subdirectory thereof.  If you installed in one of the other
- three ways, then it has been installed into your operating system's
- filesystem, perhaps in "/usr/bin" on Unix, or in "C:\Python25\Scripts" on
- Window.
-
- The "tahoe" utility is used to create, start, and stop nodes. Each node
- lives in a separate base directory, inside of which you can add files to
- configure and control the node. Nodes also read and write files within that
- directory.
-
- A grid consists of a single central 'introducer and vdrive' node and one or
- more 'client' nodes.  If you are joining an existing grid, the
- introducer-and-vdrive node will already be running, and you'll just need to
- create a client node.  If you're creating a brand new grid, you'll need to
- create both an introducer-and-vdrive and a client (and then invite other
- people to create their own client nodes and join your grid).
-
- The introducer (-and-vdrive) node is constructed by running 'tahoe
- create-introducer --basedir $HERE'. Once constructed, you can start the
- introducer by running 'tahoe start --basedir $HERE' (or, if you are already
- in the introducer's base directory, just type 'tahoe start'). Inside that
- base directory, there will be a pair of files 'introducer.furl' and
- 'vdrive.furl'. Make a copy of these, as they'll be needed on the client
- nodes.  (If you want to use a publically available test grid, get the
- introducer.furl and vdrive.furl files from
- http://allmydata.org/trac/tahoe/wiki/TestGrid instead of running your own
- introducer.)
-
- To construct a client node, pick a new working directory for it, then run
- 'tahoe create-client --basedir $HERE'. Copy the two .furl files from the
- introducer into this new directory, then run 'tahoe start --basedir $HERE'.
- After that, the client node should be off and running. The first thing it
- will do is connect to the introducer and introduce itself to all other nodes
- on the grid. You can follow its progress by looking at the
- $HERE/logs/twistd.log file.
-
- create-client will put port specification into a file named $HERE/webport,
- unless overridden by the --webport option to create-client. The presence of
- a port specification in the webport file prompts the client node to run a
- webserver on the desired port, through which you can view, upload, download,
- and delete files. The contents of the webport file is actually a "strports
- specification", defined in
- http://twistedmatrix.com/documents/current/api/twisted.application.strports.html
- , so you can have it only listen on a local interface by writing
- "tcp:8123:interface=127.0.0.1" to this file (that's what create-client does
- by default), or make it use SSL by writing
- "ssl:8123:privateKey=mykey.pem:certKey=cert.pem" instead.
-
- A client node directory can also be created without installing the code
- first.  Just use 'make create-client', and a new directory named 'CLIENTDIR'
- will be created inside the top of the source tree.  Copy the relevant .furl
- files in, set the webport, then start the node by using 'make start-client'.
- To stop it again, use 'make stop-client'.  Similar makefile targets exist
- for making and running an introducer node.
-
- If you are behind a firewall and you can configure your firewall to forward
- TCP connections on a port to the computer running your Tahoe node, then you
- can configure the Tahoe node to announce itself as being available on that
- IP address and port.  The way to do this is to create a file named
- $HERE/advertised_ip_addresses, in which you can put IP addresses and port
- numbers in "dotted-quad:port" form, e.g. "209.97.232.113:1345".  You can put
- multiple IP-address-and-port-number entries into this file, on separate
- lines.
-
- There is a public grid available for testing. The necessary .furl files are
- in docs/testgrid/*.furl . More information is available on
- http://allmydata.org/trac/tahoe/wiki/TestGrid .
-
    - -

    -The code is retrievable using the darcs -revision control tool by running the following command: -

    - -
    -darcs get http://allmydata.org/source/tahoe/trunk tahoe
-
    - -

    -This will create a directory named 

    tahoe
    in the current working -directory and put a copy of the latest source code into it. Later, if -you want to get any new changes, then cd into that directory and run -the command 
    darcs pull
    . -

    - - - -operating system -linux -cygwin -Windows -Mac OS X -Solaris -
    location of 
    tahoe
    executable
    /usr/bin
    /usr/bin
    C:\Python25\Scripts
    /Frameworks/Python/Versions/2.5/binpre>(XXX double-check this)
    /usr/bin
    (XXX double-check this)
    - -
    - + Python setuptools (build and distribution tool) >= v0.6c6
-
-   http://peak.telecommunity.com/DevCenter/EasyInstall#installation-instructions
-
-   The Tahoe install process will automatically use its own bundled copy
-   setuptools if a sufficiently new version of setuptools is not installed on
-   the system.  Unlike It will not install it into the system -- this is a build
-   dependency, not an install dependency.
-
-
-EASY_INSTALLABLE DEPENDENCIES
-
-The following Python packages are required, but they are bundled with Tahoe (in
-the misc/dependencies directory), and are built by make
-build-auto-deps.  If you install Tahoe using The Setuptools Way or The
-easy_install Way then these packages will automatically be installed along with
-Tahoe.
-
- + zfec (erasure coding library) >= v1.3.0
-
-   http://cheeseshop.python.org/pypi/zfec
-
- + foolscap (secure remote object library) >= v0.2.3
-
-   http://cheeseshop.python.org/pypi/foolscap
-
- + simplejson (JSON parser) >= v1.7.3
-
-   http://cheeseshop.python.org/pypi/simplejson
-
- + nevow (web presentation language) >= v0.6.0
-
-   http://divmod.org/trac/wiki/DivmodNevow
-
-   Note that Nevow >= v0.9.18 requires Twisted >= v2.4.0. If you are on
-   Ubuntu 6.06 ("dapper") or Debian 3.1 ("sarge"), this means you have to
-   install Twisted >= v2.4.0 yourself, or else install Nevow v0.6.0 in egg
-   form yourself (e.g. run easy_install nevow == 0.6.0).
-
- + pycryptopp (Python crypto library) >= v0.2.9
-
-   http://pypi.python.org/pypi/pycryptopp
-
- + zope.interface (programming language extensions) >= v3.1.0
-
-   http://pypi.python.org/pypi/pycryptopp
-
-   Tahoe does not require the entire Zope package, merely
-   the much smaller zope.interface component.
-
-
-
-xxx
-If you are behind a firewall and you can configure your firewall to
-forward TCP connections on a port to the computer running your Tahoe
-node, then you can configure the Tahoe node to announce itself as
-being available on that IP address and port.  The way to do this is to
-create a file named $HERE/advertised_ip_addresses, in which you can
-put IP addresses and port numbers in "dotted-quad:port" form,
-e.g. "209.97.232.113:1345".  You can put multiple
-IP-address-and-port-number entries into this file, on separate lines.
-
    - - diff --git a/docs/install.html b/docs/install.html index 9398554e..e1635edc 100644 --- a/docs/install.html +++ b/docs/install.html @@ -10,13 +10,13 @@

    About Tahoe

    -

    Welcome to the Tahoe project, a secure, decentralized, fault-tolerant filesystem.

    +

    Welcome to the Tahoe project, a secure, decentralized, fault-tolerant filesystem.

    See the about page for more information.

    How To Install Tahoe

    -

    This is the default procedure to install from source. It has been verified to work on Windows, Cygwin, Mac, Linux, and Solaris. It's likely to work on other platforms. If you encounter problems following this procedure, or for more details such as platform-specific instructions, please see install-details.html. +

    This is the default procedure to install from source. It has been verified to work on Windows, Cygwin, Mac, Linux, and Solaris. It's likely to work on other platforms. If you have trouble with this install process, please write to the tahoe-dev mailing list, where friendly hackers will help you out.

    Satisfy the Dependencies

    @@ -28,8 +28,6 @@
  • GNU make
  • Python >= v2.4.2 including development headers i.e. "Python.h"
    • - -
  • OpenSSL >= v0.9.7, including development headers
    • diff --git a/setup.py b/setup.py index 76d1a75b..d12da7e0 100644 --- a/setup.py +++ b/setup.py @@ -103,7 +103,6 @@ dependency_links.append("http://allmydata.org/trac/tahoe/wiki/Dependencies") setup_requires = [] setup_requires.append('pyutil >= 1.3.16') # used by the Windows installer builder, see misc/sub-ver.py -setup_requires.append('twisted >= 2.4.0') # for trial, and because foolscap <= 0.2.5 imports it in its setup.py # darcsver is needed only if you want "./setup.py darcsver" to write a new # version stamp in src/allmydata/_version.py, with a version number derived from