* web-API service
* SFTP service
* FTP service
-* drop-upload service
+* Magic Folder service
* helper service
* storage service.
instructions on configuring these services, and the ``[sftpd]`` and
``[ftpd]`` sections of ``tahoe.cfg``.
-Drop-Upload
+Magic Folder
- As of Tahoe-LAFS v1.9.0, a node running on Linux can be configured to
- automatically upload files that are created or changed in a specified
- local directory. See drop-upload.rst_ for details.
+ A node running on Linux or Windows can be configured to automatically
+ upload files that are created or changed in a specified local directory.
+ See `magic-folder.rst`_ for details.
.. _download-status.rst: frontends/download-status.rst
.. _CLI.rst: frontends/CLI.rst
.. _FTP-and-SFTP.rst: frontends/FTP-and-SFTP.rst
-.. _drop-upload.rst: frontends/drop-upload.rst
+.. _magic-folder.rst: frontends/magic-folder.rst
Storage Server Configuration
+++ /dev/null
-.. -*- coding: utf-8-with-signature -*-
-
-===============================
-Tahoe-LAFS Drop-Upload Frontend
-===============================
-
-1. `Introduction`_
-2. `Configuration`_
-3. `Known Issues and Limitations`_
-
-
-Introduction
-============
-
-The drop-upload frontend allows an upload to a Tahoe-LAFS grid to be triggered
-automatically whenever a file is created or changed in a specific local
-directory. This is a preview of a feature that we expect to support across
-several platforms, but it currently works only on Linux.
-
-The implementation was written as a prototype at the First International
-Tahoe-LAFS Summit in June 2011, and is not currently in as mature a state as
-the other frontends (web, CLI, SFTP and FTP). This means that you probably
-should not keep important data in the upload directory, and should not rely
-on all changes to files in the local directory to result in successful uploads.
-There might be (and have been) incompatible changes to how the feature is
-configured. There is even the possibility that it may be abandoned, for
-example if unsolveable reliability issues are found.
-
-We are very interested in feedback on how well this feature works for you, and
-suggestions to improve its usability, functionality, and reliability.
-
-
-Configuration
-=============
-
-The drop-upload frontend runs as part of a gateway node. To set it up, you
-need to choose the local directory to monitor for file changes, and a mutable
-directory on the grid to which files will be uploaded.
-
-These settings are configured in the ``[drop_upload]`` section of the
-gateway's ``tahoe.cfg`` file.
-
-``[drop_upload]``
-
-``enabled = (boolean, optional)``
-
- If this is ``True``, drop-upload will be enabled. The default value is
- ``False``.
-
-``local.directory = (UTF-8 path)``
-
- This specifies the local directory to be monitored for new or changed
- files. If the path contains non-ASCII characters, it should be encoded
- in UTF-8 regardless of the system's filesystem encoding. Relative paths
- will be interpreted starting from the node's base directory.
-
-In addition, the file ``private/drop_upload_dircap`` must contain a
-writecap pointing to an existing mutable directory to be used as the target
-of uploads. It will start with ``URI:DIR2:``, and cannot include an alias
-or path.
-
-After setting the above fields and starting or restarting the gateway,
-you can confirm that the feature is working by copying a file into the
-local directory. Then, use the WUI or CLI to check that it has appeared
-in the upload directory with the same filename. A large file may take some
-time to appear, since it is only linked into the directory after the upload
-has completed.
-
-The 'Operational Statistics' page linked from the Welcome page shows
-counts of the number of files uploaded, the number of change events currently
-queued, and the number of failed uploads. The 'Recent Uploads and Downloads'
-page and the node log_ may be helpful to determine the cause of any failures.
-
-.. _log: ../logging.rst
-
-
-Known Issues and Limitations
-============================
-
-This frontend only works on Linux. There is an even-more-experimental
-implementation for Windows (`#1431`_), and a ticket to add support for
-Mac OS X and BSD-based systems (`#1432`_).
-
-Subdirectories of the local directory are not monitored. If a subdirectory
-is created, it will be ignored. (`#1433`_)
-
-If files are created or changed in the local directory just after the gateway
-has started, it might not have connected to a sufficient number of servers
-when the upload is attempted, causing the upload to fail. (`#1449`_)
-
-Files that were created or changed in the local directory while the gateway
-was not running, will not be uploaded. (`#1458`_)
-
-The only way to determine whether uploads have failed is to look at the
-'Operational Statistics' page linked from the Welcome page. This only shows
-a count of failures, not the names of files. Uploads are never retried.
-
-The drop-upload frontend performs its uploads sequentially (i.e. it waits
-until each upload is finished before starting the next), even when there
-would be enough memory and bandwidth to efficiently perform them in parallel.
-A drop-upload can occur in parallel with an upload by a different frontend,
-though. (`#1459`_)
-
-If there are a large number of near-simultaneous file creation or
-change events (greater than the number specified in the file
-``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events
-could be missed. This is fairly unlikely under normal circumstances, because
-the default value of ``max_queued_events`` in most Linux distributions is
-16384, and events are removed from this queue immediately without waiting for
-the corresponding upload to complete. (`#1430`_)
-
-Some filesystems may not support the necessary change notifications.
-So, it is recommended for the local directory to be on a directly attached
-disk-based filesystem, not a network filesystem or one provided by a virtual
-machine.
-
-Attempts to read the mutable directory at about the same time as an uploaded
-file is being linked into it, might fail, even if they are done through the
-same gateway. (`#1105`_)
-
-When a local file is changed and closed several times in quick succession,
-it may be uploaded more times than necessary to keep the remote copy
-up-to-date. (`#1440`_)
-
-Files deleted from the local directory will not be unlinked from the upload
-directory. (`#1710`_)
-
-The ``private/drop_upload_dircap`` file cannot use an alias or path to
-specify the upload directory. (`#1711`_)
-
-Files are always uploaded as immutable. If there is an existing mutable file
-of the same name in the upload directory, it will be unlinked and replaced
-with an immutable file. (`#1712`_)
-
-If a file in the upload directory is changed (actually relinked to a new
-file), then the old file is still present on the grid, and any other caps to
-it will remain valid. See `docs/garbage-collection.rst`_ for how to reclaim
-the space used by files that are no longer needed.
-
-Unicode names are supported, but the local name of a file must be encoded
-correctly in order for it to be uploaded. The expected encoding is that
-printed by ``python -c "import sys; print sys.getfilesystemencoding()"``.
-
-.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105
-.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430
-.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431
-.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432
-.. _`#1433`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1433
-.. _`#1440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1440
-.. _`#1449`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1449
-.. _`#1458`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1458
-.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459
-.. _`#1710`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1710
-.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711
-.. _`#1712`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1712
-
-.. _docs/garbage-collection.rst: ../garbage-collection.rst
-
--- /dev/null
+.. -*- coding: utf-8-with-signature -*-
+
+================================
+Tahoe-LAFS Magic Folder Frontend
+================================
+
+1. `Introduction`_
+2. `Configuration`_
+3. `Known Issues and Limitations`_
+
+
+Introduction
+============
+
+The Magic Folder frontend synchronizes local directories on two or more
+clients, using a Tahoe-LAFS grid for storage. Whenever a file is created
+or changed under the local directory of one of the clients, the change is
+propagated to the grid and then to the other clients.
+
+The implementation of the "drop-upload" frontend, on which Magic Folder is
+based, was written as a prototype at the First International Tahoe-LAFS
+Summit in June 2011. In 2015, with the support of a grant from the
+`Open Technology Fund`_, it was redesigned and extended to support
+synchronization between clients. It currently works on Linux and Windows.
+
+Magic Folder is not currently in as mature a state as the other frontends
+(web, CLI, SFTP and FTP). This means that you probably should not rely on
+all changes to files in the local directory to result in successful uploads.
+There might be (and have been) incompatible changes to how the feature is
+configured.
+
+We are very interested in feedback on how well this feature works for you, and
+suggestions to improve its usability, functionality, and reliability.
+
+.. _`Open Technology Fund`: https://www.opentech.fund/
+
+
+Configuration
+=============
+
+The Magic Folder frontend runs as part of a gateway node. To set it up, you
+must use the tahoe magic-folder CLI. For detailed information see our
+`Magic-Folder CLI design documentation`_. For a given Magic-Folder collective
+directory you need to run the ``tahoe magic-folder create`` command. After
+that the ``tahoe magic-folder invite`` command must used to generate an
+*invite code* for each member of the magic-folder collective. A confidential,
+authenticated communications channel should be used to transmit the invite code
+to each member, who will be joining using the ``tahoe magic-folder join``
+command.
+
+These settings are persisted in the ``[magic_folder]`` section of the
+gateway's ``tahoe.cfg`` file.
+
+``[magic_folder]``
+
+``enabled = (boolean, optional)``
+
+ If this is ``True``, Magic Folder will be enabled. The default value is
+ ``False``.
+
+``local.directory = (UTF-8 path)``
+
+ This specifies the local directory to be monitored for new or changed
+ files. If the path contains non-ASCII characters, it should be encoded
+ in UTF-8 regardless of the system's filesystem encoding. Relative paths
+ will be interpreted starting from the node's base directory.
+
+You should not normally need to set these fields manually because they are
+set by the ``tahoe magic-folder create`` and/or ``tahoe magic-folder join``
+commands. Use the ``--help`` option to these commands for more information.
+
+After setting up a Magic Folder collective and starting or restarting each
+gateway, you can confirm that the feature is working by copying a file into
+any local directory, and checking that it appears on other clients.
+Large files may take some time to appear.
+
+The 'Operational Statistics' page linked from the Welcome page shows
+counts of the number of files uploaded, the number of change events currently
+queued, and the number of failed uploads. The 'Recent Uploads and Downloads'
+page and the node log_ may be helpful to determine the cause of any failures.
+
+.. _log: ../logging.rst
+
+
+Known Issues and Limitations
+============================
+
+This feature only works on Linux and Windows. There is a ticket to add
+support for Mac OS X and BSD-based systems (`#1432`_).
+
+The only way to determine whether uploads have failed is to look at the
+'Operational Statistics' page linked from the Welcome page. This only shows
+a count of failures, not the names of files. Uploads are never retried.
+
+The Magic Folder frontend performs its uploads sequentially (i.e. it waits
+until each upload is finished before starting the next), even when there
+would be enough memory and bandwidth to efficiently perform them in parallel.
+A Magic Folder upload can occur in parallel with an upload by a different
+frontend, though. (`#1459`_)
+
+On Linux, if there are a large number of near-simultaneous file creation or
+change events (greater than the number specified in the file
+``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events
+could be missed. This is fairly unlikely under normal circumstances, because
+the default value of ``max_queued_events`` in most Linux distributions is
+16384, and events are removed from this queue immediately without waiting for
+the corresponding upload to complete. (`#1430`_)
+
+The Windows implementation might also occasionally miss file creation or
+change events, due to limitations of the underlying Windows API
+(ReadDirectoryChangesW). We do not know how likely or unlikely this is.
+(`#1431`_)
+
+Some filesystems may not support the necessary change notifications.
+So, it is recommended for the local directory to be on a directly attached
+disk-based filesystem, not a network filesystem or one provided by a virtual
+machine.
+
+The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files
+cannot use an alias or path to specify the upload directory. (`#1711`_)
+
+If a file in the upload directory is changed (actually relinked to a new
+file), then the old file is still present on the grid, and any other caps
+to it will remain valid. Eventually it will be possible to use
+`garbage collection`_ to reclaim the space used by these files; however
+currently they are retained indefinitely. (`#2440`_)
+
+Unicode filenames are supported on both Linux and Windows, but on Linux, the
+local name of a file must be encoded correctly in order for it to be uploaded.
+The expected encoding is that printed by
+``python -c "import sys; print sys.getfilesystemencoding()"``.
+
+On Windows, local directories with non-ASCII names are not currently working.
+(`#2219`_)
+
+On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C
+(it can only be killed using Task Manager or similar). (`#2218`_)
+
+.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430
+.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431
+.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432
+.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459
+.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711
+.. _`#2218`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2218
+.. _`#2219`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2219
+.. _`#2440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2440
+
+.. _`garbage collection`: ../garbage-collection.rst
+.. _`Magic-Folder CLI design documentation`: ../proposed/magic-folder/user-interface-design.rst
--- /dev/null
+=========================
+Magic Folder Set-up Howto
+=========================
+
+1. `This document`_
+2. `Preparation`_
+3. `Setting up a local test grid`_
+4. `Setting up Magic Folder`_
+5. `Testing`_
+
+
+This document
+=============
+
+This is preliminary documentation of how to set up the
+Magic Folder pre-release using a test grid on a single Linux
+or Windows machine, with two clients and one server. It is
+aimed at a fairly technical audience.
+
+For an introduction to Magic Folder and how to configure it
+more generally, see `docs/frontends/magic-folder.rst`_.
+
+It it possible to adapt these instructions to run the nodes on
+different machines, to synchronize between three or more clients,
+to mix Windows and Linux clients, and to use multiple servers
+(if the Tahoe-LAFS encoding parameters are changed).
+
+.. _`docs/frontends/magic-folder.rst`: ../docs/frontends/magic-folder.rst
+
+
+Preparation
+===========
+
+Linux
+-----
+
+Install ``git`` from your distribution's package manager.
+Then run these commands::
+
+ git clone -b 2438.magic-folder-stable.8 https://github.com/tahoe-lafs/tahoe-lafs.git
+ cd tahoe-lafs
+ python setup.py test
+
+The test suite usually takes about 15 minutes to run.
+Note that it is normal for some tests to be skipped.
+In the current branch, the Magic Folder tests produce
+considerable debugging output.
+
+If you see an error like ``fatal error: Python.h: No such file or directory``
+while compiling the dependencies, you need the Python development headers. If
+you are on a Debian or Ubuntu system, you can install them with ``sudo
+apt-get install python-dev``. On RedHat/Fedora, install ``python-devel``.
+
+
+Windows
+-------
+
+Windows 7 or above is required.
+
+For 64-bit Windows:
+
+* Install Python 2.7 from
+ https://www.python.org/ftp/python/2.7/python-2.7.amd64.msi
+* Install pywin32 from
+ https://tahoe-lafs.org/source/tahoe-lafs/deps/windows/pywin32-219.win-amd64-py2.7.exe
+* Install git from
+ https://github.com/git-for-windows/git/releases/download/v2.6.2.windows.1/Git-2.6.2-64-bit.exe
+
+For 32-bit Windows:
+
+* Install Python 2.7 from
+ https://www.python.org/ftp/python/2.7/python-2.7.msi
+* Install pywin32 from
+ https://tahoe-lafs.org/source/tahoe-lafs/deps/windows/pywin32-219.win32-py2.7.exe
+* Install git from
+ https://github.com/git-for-windows/git/releases/download/v2.6.2.windows.1/Git-2.6.2-32-bit.exe
+
+Then (for any version) run these commands in a Command Prompt::
+
+ git clone -b 2438.magic-folder-stable.5 https://github.com/tahoe-lafs/tahoe-lafs.git
+ cd tahoe-lafs
+ python setup.py build
+
+Open a new Command Prompt with the same current directory,
+then run::
+
+ bin\tahoe --version-and-path
+
+It is normal for this command to print warnings and debugging output
+on some systems. ``python setup.py test`` can also be run, but there
+are some known sources of nondeterministic errors in tests on Windows
+that are unrelated to Magic Folder.
+
+
+Setting up a local test grid
+============================
+
+Linux
+-----
+
+Run these commands::
+
+ mkdir ../grid
+ bin/tahoe create-introducer ../grid/introducer
+ bin/tahoe start ../grid/introducer
+ export FURL=`cat ../grid/introducer/private/introducer.furl`
+ bin/tahoe create-node --introducer="$FURL" ../grid/server
+ bin/tahoe create-client --introducer="$FURL" ../grid/alice
+ bin/tahoe create-client --introducer="$FURL" ../grid/bob
+
+
+Windows
+-------
+
+Run::
+
+ mkdir ..\grid
+ bin\tahoe create-introducer ..\grid\introducer
+ bin\tahoe start ..\grid\introducer
+
+Leave the introducer running in that Command Prompt,
+and in a separate Command Prompt (with the same current
+directory), run::
+
+ set /p FURL=<..\grid\introducer\private\introducer.furl
+ bin\tahoe create-node --introducer=%FURL% ..\grid\server
+ bin\tahoe create-client --introducer=%FURL% ..\grid\alice
+ bin\tahoe create-client --introducer=%FURL% ..\grid\bob
+
+
+Both Linux and Windows
+----------------------
+
+(Replace ``/`` with ``\`` for Windows paths.)
+
+Edit ``../grid/alice/tahoe.cfg``, and make the following
+changes to the ``[node]`` and ``[client]`` sections::
+
+ [node]
+ nickname = alice
+ web.port = tcp:3457:interface=127.0.0.1
+
+ [client]
+ shares.needed = 1
+ shares.happy = 1
+ shares.total = 1
+
+Edit ``../grid/bob/tahoe.cfg``, and make the following
+change to the ``[node]`` section, and the same change as
+above to the ``[client]`` section::
+
+ [node]
+ nickname = bob
+ web.port = tcp:3458:interface=127.0.0.1
+
+Note that when running nodes on a single machine,
+unique port numbers must be used for each node (and they
+must not clash with ports used by other server software).
+Here we have used the default of 3456 for the server,
+3457 for alice, and 3458 for bob.
+
+Now start all of the nodes (the introducer should still be
+running from above)::
+
+ bin/tahoe start ../grid/server
+ bin/tahoe start ../grid/alice
+ bin/tahoe start ../grid/bob
+
+On Windows, a separate Command Prompt is needed to run each
+node.
+
+Open a web browser on http://127.0.0.1:3457/ and verify that
+alice is connected to the introducer and one storage server.
+Then do the same for http://127.0.0.1:3568/ to verify that
+bob is connected. Leave all of the nodes running for the
+next stage.
+
+
+Setting up Magic Folder
+=======================
+
+Linux
+-----
+
+Run::
+
+ mkdir -p ../local/alice ../local/bob
+ bin/tahoe -d ../grid/alice magic-folder create magic: alice ../local/alice
+ bin/tahoe -d ../grid/alice magic-folder invite magic: bob >invitecode
+ export INVITECODE=`cat invitecode`
+ bin/tahoe -d ../grid/bob magic-folder join "$INVITECODE" ../local/bob
+
+ bin/tahoe restart ../grid/alice
+ bin/tahoe restart ../grid/bob
+
+Windows
+-------
+
+Run::
+
+ mkdir ..\local\alice ..\local\bob
+ bin\tahoe -d ..\grid\alice magic-folder create magic: alice ..\local\alice
+ bin\tahoe -d ..\grid\alice magic-folder invite magic: bob >invitecode
+ set /p INVITECODE=<invitecode
+ bin\tahoe -d ..\grid\bob magic-folder join %INVITECODE% ..\local\bob
+
+Then close the Command Prompt windows that are running the alice and bob
+nodes, and open two new ones in which to run::
+
+ bin\tahoe start ..\grid\alice
+ bin\tahoe start ..\grid\bob
+
+
+Testing
+=======
+
+You can now experiment with creating files and directories in
+``../local/alice`` and ``/local/bob``; any changes should be
+propagated to the other directory.
+
+Note that when a file is deleted, the corresponding file in the
+other directory will be renamed to a filename ending in ``.backup``.
+Deleting a directory will have no effect.
+
+For other known issues and limitations, see
+https://github.com/tahoe-lafs/tahoe-lafs/blob/2438.magic-folder-stable.8/docs/frontends/magic-folder.rst#known-issues-and-limitations
+
+As mentioned earlier, it is also possible to run the nodes on
+different machines, to synchronize between three or more clients,
+to mix Windows and Linux clients, and to use multiple servers
+(if the Tahoe-LAFS encoding parameters are changed).
projects / tahoe-lafs / tahoe-lafs.git / commitdiff