From 567570a130b245f8423a0265d4247a99139fc152 Mon Sep 17 00:00:00 2001 From: Daira Hopwood Date: Mon, 28 Dec 2015 19:28:26 +0000 Subject: [PATCH] Documentation for Magic Folder. Signed-off-by: Daira Hopwood --- docs/configuration.rst | 12 +- docs/frontends/drop-upload.rst | 158 ---------------------- docs/frontends/magic-folder.rst | 149 ++++++++++++++++++++ docs/magic-folder-howto.rst | 231 ++++++++++++++++++++++++++++++++ 4 files changed, 386 insertions(+), 164 deletions(-) delete mode 100644 docs/frontends/drop-upload.rst create mode 100644 docs/frontends/magic-folder.rst create mode 100644 docs/magic-folder-howto.rst diff --git a/docs/configuration.rst b/docs/configuration.rst index f9179376..e632075e 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -68,7 +68,7 @@ Client/server nodes provide one or more of the following services: * web-API service * SFTP service * FTP service -* drop-upload service +* Magic Folder service * helper service * storage service. @@ -453,16 +453,16 @@ SFTP, FTP instructions on configuring these services, and the ``[sftpd]`` and ``[ftpd]`` sections of ``tahoe.cfg``. -Drop-Upload +Magic Folder - As of Tahoe-LAFS v1.9.0, a node running on Linux can be configured to - automatically upload files that are created or changed in a specified - local directory. See drop-upload.rst_ for details. + A node running on Linux or Windows can be configured to automatically + upload files that are created or changed in a specified local directory. + See `magic-folder.rst`_ for details. .. _download-status.rst: frontends/download-status.rst .. _CLI.rst: frontends/CLI.rst .. _FTP-and-SFTP.rst: frontends/FTP-and-SFTP.rst -.. _drop-upload.rst: frontends/drop-upload.rst +.. _magic-folder.rst: frontends/magic-folder.rst Storage Server Configuration diff --git a/docs/frontends/drop-upload.rst b/docs/frontends/drop-upload.rst deleted file mode 100644 index bc92609b..00000000 --- a/docs/frontends/drop-upload.rst +++ /dev/null @@ -1,158 +0,0 @@ -.. -*- coding: utf-8-with-signature -*- - -=============================== -Tahoe-LAFS Drop-Upload Frontend -=============================== - -1. `Introduction`_ -2. `Configuration`_ -3. `Known Issues and Limitations`_ - - -Introduction -============ - -The drop-upload frontend allows an upload to a Tahoe-LAFS grid to be triggered -automatically whenever a file is created or changed in a specific local -directory. This is a preview of a feature that we expect to support across -several platforms, but it currently works only on Linux. - -The implementation was written as a prototype at the First International -Tahoe-LAFS Summit in June 2011, and is not currently in as mature a state as -the other frontends (web, CLI, SFTP and FTP). This means that you probably -should not keep important data in the upload directory, and should not rely -on all changes to files in the local directory to result in successful uploads. -There might be (and have been) incompatible changes to how the feature is -configured. There is even the possibility that it may be abandoned, for -example if unsolveable reliability issues are found. - -We are very interested in feedback on how well this feature works for you, and -suggestions to improve its usability, functionality, and reliability. - - -Configuration -============= - -The drop-upload frontend runs as part of a gateway node. To set it up, you -need to choose the local directory to monitor for file changes, and a mutable -directory on the grid to which files will be uploaded. - -These settings are configured in the ``[drop_upload]`` section of the -gateway's ``tahoe.cfg`` file. - -``[drop_upload]`` - -``enabled = (boolean, optional)`` - - If this is ``True``, drop-upload will be enabled. The default value is - ``False``. - -``local.directory = (UTF-8 path)`` - - This specifies the local directory to be monitored for new or changed - files. If the path contains non-ASCII characters, it should be encoded - in UTF-8 regardless of the system's filesystem encoding. Relative paths - will be interpreted starting from the node's base directory. - -In addition, the file ``private/drop_upload_dircap`` must contain a -writecap pointing to an existing mutable directory to be used as the target -of uploads. It will start with ``URI:DIR2:``, and cannot include an alias -or path. - -After setting the above fields and starting or restarting the gateway, -you can confirm that the feature is working by copying a file into the -local directory. Then, use the WUI or CLI to check that it has appeared -in the upload directory with the same filename. A large file may take some -time to appear, since it is only linked into the directory after the upload -has completed. - -The 'Operational Statistics' page linked from the Welcome page shows -counts of the number of files uploaded, the number of change events currently -queued, and the number of failed uploads. The 'Recent Uploads and Downloads' -page and the node log_ may be helpful to determine the cause of any failures. - -.. _log: ../logging.rst - - -Known Issues and Limitations -============================ - -This frontend only works on Linux. There is an even-more-experimental -implementation for Windows (`#1431`_), and a ticket to add support for -Mac OS X and BSD-based systems (`#1432`_). - -Subdirectories of the local directory are not monitored. If a subdirectory -is created, it will be ignored. (`#1433`_) - -If files are created or changed in the local directory just after the gateway -has started, it might not have connected to a sufficient number of servers -when the upload is attempted, causing the upload to fail. (`#1449`_) - -Files that were created or changed in the local directory while the gateway -was not running, will not be uploaded. (`#1458`_) - -The only way to determine whether uploads have failed is to look at the -'Operational Statistics' page linked from the Welcome page. This only shows -a count of failures, not the names of files. Uploads are never retried. - -The drop-upload frontend performs its uploads sequentially (i.e. it waits -until each upload is finished before starting the next), even when there -would be enough memory and bandwidth to efficiently perform them in parallel. -A drop-upload can occur in parallel with an upload by a different frontend, -though. (`#1459`_) - -If there are a large number of near-simultaneous file creation or -change events (greater than the number specified in the file -``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events -could be missed. This is fairly unlikely under normal circumstances, because -the default value of ``max_queued_events`` in most Linux distributions is -16384, and events are removed from this queue immediately without waiting for -the corresponding upload to complete. (`#1430`_) - -Some filesystems may not support the necessary change notifications. -So, it is recommended for the local directory to be on a directly attached -disk-based filesystem, not a network filesystem or one provided by a virtual -machine. - -Attempts to read the mutable directory at about the same time as an uploaded -file is being linked into it, might fail, even if they are done through the -same gateway. (`#1105`_) - -When a local file is changed and closed several times in quick succession, -it may be uploaded more times than necessary to keep the remote copy -up-to-date. (`#1440`_) - -Files deleted from the local directory will not be unlinked from the upload -directory. (`#1710`_) - -The ``private/drop_upload_dircap`` file cannot use an alias or path to -specify the upload directory. (`#1711`_) - -Files are always uploaded as immutable. If there is an existing mutable file -of the same name in the upload directory, it will be unlinked and replaced -with an immutable file. (`#1712`_) - -If a file in the upload directory is changed (actually relinked to a new -file), then the old file is still present on the grid, and any other caps to -it will remain valid. See `docs/garbage-collection.rst`_ for how to reclaim -the space used by files that are no longer needed. - -Unicode names are supported, but the local name of a file must be encoded -correctly in order for it to be uploaded. The expected encoding is that -printed by ``python -c "import sys; print sys.getfilesystemencoding()"``. - -.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105 -.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430 -.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431 -.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432 -.. _`#1433`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1433 -.. _`#1440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1440 -.. _`#1449`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1449 -.. _`#1458`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1458 -.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459 -.. _`#1710`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1710 -.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711 -.. _`#1712`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1712 - -.. _docs/garbage-collection.rst: ../garbage-collection.rst - diff --git a/docs/frontends/magic-folder.rst b/docs/frontends/magic-folder.rst new file mode 100644 index 00000000..9642d139 --- /dev/null +++ b/docs/frontends/magic-folder.rst @@ -0,0 +1,149 @@ +.. -*- coding: utf-8-with-signature -*- + +================================ +Tahoe-LAFS Magic Folder Frontend +================================ + +1. `Introduction`_ +2. `Configuration`_ +3. `Known Issues and Limitations`_ + + +Introduction +============ + +The Magic Folder frontend synchronizes local directories on two or more +clients, using a Tahoe-LAFS grid for storage. Whenever a file is created +or changed under the local directory of one of the clients, the change is +propagated to the grid and then to the other clients. + +The implementation of the "drop-upload" frontend, on which Magic Folder is +based, was written as a prototype at the First International Tahoe-LAFS +Summit in June 2011. In 2015, with the support of a grant from the +`Open Technology Fund`_, it was redesigned and extended to support +synchronization between clients. It currently works on Linux and Windows. + +Magic Folder is not currently in as mature a state as the other frontends +(web, CLI, SFTP and FTP). This means that you probably should not rely on +all changes to files in the local directory to result in successful uploads. +There might be (and have been) incompatible changes to how the feature is +configured. + +We are very interested in feedback on how well this feature works for you, and +suggestions to improve its usability, functionality, and reliability. + +.. _`Open Technology Fund`: https://www.opentech.fund/ + + +Configuration +============= + +The Magic Folder frontend runs as part of a gateway node. To set it up, you +must use the tahoe magic-folder CLI. For detailed information see our +`Magic-Folder CLI design documentation`_. For a given Magic-Folder collective +directory you need to run the ``tahoe magic-folder create`` command. After +that the ``tahoe magic-folder invite`` command must used to generate an +*invite code* for each member of the magic-folder collective. A confidential, +authenticated communications channel should be used to transmit the invite code +to each member, who will be joining using the ``tahoe magic-folder join`` +command. + +These settings are persisted in the ``[magic_folder]`` section of the +gateway's ``tahoe.cfg`` file. + +``[magic_folder]`` + +``enabled = (boolean, optional)`` + + If this is ``True``, Magic Folder will be enabled. The default value is + ``False``. + +``local.directory = (UTF-8 path)`` + + This specifies the local directory to be monitored for new or changed + files. If the path contains non-ASCII characters, it should be encoded + in UTF-8 regardless of the system's filesystem encoding. Relative paths + will be interpreted starting from the node's base directory. + +You should not normally need to set these fields manually because they are +set by the ``tahoe magic-folder create`` and/or ``tahoe magic-folder join`` +commands. Use the ``--help`` option to these commands for more information. + +After setting up a Magic Folder collective and starting or restarting each +gateway, you can confirm that the feature is working by copying a file into +any local directory, and checking that it appears on other clients. +Large files may take some time to appear. + +The 'Operational Statistics' page linked from the Welcome page shows +counts of the number of files uploaded, the number of change events currently +queued, and the number of failed uploads. The 'Recent Uploads and Downloads' +page and the node log_ may be helpful to determine the cause of any failures. + +.. _log: ../logging.rst + + +Known Issues and Limitations +============================ + +This feature only works on Linux and Windows. There is a ticket to add +support for Mac OS X and BSD-based systems (`#1432`_). + +The only way to determine whether uploads have failed is to look at the +'Operational Statistics' page linked from the Welcome page. This only shows +a count of failures, not the names of files. Uploads are never retried. + +The Magic Folder frontend performs its uploads sequentially (i.e. it waits +until each upload is finished before starting the next), even when there +would be enough memory and bandwidth to efficiently perform them in parallel. +A Magic Folder upload can occur in parallel with an upload by a different +frontend, though. (`#1459`_) + +On Linux, if there are a large number of near-simultaneous file creation or +change events (greater than the number specified in the file +``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events +could be missed. This is fairly unlikely under normal circumstances, because +the default value of ``max_queued_events`` in most Linux distributions is +16384, and events are removed from this queue immediately without waiting for +the corresponding upload to complete. (`#1430`_) + +The Windows implementation might also occasionally miss file creation or +change events, due to limitations of the underlying Windows API +(ReadDirectoryChangesW). We do not know how likely or unlikely this is. +(`#1431`_) + +Some filesystems may not support the necessary change notifications. +So, it is recommended for the local directory to be on a directly attached +disk-based filesystem, not a network filesystem or one provided by a virtual +machine. + +The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files +cannot use an alias or path to specify the upload directory. (`#1711`_) + +If a file in the upload directory is changed (actually relinked to a new +file), then the old file is still present on the grid, and any other caps +to it will remain valid. Eventually it will be possible to use +`garbage collection`_ to reclaim the space used by these files; however +currently they are retained indefinitely. (`#2440`_) + +Unicode filenames are supported on both Linux and Windows, but on Linux, the +local name of a file must be encoded correctly in order for it to be uploaded. +The expected encoding is that printed by +``python -c "import sys; print sys.getfilesystemencoding()"``. + +On Windows, local directories with non-ASCII names are not currently working. +(`#2219`_) + +On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C +(it can only be killed using Task Manager or similar). (`#2218`_) + +.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430 +.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431 +.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432 +.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459 +.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711 +.. _`#2218`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2218 +.. _`#2219`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2219 +.. _`#2440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2440 + +.. _`garbage collection`: ../garbage-collection.rst +.. _`Magic-Folder CLI design documentation`: ../proposed/magic-folder/user-interface-design.rst diff --git a/docs/magic-folder-howto.rst b/docs/magic-folder-howto.rst new file mode 100644 index 00000000..0992ce22 --- /dev/null +++ b/docs/magic-folder-howto.rst @@ -0,0 +1,231 @@ +========================= +Magic Folder Set-up Howto +========================= + +1. `This document`_ +2. `Preparation`_ +3. `Setting up a local test grid`_ +4. `Setting up Magic Folder`_ +5. `Testing`_ + + +This document +============= + +This is preliminary documentation of how to set up the +Magic Folder pre-release using a test grid on a single Linux +or Windows machine, with two clients and one server. It is +aimed at a fairly technical audience. + +For an introduction to Magic Folder and how to configure it +more generally, see `docs/frontends/magic-folder.rst`_. + +It it possible to adapt these instructions to run the nodes on +different machines, to synchronize between three or more clients, +to mix Windows and Linux clients, and to use multiple servers +(if the Tahoe-LAFS encoding parameters are changed). + +.. _`docs/frontends/magic-folder.rst`: ../docs/frontends/magic-folder.rst + + +Preparation +=========== + +Linux +----- + +Install ``git`` from your distribution's package manager. +Then run these commands:: + + git clone -b 2438.magic-folder-stable.8 https://github.com/tahoe-lafs/tahoe-lafs.git + cd tahoe-lafs + python setup.py test + +The test suite usually takes about 15 minutes to run. +Note that it is normal for some tests to be skipped. +In the current branch, the Magic Folder tests produce +considerable debugging output. + +If you see an error like ``fatal error: Python.h: No such file or directory`` +while compiling the dependencies, you need the Python development headers. If +you are on a Debian or Ubuntu system, you can install them with ``sudo +apt-get install python-dev``. On RedHat/Fedora, install ``python-devel``. + + +Windows +------- + +Windows 7 or above is required. + +For 64-bit Windows: + +* Install Python 2.7 from + https://www.python.org/ftp/python/2.7/python-2.7.amd64.msi +* Install pywin32 from + https://tahoe-lafs.org/source/tahoe-lafs/deps/windows/pywin32-219.win-amd64-py2.7.exe +* Install git from + https://github.com/git-for-windows/git/releases/download/v2.6.2.windows.1/Git-2.6.2-64-bit.exe + +For 32-bit Windows: + +* Install Python 2.7 from + https://www.python.org/ftp/python/2.7/python-2.7.msi +* Install pywin32 from + https://tahoe-lafs.org/source/tahoe-lafs/deps/windows/pywin32-219.win32-py2.7.exe +* Install git from + https://github.com/git-for-windows/git/releases/download/v2.6.2.windows.1/Git-2.6.2-32-bit.exe + +Then (for any version) run these commands in a Command Prompt:: + + git clone -b 2438.magic-folder-stable.5 https://github.com/tahoe-lafs/tahoe-lafs.git + cd tahoe-lafs + python setup.py build + +Open a new Command Prompt with the same current directory, +then run:: + + bin\tahoe --version-and-path + +It is normal for this command to print warnings and debugging output +on some systems. ``python setup.py test`` can also be run, but there +are some known sources of nondeterministic errors in tests on Windows +that are unrelated to Magic Folder. + + +Setting up a local test grid +============================ + +Linux +----- + +Run these commands:: + + mkdir ../grid + bin/tahoe create-introducer ../grid/introducer + bin/tahoe start ../grid/introducer + export FURL=`cat ../grid/introducer/private/introducer.furl` + bin/tahoe create-node --introducer="$FURL" ../grid/server + bin/tahoe create-client --introducer="$FURL" ../grid/alice + bin/tahoe create-client --introducer="$FURL" ../grid/bob + + +Windows +------- + +Run:: + + mkdir ..\grid + bin\tahoe create-introducer ..\grid\introducer + bin\tahoe start ..\grid\introducer + +Leave the introducer running in that Command Prompt, +and in a separate Command Prompt (with the same current +directory), run:: + + set /p FURL=<..\grid\introducer\private\introducer.furl + bin\tahoe create-node --introducer=%FURL% ..\grid\server + bin\tahoe create-client --introducer=%FURL% ..\grid\alice + bin\tahoe create-client --introducer=%FURL% ..\grid\bob + + +Both Linux and Windows +---------------------- + +(Replace ``/`` with ``\`` for Windows paths.) + +Edit ``../grid/alice/tahoe.cfg``, and make the following +changes to the ``[node]`` and ``[client]`` sections:: + + [node] + nickname = alice + web.port = tcp:3457:interface=127.0.0.1 + + [client] + shares.needed = 1 + shares.happy = 1 + shares.total = 1 + +Edit ``../grid/bob/tahoe.cfg``, and make the following +change to the ``[node]`` section, and the same change as +above to the ``[client]`` section:: + + [node] + nickname = bob + web.port = tcp:3458:interface=127.0.0.1 + +Note that when running nodes on a single machine, +unique port numbers must be used for each node (and they +must not clash with ports used by other server software). +Here we have used the default of 3456 for the server, +3457 for alice, and 3458 for bob. + +Now start all of the nodes (the introducer should still be +running from above):: + + bin/tahoe start ../grid/server + bin/tahoe start ../grid/alice + bin/tahoe start ../grid/bob + +On Windows, a separate Command Prompt is needed to run each +node. + +Open a web browser on http://127.0.0.1:3457/ and verify that +alice is connected to the introducer and one storage server. +Then do the same for http://127.0.0.1:3568/ to verify that +bob is connected. Leave all of the nodes running for the +next stage. + + +Setting up Magic Folder +======================= + +Linux +----- + +Run:: + + mkdir -p ../local/alice ../local/bob + bin/tahoe -d ../grid/alice magic-folder create magic: alice ../local/alice + bin/tahoe -d ../grid/alice magic-folder invite magic: bob >invitecode + export INVITECODE=`cat invitecode` + bin/tahoe -d ../grid/bob magic-folder join "$INVITECODE" ../local/bob + + bin/tahoe restart ../grid/alice + bin/tahoe restart ../grid/bob + +Windows +------- + +Run:: + + mkdir ..\local\alice ..\local\bob + bin\tahoe -d ..\grid\alice magic-folder create magic: alice ..\local\alice + bin\tahoe -d ..\grid\alice magic-folder invite magic: bob >invitecode + set /p INVITECODE=