Drop-Upload
- As of Tahoe-LAFS v1.9.0, a node running on Linux can be configured to
- automatically upload files that are created or changed in a specified
- local directory. See drop-upload.rst_ for details.
+ A node running on Linux or Windows can be configured to automatically
+ upload files that are created or changed in a specified local directory.
+ See `drop-upload.rst`_ for details.
.. _download-status.rst: frontends/download-status.rst
.. _CLI.rst: frontends/CLI.rst
+++ /dev/null
-.. -*- coding: utf-8-with-signature -*-
-
-===============================
-Tahoe-LAFS Drop-Upload Frontend
-===============================
-
-1. `Introduction`_
-2. `Configuration`_
-3. `Known Issues and Limitations`_
-
-
-Introduction
-============
-
-The drop-upload frontend allows an upload to a Tahoe-LAFS grid to be triggered
-automatically whenever a file is created or changed in a specific local
-directory. This is a preview of a feature that we expect to support across
-several platforms, but it currently works only on Linux.
-
-The implementation was written as a prototype at the First International
-Tahoe-LAFS Summit in June 2011, and is not currently in as mature a state as
-the other frontends (web, CLI, SFTP and FTP). This means that you probably
-should not keep important data in the upload directory, and should not rely
-on all changes to files in the local directory to result in successful uploads.
-There might be (and have been) incompatible changes to how the feature is
-configured. There is even the possibility that it may be abandoned, for
-example if unsolveable reliability issues are found.
-
-We are very interested in feedback on how well this feature works for you, and
-suggestions to improve its usability, functionality, and reliability.
-
-
-Configuration
-=============
-
-The drop-upload frontend runs as part of a gateway node. To set it up, you
-need to choose the local directory to monitor for file changes, and a mutable
-directory on the grid to which files will be uploaded.
-
-These settings are configured in the ``[drop_upload]`` section of the
-gateway's ``tahoe.cfg`` file.
-
-``[drop_upload]``
-
-``enabled = (boolean, optional)``
-
- If this is ``True``, drop-upload will be enabled. The default value is
- ``False``.
-
-``local.directory = (UTF-8 path)``
-
- This specifies the local directory to be monitored for new or changed
- files. If the path contains non-ASCII characters, it should be encoded
- in UTF-8 regardless of the system's filesystem encoding. Relative paths
- will be interpreted starting from the node's base directory.
-
-In addition, the file ``private/drop_upload_dircap`` must contain a
-writecap pointing to an existing mutable directory to be used as the target
-of uploads. It will start with ``URI:DIR2:``, and cannot include an alias
-or path.
-
-After setting the above fields and starting or restarting the gateway,
-you can confirm that the feature is working by copying a file into the
-local directory. Then, use the WUI or CLI to check that it has appeared
-in the upload directory with the same filename. A large file may take some
-time to appear, since it is only linked into the directory after the upload
-has completed.
-
-The 'Operational Statistics' page linked from the Welcome page shows
-counts of the number of files uploaded, the number of change events currently
-queued, and the number of failed uploads. The 'Recent Uploads and Downloads'
-page and the node log_ may be helpful to determine the cause of any failures.
-
-.. _log: ../logging.rst
-
-
-Known Issues and Limitations
-============================
-
-This frontend only works on Linux. There is an even-more-experimental
-implementation for Windows (`#1431`_), and a ticket to add support for
-Mac OS X and BSD-based systems (`#1432`_).
-
-Subdirectories of the local directory are not monitored. If a subdirectory
-is created, it will be ignored. (`#1433`_)
-
-If files are created or changed in the local directory just after the gateway
-has started, it might not have connected to a sufficient number of servers
-when the upload is attempted, causing the upload to fail. (`#1449`_)
-
-Files that were created or changed in the local directory while the gateway
-was not running, will not be uploaded. (`#1458`_)
-
-The only way to determine whether uploads have failed is to look at the
-'Operational Statistics' page linked from the Welcome page. This only shows
-a count of failures, not the names of files. Uploads are never retried.
-
-The drop-upload frontend performs its uploads sequentially (i.e. it waits
-until each upload is finished before starting the next), even when there
-would be enough memory and bandwidth to efficiently perform them in parallel.
-A drop-upload can occur in parallel with an upload by a different frontend,
-though. (`#1459`_)
-
-If there are a large number of near-simultaneous file creation or
-change events (greater than the number specified in the file
-``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events
-could be missed. This is fairly unlikely under normal circumstances, because
-the default value of ``max_queued_events`` in most Linux distributions is
-16384, and events are removed from this queue immediately without waiting for
-the corresponding upload to complete. (`#1430`_)
-
-Some filesystems may not support the necessary change notifications.
-So, it is recommended for the local directory to be on a directly attached
-disk-based filesystem, not a network filesystem or one provided by a virtual
-machine.
-
-Attempts to read the mutable directory at about the same time as an uploaded
-file is being linked into it, might fail, even if they are done through the
-same gateway. (`#1105`_)
-
-When a local file is changed and closed several times in quick succession,
-it may be uploaded more times than necessary to keep the remote copy
-up-to-date. (`#1440`_)
-
-Files deleted from the local directory will not be unlinked from the upload
-directory. (`#1710`_)
-
-The ``private/drop_upload_dircap`` file cannot use an alias or path to
-specify the upload directory. (`#1711`_)
-
-Files are always uploaded as immutable. If there is an existing mutable file
-of the same name in the upload directory, it will be unlinked and replaced
-with an immutable file. (`#1712`_)
-
-If a file in the upload directory is changed (actually relinked to a new
-file), then the old file is still present on the grid, and any other caps to
-it will remain valid. See `docs/garbage-collection.rst`_ for how to reclaim
-the space used by files that are no longer needed.
-
-Unicode names are supported, but the local name of a file must be encoded
-correctly in order for it to be uploaded. The expected encoding is that
-printed by ``python -c "import sys; print sys.getfilesystemencoding()"``.
-
-.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105
-.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430
-.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431
-.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432
-.. _`#1433`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1433
-.. _`#1440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1440
-.. _`#1449`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1449
-.. _`#1458`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1458
-.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459
-.. _`#1710`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1710
-.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711
-.. _`#1712`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1712
-
-.. _docs/garbage-collection.rst: ../garbage-collection.rst
-
--- /dev/null
+.. -*- coding: utf-8-with-signature -*-
+
+================================
+Tahoe-LAFS Magic Folder Frontend
+================================
+
+1. `Introduction`_
+2. `Configuration`_
+3. `Known Issues and Limitations`_
+
+
+Introduction
+============
+
+The Magic Folder frontend synchronizes local directories on two or more
+clients, using a Tahoe-LAFS grid for storage. Whenever a file is created
+or changed under the local directory of one of the clients, the change is
+propagated to the grid and then to the other clients.
+
+The implementation of the "drop-upload" frontend, on which Magic Folder is
+based, was written as a prototype at the First International Tahoe-LAFS
+Summit in June 2011. In 2015, with the support of a grant from the
+`Open Technology Fund`_, it was redesigned and extended to support
+synchronization between clients. It currently works on Linux and Windows.
+
+Magic Folder is not currently in as mature a state as the other frontends
+(web, CLI, SFTP and FTP). This means that you probably should not rely on
+all changes to files in the local directory to result in successful uploads.
+There might be (and have been) incompatible changes to how the feature is
+configured.
+
+We are very interested in feedback on how well this feature works for you, and
+suggestions to improve its usability, functionality, and reliability.
+
+.. _`Open Technology Fund`: https://www.opentech.fund/
+
+
+Configuration
+=============
+
+The Magic Folder frontend runs as part of a gateway node. To set it up, you
+must use the tahoe magic-folder CLI. For detailed information see our
+`Magic-Folder CLI design documentation`_. For a given Magic-Folder collective
+directory you need to run the ``tahoe magic-folder create`` command. After
+that the ``tahoe magic-folder invite`` command must used to generate an
+*invite code* for each member of the magic-folder collective. A confidential,
+authenticated communications channel should be used to transmit the invite code
+to each member, who will be joining using the ``tahoe magic-folder join``
+command.
+
+These settings are persisted in the ``[magic_folder]`` section of the
+gateway's ``tahoe.cfg`` file.
+
+``[magic_folder]``
+
+``enabled = (boolean, optional)``
+
+ If this is ``True``, Magic Folder will be enabled. The default value is
+ ``False``.
+
+``local.directory = (UTF-8 path)``
+
+ This specifies the local directory to be monitored for new or changed
+ files. If the path contains non-ASCII characters, it should be encoded
+ in UTF-8 regardless of the system's filesystem encoding. Relative paths
+ will be interpreted starting from the node's base directory.
+
+You should not normally need to set these fields manually because they are
+set by the ``tahoe magic-folder create`` and/or ``tahoe magic-folder join``
+commands. Use the ``--help`` option to these commands for more information.
+
+After setting up a Magic Folder collective and starting or restarting each
+gateway, you can confirm that the feature is working by copying a file into
+any local directory, and checking that it appears on other clients.
+Large files may take some time to appear.
+
+The 'Operational Statistics' page linked from the Welcome page shows
+counts of the number of files uploaded, the number of change events currently
+queued, and the number of failed uploads. The 'Recent Uploads and Downloads'
+page and the node log_ may be helpful to determine the cause of any failures.
+
+.. _log: ../logging.rst
+
+
+Known Issues and Limitations
+============================
+
+This feature only works on Linux and Windows. There is a ticket to add
+support for Mac OS X and BSD-based systems (`#1432`_).
+
+The only way to determine whether uploads have failed is to look at the
+'Operational Statistics' page linked from the Welcome page. This only shows
+a count of failures, not the names of files. Uploads are never retried.
+
+The Magic Folder frontend performs its uploads sequentially (i.e. it waits
+until each upload is finished before starting the next), even when there
+would be enough memory and bandwidth to efficiently perform them in parallel.
+A Magic Folder upload can occur in parallel with an upload by a different
+frontend, though. (`#1459`_)
+
+On Linux, if there are a large number of near-simultaneous file creation or
+change events (greater than the number specified in the file
+``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events
+could be missed. This is fairly unlikely under normal circumstances, because
+the default value of ``max_queued_events`` in most Linux distributions is
+16384, and events are removed from this queue immediately without waiting for
+the corresponding upload to complete. (`#1430`_)
+
+The Windows implementation might also occasionally miss file creation or
+change events, due to limitations of the underlying Windows API
+(ReadDirectoryChangesW). We do not know how likely or unlikely this is.
+(`#1431`_)
+
+Some filesystems may not support the necessary change notifications.
+So, it is recommended for the local directory to be on a directly attached
+disk-based filesystem, not a network filesystem or one provided by a virtual
+machine.
+
+The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files
+cannot use an alias or path to specify the upload directory. (`#1711`_)
+
+If a file in the upload directory is changed (actually relinked to a new
+file), then the old file is still present on the grid, and any other caps
+to it will remain valid. Eventually it will be possible to use
+`garbage collection`_ to reclaim the space used by these files; however
+currently they are retained indefinitely. (`#2440`_)
+
+Unicode filenames are supported on both Linux and Windows, but on Linux, the
+local name of a file must be encoded correctly in order for it to be uploaded.
+The expected encoding is that printed by
+``python -c "import sys; print sys.getfilesystemencoding()"``.
+
+On Windows, local directories with non-ASCII names are not currently working.
+(`#2219`_)
+
+On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C
+(it can only be killed using Task Manager or similar). (`#2218`_)
+
+.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430
+.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431
+.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432
+.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459
+.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711
+.. _`#2218`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2218
+.. _`#2219`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2219
+.. _`#2440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2440
+
+.. _`garbage collection`: ../garbage-collection.rst
+.. _`Magic-Folder CLI design documentation`: ../proposed/magic-folder/user-interface-design.rst
--- /dev/null
+=========================
+Magic Folder Set-up Howto
+=========================
+
+1. `This document`_
+2. `Preparation`_
+3. `Setting up a local test grid`_
+4. `Setting up Magic Folder`_
+5. `Testing`_
+
+
+This document
+=============
+
+This is preliminary documentation of how to set up the
+Magic Folder pre-release using a test grid on a single Linux
+or Windows machine, with two clients and one server. It is
+aimed at a fairly technical audience.
+
+For an introduction to Magic Folder and how to configure it
+more generally, see `docs/frontends/magic-folder.rst`_.
+
+It it possible to adapt these instructions to run the nodes on
+different machines, to synchronize between three or more clients,
+to mix Windows and Linux clients, and to use multiple servers
+(if the Tahoe-LAFS encoding parameters are changed).
+
+.. _`docs/frontends/magic-folder.rst`: ../docs/frontends/magic-folder.rst
+
+
+Preparation
+===========
+
+Linux
+-----
+
+Install ``git`` from your distribution's package manager.
+Then run these commands::
+
+ git clone -b 2438.magic-folder-stable.5 https://github.com/tahoe-lafs/tahoe-lafs.git
+ cd tahoe-lafs
+ python setup.py test
+
+The test suite usually takes about 15 minutes to run.
+Note that it is normal for some tests to be skipped.
+In the current branch, the Magic Folder tests produce
+considerable debugging output.
+
+If you see an error like ``fatal error: Python.h: No such file or directory``
+while compiling the dependencies, you need the Python development headers. If
+you are on a Debian or Ubuntu system, you can install them with ``sudo
+apt-get install python-dev``. On RedHat/Fedora, install ``python-devel``.
+
+
+Windows
+-------
+
+Windows 7 or above is required.
+
+For 64-bit Windows:
+
+* Install Python 2.7 from
+ https://www.python.org/ftp/python/2.7/python-2.7.amd64.msi
+* Install pywin32 from
+ https://tahoe-lafs.org/source/tahoe-lafs/deps/windows/pywin32-219.win-amd64-py2.7.exe
+* Install git from
+ https://github.com/git-for-windows/git/releases/download/v2.6.2.windows.1/Git-2.6.2-64-bit.exe
+
+For 32-bit Windows:
+
+* Install Python 2.7 from
+ https://www.python.org/ftp/python/2.7/python-2.7.msi
+* Install pywin32 from
+ https://tahoe-lafs.org/source/tahoe-lafs/deps/windows/pywin32-219.win32-py2.7.exe
+* Install git from
+ https://github.com/git-for-windows/git/releases/download/v2.6.2.windows.1/Git-2.6.2-32-bit.exe
+
+Then (for any version) run these commands in a Command Prompt::
+
+ git clone -b 2438.magic-folder-stable.5 https://github.com/tahoe-lafs/tahoe-lafs.git
+ cd tahoe-lafs
+ python setup.py build
+
+Open a new Command Prompt with the same current directory,
+then run::
+
+ bin\tahoe --version-and-path
+
+It is normal for this command to print warnings and debugging output
+on some systems. ``python setup.py test`` can also be run, but there
+are some known sources of nondeterministic errors in tests on Windows
+that are unrelated to Magic Folder.
+
+
+Setting up a local test grid
+============================
+
+Linux
+-----
+
+Run these commands::
+
+ mkdir ../grid
+ bin/tahoe create-introducer ../grid/introducer
+ bin/tahoe start ../grid/introducer
+ export FURL=`cat ../grid/introducer/private/introducer.furl`
+ bin/tahoe create-node --introducer="$FURL" ../grid/server
+ bin/tahoe create-client --introducer="$FURL" ../grid/alice
+ bin/tahoe create-client --introducer="$FURL" ../grid/bob
+
+
+Windows
+-------
+
+Run::
+
+ mkdir ..\grid
+ bin\tahoe create-introducer ..\grid\introducer
+ bin\tahoe start ..\grid\introducer
+
+Leave the introducer running in that Command Prompt,
+and in a separate Command Prompt (with the same current
+directory), run::
+
+ set /p FURL=<..\grid\introducer\private\introducer.furl
+ bin\tahoe create-node --introducer=%FURL% ..\grid\server
+ bin\tahoe create-client --introducer=%FURL% ..\grid\alice
+ bin\tahoe create-client --introducer=%FURL% ..\grid\bob
+
+
+Both Linux and Windows
+----------------------
+
+(Replace ``/`` with ``\`` for Windows paths.)
+
+Edit ``../grid/alice/tahoe.cfg``, and make the following
+changes to the ``[node]`` and ``[client]`` sections::
+
+ [node]
+ nickname = alice
+ web.port = tcp:3457:interface=127.0.0.1
+
+ [client]
+ shares.needed = 1
+ shares.happy = 1
+ shares.total = 1
+
+Edit ``../grid/bob/tahoe.cfg``, and make the following
+change to the ``[node]`` section, and the same change as
+above to the ``[client]`` section::
+
+ [node]
+ nickname = bob
+ web.port = tcp:3458:interface=127.0.0.1
+
+Note that when running nodes on a single machine,
+unique port numbers must be used for each node (and they
+must not clash with ports used by other server software).
+Here we have used the default of 3456 for the server,
+3457 for alice, and 3458 for bob.
+
+Now start all of the nodes (the introducer should still be
+running from above)::
+
+ bin/tahoe start ../grid/server
+ bin/tahoe start ../grid/alice
+ bin/tahoe start ../grid/bob
+
+On Windows, a separate Command Prompt is needed to run each
+node.
+
+Open a web browser on http://127.0.0.1:3457/ and verify that
+alice is connected to the introducer and one storage server.
+Then do the same for http://127.0.0.1:3568/ to verify that
+bob is connected. Leave all of the nodes running for the
+next stage.
+
+
+Setting up Magic Folder
+=======================
+
+Linux
+-----
+
+Run::
+
+ mkdir -p ../local/alice ../local/bob
+ bin/tahoe -d ../grid/alice magic-folder create magic: alice ../local/alice
+ bin/tahoe -d ../grid/alice magic-folder invite magic: bob >invitecode
+ export INVITECODE=`cat invitecode`
+ bin/tahoe -d ../grid/bob magic-folder join "$INVITECODE" ../local/bob
+
+ bin/tahoe restart ../grid/alice
+ bin/tahoe restart ../grid/bob
+
+Windows
+-------
+
+Run::
+
+ mkdir ..\local\alice ..\local\bob
+ bin\tahoe -d ..\grid\alice magic-folder create magic: alice ..\local\alice
+ bin\tahoe -d ..\grid\alice magic-folder invite magic: bob >invitecode
+ set /p INVITECODE=<invitecode
+ bin\tahoe -d ..\grid\bob magic-folder join %INVITECODE% ..\local\bob
+
+Then close the Command Prompt windows that are running the alice and bob
+nodes, and open two new ones in which to run::
+
+ bin\tahoe start ..\grid\alice
+ bin\tahoe start ..\grid\bob
+
+
+Testing
+=======
+
+You can now experiment with creating files and directories in
+``../local/alice`` and ``/local/bob``; any changes should be
+propagated to the other directory.
+
+Note that when a file is deleted, the corresponding file in the
+other directory will be renamed to a filename ending in ``.backup``.
+Deleting a directory will have no effect.
+
+For other known issues and limitations, see
+https://github.com/tahoe-lafs/tahoe-lafs/blob/2438.magic-folder-stable.5/docs/frontends/magic-folder.rst#known-issues-and-limitations
+
+As mentioned earlier, it is also possible to run the nodes on
+different machines, to synchronize between three or more clients,
+to mix Windows and Linux clients, and to use multiple servers
+(if the Tahoe-LAFS encoding parameters are changed).
single DMD can easily handle the number of files expected to be written
by a client, so this is unlikely to be a significant issue.
-123‒ ‒ ‒: In these designs, the set of files in a Magic Folder is
+123‒ ‒: In these designs, the set of files in a Magic Folder is
represented as the union of the files in all client DMDs. However,
when a file is modified by more than one client, it will be linked
from multiple client DMDs. We therefore need a mechanism, such as a
+------------------------------------------------+------+------+------+------+------+------+
| Can result in large DMDs |‒ | | | | | |
+------------------------------------------------+------+------+------+------+------+------+
-| Need version number to determine priority |‒ |‒ |‒ | | | |
+| Need version number to determine priority |‒ ‒ |‒ ‒ |‒ ‒ | | | |
+------------------------------------------------+------+------+------+------+------+------+
| Must traverse immutable directory structure | | |‒ ‒ | |‒ ‒ | |
+------------------------------------------------+------+------+------+------+------+------+
.. _`Fire Dragons`: #fire-dragons-distinguishing-conflicts-from-overwrites
+Note that writing a file that does not already have an entry in
+the `magic folder db`_ is initially classed as an overwrite.
+
A *write/download collision* occurs when another program writes
to ``foo`` in the local filesystem, concurrently with the new
version being written by the Magic Folder client. We need to
The implementation of file replacement differs between Unix
and Windows. On Unix, it can be implemented as follows:
-* 4a. Set the permissions of the replacement file to be the
- same as the replaced file, bitwise-or'd with octal 600
- (``rw-------``).
+* 4a. Stat the replaced path, and set the permissions of the
+ replacement file to be the same as the replaced file,
+ bitwise-or'd with octal 600 (``rw-------``). If the replaced
+ file does not exist, set the permissions according to the
+ user's umask. If there is a directory at the replaced path,
+ fail.
* 4b. Attempt to move the replaced file (``foo``) to the
- backup filename (``foo.backup``).
+ backup filename (``foo.backup``). If an ``ENOENT`` error
+ occurs because the replaced file does not exist, ignore this
+ error and continue with steps 4c and 4d.
* 4c. Attempt to create a hard link at the replaced filename
(``foo``) pointing to the replacement file (``.foo.tmp``).
* 4d. Attempt to unlink the replacement file (``.foo.tmp``),
Note that, if there is no conflict, the entry for ``foo``
recorded in the `magic folder db`_ will reflect the ``mtime``
-set in step 3. The link operation in step 4c will cause an
-``IN_CREATE`` event for ``foo``, but this will not trigger an
-upload, because the metadata recorded in the database entry
-will exactly match the metadata for the file's inode on disk.
-(The two hard links — ``foo`` and, while it still exists,
-``.foo.tmp`` — share the same inode and therefore the same
-metadata.)
+set in step 3. The move operation in step 4b will cause a
+``MOVED_FROM`` event for ``foo``, and the link operation in
+step 4c will cause an ``IN_CREATE`` event for ``foo``.
+However, these events will not trigger an upload, because they
+are guaranteed to be processed only after the file replacement
+has finished, at which point the metadata recorded in the
+database entry will exactly match the metadata for the file's
+inode on disk. (The two hard links — ``foo`` and, while it
+still exists, ``.foo.tmp`` — share the same inode and
+therefore the same metadata.)
.. _`magic folder db`: filesystem_integration.rst#local-scanning-and-database
-On Windows, file replacement can be implemented as a single
-call to the `ReplaceFileW`_ API (with the
-``REPLACEFILE_IGNORE_MERGE_ERRORS`` flag).
+On Windows, file replacement can be implemented by a call to
+the `ReplaceFileW`_ API (with the
+``REPLACEFILE_IGNORE_MERGE_ERRORS`` flag). If an error occurs
+because the replaced file does not exist, then we ignore this
+error and attempt to move the replacement file to the replaced
+file.
Similar to the Unix case, the `ReplaceFileW`_ operation will
-cause a change notification for ``foo``. The replaced ``foo``
-has the same ``mtime`` as the replacement file, and so this
-notification will not trigger an unwanted upload.
+cause one or more change notifications for ``foo``. The replaced
+``foo`` has the same ``mtime`` as the replacement file, and so any
+such notification(s) will not trigger an unwanted upload.
.. _`ReplaceFileW`: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365512%28v=vs.85%29.aspx
(Note that atomic operations on a directory are totally ordered.)
The set of possible interleavings differs between Windows and Unix.
-On Unix, we have:
+On Unix, for the case where the replaced file already exists, we have:
* Interleaving A: the other process' rename precedes our rename in
step 4b, and we get an ``IN_MOVED_TO`` event for its rename by
Therefore, an upload will be triggered for ``foo`` after its
change, which is correct and avoids data loss.
+If the replaced file did not already exist, an ``ENOENT`` error
+occurs at step 4b, and we continue with steps 4c and 4d. The other
+process' rename races with our link operation in step 4c. If the
+other process wins the race then the effect is similar to
+Interleaving C, and if we win the race this it is similar to
+Interleaving D. Either case avoids data loss.
+
+
On Windows, the internal implementation of `ReplaceFileW`_ is similar
to what we have described above for Unix; it works like this:
completed, the `ReplaceFileW`_ call will fail with return code
``ERROR_UNABLE_TO_MOVE_REPLACEMENT_2``. However, it is still
preferable to use this API over two `MoveFileExW`_ calls, because
-it retains the attributes and ACLs of ``foo`` where possible.)
+it retains the attributes and ACLs of ``foo`` where possible.
+Also note that if the `ReplaceFileW`_ call fails with
+``ERROR_FILE_NOT_FOUND`` because the replaced file does not exist,
+then the replacment operation ignores this error and continues with
+the equivalent of step 4c′, as on Unix.)
However, on Windows the other application will not be able to
directly rename ``foo.other`` onto ``foo`` (which would fail because
deleted. This complicates the interleaving analysis, because we
have two operations done by the other process interleaving with
three done by the magic folder process (rather than one operation
-interleaving with four as on Unix). The cases are:
+interleaving with four as on Unix).
+
+So on Windows, for the case where the replaced file already exists,
+we have:
* Interleaving A′: the other process' deletion of ``foo`` and its
rename of ``foo.other`` to ``foo`` both precede our rename in
our rename of ``foo`` to ``foo.backup`` done by `ReplaceFileW`_,
but its rename of ``foo.other`` to ``foo`` does not, so we get
an ``ERROR_FILE_NOT_FOUND`` error from `ReplaceFileW`_ indicating
- that the replaced file does not exist. Then we reclassify as a
- conflict; the other process' changes end up at ``foo`` (after
- it has renamed ``foo.other`` to ``foo``) and our changes end up
- at ``foo.conflicted``. This avoids data loss.
+ that the replaced file does not exist. We ignore this error and
+ attempt to move ``foo.tmp`` to ``foo``, racing with the other
+ process which is attempting to move ``foo.other`` to ``foo``.
+ If we win the race, then our changes end up at ``foo``, and the
+ other process' move fails. If the other process wins the race,
+ then its changes end up at ``foo``, our move fails, and we
+ reclassify as a conflict, so that our changes end up at
+ ``foo.conflicted``. Either possibility avoids data loss.
* Interleaving D′: the other process' deletion and/or rename happen
during the call to `ReplaceFileW`_, causing the latter to fail.
.. _`MoveFileExW`: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx
+If the replaced file did not already exist, we get an
+``ERROR_FILE_NOT_FOUND`` error from `ReplaceFileW`_, and attempt to
+move ``foo.tmp`` to ``foo``. This is similar to Interleaving C, and
+either possibility for the resulting race avoids data loss.
+
We also need to consider what happens if another process opens ``foo``
and writes to it directly, rather than renaming another file onto it:
For example, suppose that Alice's Magic Folder client sees a change
to ``foo`` in Bob's DMD. If the version it downloads from Bob's DMD
is "based on" the version currently in Alice's local filesystem at
-the time Alice's client attempts to write the downloaded file, then
-it is an overwrite. Otherwise it is initially classified as a
+the time Alice's client attempts to write the downloaded file ‒or if
+there is no existing version in Alice's local filesystem at that time‒
+then it is an overwrite. Otherwise it is initially classified as a
conflict.
This initial classification is used by the procedure for writing a
* the *last-uploaded statinfo*, if any (this is the size in
bytes, ``mtime``, and ``ctime`` stored in the ``local_files``
table when the file was last uploaded);
- * the ``filecap`` field of the ``caps`` table for this file,
- which is the URI under which the file was last uploaded.
- Call this ``last_uploaded_uri``.
+ * the ``last_uploaded_uri`` field of the ``local_files`` table
+ for this file, which is the URI under which the file was last
+ uploaded.
* 2d. If any of the following are true, then classify as a conflict:
Note that the entry for this zero-length file has a version number as
usual, and later versions may restore the file.
+When the downloader deletes a file (or renames it to a filename
+ending in ``.backup``) in response to a remote change, a local
+filesystem notification will occur, and we must make sure that this
+is not treated as a local change. To do this we have the downloader
+set the ``size`` field in the magic folder db to ``None`` (SQL NULL)
+just before deleting the file, and suppress notifications for which
+the local file does not exist, and the recorded ``size`` field is
+``None``.
+
When a Magic Folder client restarts, we can detect files that had
been downloaded but were deleted while it was not running, because
their paths will have last-downloaded records in the magic folder db
-without any corresponding local file.
+with a ``size`` other than ``None``, and without any corresponding
+local file.
Deletion of a directory
~~~~~~~~~~~~~~~~~~~~~~~
projects / tahoe-lafs / tahoe-lafs.git / commitdiff