Rename drop-upload to Magic Folder. fixes ticket:2405
authorDaira Hopwood <daira@jacaranda.org>
Mon, 20 Jul 2015 23:42:15 +0000 (00:42 +0100)
committerDaira Hopwood <daira@jacaranda.org>
Thu, 1 Oct 2015 00:19:43 +0000 (01:19 +0100)
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
docs/frontends/drop-upload.rst [deleted file]
docs/frontends/magic-folder.rst [new file with mode: 0644]
src/allmydata/client.py
src/allmydata/frontends/drop_upload.py [deleted file]
src/allmydata/frontends/magic_folder.py [new file with mode: 0644]
src/allmydata/test/test_client.py
src/allmydata/test/test_drop_upload.py [deleted file]
src/allmydata/test/test_magic_folder.py [new file with mode: 0644]

diff --git a/docs/frontends/drop-upload.rst b/docs/frontends/drop-upload.rst
deleted file mode 100644 (file)
index b6fcd92..0000000
+++ /dev/null
@@ -1,168 +0,0 @@
-.. -*- coding: utf-8-with-signature -*-
-
-===============================
-Tahoe-LAFS Drop-Upload Frontend
-===============================
-
-1.  `Introduction`_
-2.  `Configuration`_
-3.  `Known Issues and Limitations`_
-
-
-Introduction
-============
-
-The drop-upload frontend allows an upload to a Tahoe-LAFS grid to be triggered
-automatically whenever a file is created or changed in a specific local
-directory. It currently works on Linux and Windows.
-
-The implementation was written as a prototype at the First International
-Tahoe-LAFS Summit in June 2011, and is not currently in as mature a state as
-the other frontends (web, CLI, SFTP and FTP). This means that you probably
-should not rely on all changes to files in the local directory to result in
-successful uploads. There might be (and have been) incompatible changes to
-how the feature is configured.
-
-We are very interested in feedback on how well this feature works for you, and
-suggestions to improve its usability, functionality, and reliability.
-
-
-Configuration
-=============
-
-The drop-upload frontend runs as part of a gateway node. To set it up, you
-need to choose the local directory to monitor for file changes, and a mutable
-directory on the grid to which files will be uploaded.
-
-These settings are configured in the ``[drop_upload]`` section of the
-gateway's ``tahoe.cfg`` file.
-
-``[drop_upload]``
-
-``enabled = (boolean, optional)``
-
-    If this is ``True``, drop-upload will be enabled. The default value is
-    ``False``.
-
-``local.directory = (UTF-8 path)``
-
-    This specifies the local directory to be monitored for new or changed
-    files. If the path contains non-ASCII characters, it should be encoded
-    in UTF-8 regardless of the system's filesystem encoding. Relative paths
-    will be interpreted starting from the node's base directory.
-
-In addition, the file  ``private/drop_upload_dircap`` must contain a
-writecap pointing to an existing mutable directory to be used as the target
-of uploads. It will start with ``URI:DIR2:``, and cannot include an alias
-or path.
-
-After setting the above fields and starting or restarting the gateway,
-you can confirm that the feature is working by copying a file into the
-local directory. Then, use the WUI or CLI to check that it has appeared
-in the upload directory with the same filename. A large file may take some
-time to appear, since it is only linked into the directory after the upload
-has completed.
-
-The 'Operational Statistics' page linked from the Welcome page shows
-counts of the number of files uploaded, the number of change events currently
-queued, and the number of failed uploads. The 'Recent Uploads and Downloads'
-page and the node log_ may be helpful to determine the cause of any failures.
-
-.. _log: ../logging.rst
-
-
-Known Issues and Limitations
-============================
-
-This frontend only works on Linux and Windows. There is a ticket to add
-support for Mac OS X and BSD-based systems (`#1432`_).
-
-Subdirectories of the local directory are not monitored. If a subdirectory
-is created, it will be ignored. (`#1433`_)
-
-If files are created or changed in the local directory just after the gateway
-has started, it might not have connected to a sufficient number of servers
-when the upload is attempted, causing the upload to fail. (`#1449`_)
-
-Files that were created or changed in the local directory while the gateway
-was not running, will not be uploaded. (`#1458`_)
-
-The only way to determine whether uploads have failed is to look at the
-'Operational Statistics' page linked from the Welcome page. This only shows
-a count of failures, not the names of files. Uploads are never retried.
-
-The drop-upload frontend performs its uploads sequentially (i.e. it waits
-until each upload is finished before starting the next), even when there
-would be enough memory and bandwidth to efficiently perform them in parallel.
-A drop-upload can occur in parallel with an upload by a different frontend,
-though. (`#1459`_)
-
-On Linux, if there are a large number of near-simultaneous file creation or
-change events (greater than the number specified in the file
-``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events
-could be missed. This is fairly unlikely under normal circumstances, because
-the default value of ``max_queued_events`` in most Linux distributions is
-16384, and events are removed from this queue immediately without waiting for
-the corresponding upload to complete. (`#1430`_)
-
-The Windows implementation might also occasionally miss file creation or
-change events, due to limitations of the underlying Windows API
-(ReadDirectoryChangesW). We do not know how likely or unlikely this is.
-(`#1431`_)
-
-Some filesystems may not support the necessary change notifications.
-So, it is recommended for the local directory to be on a directly attached
-disk-based filesystem, not a network filesystem or one provided by a virtual
-machine.
-
-Attempts to read the mutable directory at about the same time as an uploaded
-file is being linked into it, might fail, even if they are done through the
-same gateway. (`#1105`_)
-
-When a local file is changed and closed several times in quick succession,
-it may be uploaded more times than necessary to keep the remote copy
-up-to-date. (`#1440`_)
-
-Files deleted from the local directory will not be unlinked from the upload
-directory. (`#1710`_)
-
-The ``private/drop_upload_dircap`` file cannot use an alias or path to
-specify the upload directory. (`#1711`_)
-
-Files are always uploaded as immutable. If there is an existing mutable file
-of the same name in the upload directory, it will be unlinked and replaced
-with an immutable file. (`#1712`_)
-
-If a file in the upload directory is changed (actually relinked to a new
-file), then the old file is still present on the grid, and any other caps to
-it will remain valid. See `docs/garbage-collection.rst`_ for how to reclaim
-the space used by files that are no longer needed.
-
-Unicode filenames are supported on both Linux and Windows, but on Linux, the
-local name of a file must be encoded correctly in order for it to be uploaded.
-The expected encoding is that printed by
-``python -c "import sys; print sys.getfilesystemencoding()"``.
-
-On Windows, local directories with non-ASCII names are not currently working.
-(`#2219`_)
-
-On Windows, when a node has drop-upload enabled, it is unresponsive to Ctrl-C
-(it can only be killed using Task Manager or similar). (`#2218`_)
-
-.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105
-.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430
-.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431
-.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432
-.. _`#1433`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1433
-.. _`#1440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1440
-.. _`#1449`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1449
-.. _`#1458`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1458
-.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459
-.. _`#1710`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1710
-.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711
-.. _`#1712`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1712
-.. _`#2218`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2218
-.. _`#2219`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2219
-
-.. _docs/garbage-collection.rst: ../garbage-collection.rst
-
diff --git a/docs/frontends/magic-folder.rst b/docs/frontends/magic-folder.rst
new file mode 100644 (file)
index 0000000..7835c99
--- /dev/null
@@ -0,0 +1,169 @@
+.. -*- coding: utf-8-with-signature -*-
+
+================================
+Tahoe-LAFS Magic Folder Frontend
+================================
+
+1.  `Introduction`_
+2.  `Configuration`_
+3.  `Known Issues and Limitations`_
+
+
+Introduction
+============
+
+The Magic Folder frontend allows an upload to a Tahoe-LAFS grid to be triggered
+automatically whenever a file is created or changed in a specific local
+directory. It currently works on Linux and Windows.
+
+The implementation was written as a prototype at the First International
+Tahoe-LAFS Summit in June 2011, and is not currently in as mature a state as
+the other frontends (web, CLI, SFTP and FTP). This means that you probably
+should not rely on all changes to files in the local directory to result in
+successful uploads. There might be (and have been) incompatible changes to
+how the feature is configured.
+
+We are very interested in feedback on how well this feature works for you, and
+suggestions to improve its usability, functionality, and reliability.
+
+
+Configuration
+=============
+
+The Magic Folder frontend runs as part of a gateway node. To set it up, you
+need to choose the local directory to monitor for file changes, and a mutable
+directory on the grid to which files will be uploaded.
+
+These settings are configured in the ``[magic_folder]`` section of the
+gateway's ``tahoe.cfg`` file.
+
+``[magic_folder]``
+
+``enabled = (boolean, optional)``
+
+    If this is ``True``, Magic Folder will be enabled. The default value is
+    ``False``.
+
+``local.directory = (UTF-8 path)``
+
+    This specifies the local directory to be monitored for new or changed
+    files. If the path contains non-ASCII characters, it should be encoded
+    in UTF-8 regardless of the system's filesystem encoding. Relative paths
+    will be interpreted starting from the node's base directory.
+
+In addition:
+ * the file ``private/magic_folder_dircap`` must contain a writecap pointing
+   to an existing mutable directory to be used as the target of uploads.
+   It will start with ``URI:DIR2:``, and cannot include an alias or path.
+ * the file ``private/collective_dircap`` must contain a readcap
+
+After setting the above fields and starting or restarting the gateway,
+you can confirm that the feature is working by copying a file into the
+local directory. Then, use the WUI or CLI to check that it has appeared
+in the upload directory with the same filename. A large file may take some
+time to appear, since it is only linked into the directory after the upload
+has completed.
+
+The 'Operational Statistics' page linked from the Welcome page shows
+counts of the number of files uploaded, the number of change events currently
+queued, and the number of failed uploads. The 'Recent Uploads and Downloads'
+page and the node log_ may be helpful to determine the cause of any failures.
+
+.. _log: ../logging.rst
+
+
+Known Issues and Limitations
+============================
+
+This frontend only works on Linux and Windows. There is a ticket to add
+support for Mac OS X and BSD-based systems (`#1432`_).
+
+Subdirectories of the local directory are not monitored. If a subdirectory
+is created, it will be ignored. (`#1433`_)
+
+If files are created or changed in the local directory just after the gateway
+has started, it might not have connected to a sufficient number of servers
+when the upload is attempted, causing the upload to fail. (`#1449`_)
+
+Files that were created or changed in the local directory while the gateway
+was not running, will not be uploaded. (`#1458`_)
+
+The only way to determine whether uploads have failed is to look at the
+'Operational Statistics' page linked from the Welcome page. This only shows
+a count of failures, not the names of files. Uploads are never retried.
+
+The Magic Folder frontend performs its uploads sequentially (i.e. it waits
+until each upload is finished before starting the next), even when there
+would be enough memory and bandwidth to efficiently perform them in parallel.
+A Magic Folder upload can occur in parallel with an upload by a different
+frontend, though. (`#1459`_)
+
+On Linux, if there are a large number of near-simultaneous file creation or
+change events (greater than the number specified in the file
+``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events
+could be missed. This is fairly unlikely under normal circumstances, because
+the default value of ``max_queued_events`` in most Linux distributions is
+16384, and events are removed from this queue immediately without waiting for
+the corresponding upload to complete. (`#1430`_)
+
+The Windows implementation might also occasionally miss file creation or
+change events, due to limitations of the underlying Windows API
+(ReadDirectoryChangesW). We do not know how likely or unlikely this is.
+(`#1431`_)
+
+Some filesystems may not support the necessary change notifications.
+So, it is recommended for the local directory to be on a directly attached
+disk-based filesystem, not a network filesystem or one provided by a virtual
+machine.
+
+Attempts to read the mutable directory at about the same time as an uploaded
+file is being linked into it, might fail, even if they are done through the
+same gateway. (`#1105`_)
+
+When a local file is changed and closed several times in quick succession,
+it may be uploaded more times than necessary to keep the remote copy
+up-to-date. (`#1440`_)
+
+Files deleted from the local directory will not be unlinked from the upload
+directory. (`#1710`_)
+
+The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files
+cannot use an alias or path to specify the upload directory. (`#1711`_)
+
+Files are always uploaded as immutable. If there is an existing mutable file
+of the same name in the upload directory, it will be unlinked and replaced
+with an immutable file. (`#1712`_)
+
+If a file in the upload directory is changed (actually relinked to a new
+file), then the old file is still present on the grid, and any other caps to
+it will remain valid. See `docs/garbage-collection.rst`_ for how to reclaim
+the space used by files that are no longer needed.
+
+Unicode filenames are supported on both Linux and Windows, but on Linux, the
+local name of a file must be encoded correctly in order for it to be uploaded.
+The expected encoding is that printed by
+``python -c "import sys; print sys.getfilesystemencoding()"``.
+
+On Windows, local directories with non-ASCII names are not currently working.
+(`#2219`_)
+
+On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C
+(it can only be killed using Task Manager or similar). (`#2218`_)
+
+.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105
+.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430
+.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431
+.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432
+.. _`#1433`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1433
+.. _`#1440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1440
+.. _`#1449`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1449
+.. _`#1458`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1458
+.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459
+.. _`#1710`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1710
+.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711
+.. _`#1712`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1712
+.. _`#2218`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2218
+.. _`#2219`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2219
+
+.. _docs/garbage-collection.rst: ../garbage-collection.rst
+
index b0d598ba60485b8b1bf90b3b9f38d3c4540c26e0..86ae9ac23a4cfaf31bfbf533d287a7fa46688a28 100644 (file)
@@ -151,7 +151,7 @@ class Client(node.Node, pollmixin.PollMixin):
         # ControlServer and Helper are attached after Tub startup
         self.init_ftp_server()
         self.init_sftp_server()
-        self.init_drop_uploader()
+        self.init_magic_folder()
 
         # If the node sees an exit_trigger file, it will poll every second to see
         # whether the file still exists, and what its mtime is. If the file does not
@@ -492,33 +492,33 @@ class Client(node.Node, pollmixin.PollMixin):
                                  sftp_portstr, pubkey_file, privkey_file)
             s.setServiceParent(self)
 
-    def init_drop_uploader(self):
+    def init_magic_folder(self):
         if self.get_config("drop_upload", "enabled", False, boolean=True):
-            if self.get_config("drop_upload", "upload.dircap", None):
-                raise OldConfigOptionError("The [drop_upload]upload.dircap option is no longer supported; please "
-                                           "put the cap in a 'private/drop_upload_dircap' file, and delete this option.")
+            raise OldConfigOptionError("The [drop_upload] section must be renamed to [magic_folder].\n"
+                                       "See docs/frontends/magic-folder.rst for more information.")
 
-            upload_dircap = self.get_or_create_private_config("drop_upload_dircap")
-            local_dir_config = self.get_config("drop_upload", "local.directory").decode("utf-8")
+        if self.get_config("magic_folder", "enabled", False, boolean=True):
+            upload_dircap = self.get_or_create_private_config("magic_folder_dircap")
+            local_dir_config = self.get_config("magic_folder", "local.directory").decode("utf-8")
             local_dir = abspath_expanduser_unicode(local_dir_config, base=self.basedir)
 
             try:
-                from allmydata.frontends import drop_upload
+                from allmydata.frontends import magic_folder
                 dbfile = os.path.join(self.basedir, "private", "magicfolderdb.sqlite")
                 dbfile = abspath_expanduser_unicode(dbfile)
 
-                parent_dircap_path = os.path.join(self.basedir, "private", "magic_folder_parent_dircap")
-                parent_dircap_path = abspath_expanduser_unicode(parent_dircap_path)
-                parent_dircap = fileutil.read(parent_dircap_path).strip()
+                collective_dircap_path = os.path.join(self.basedir, "private", "collective_dircap")
+                collective_dircap_path = abspath_expanduser_unicode(collective_dircap_path)
+                collective_dircap = fileutil.read(collective_dircap_path).strip()
 
-                s = drop_upload.DropUploader(self, upload_dircap, parent_dircap, local_dir, dbfile)
+                s = magic_folder.MagicFolder(self, upload_dircap, collective_dircap, local_dir, dbfile)
                 s.setServiceParent(self)
                 s.startService()
 
                 # start processing the upload queue when we've connected to enough servers
                 self.upload_ready_d.addCallback(s.upload_ready)
             except Exception, e:
-                self.log("couldn't start drop-uploader: %r", args=(e,))
+                self.log("couldn't start Magic Folder: %r", args=(e,))
 
     def _check_exit_trigger(self, exit_trigger_file):
         if os.path.exists(exit_trigger_file):
diff --git a/src/allmydata/frontends/drop_upload.py b/src/allmydata/frontends/drop_upload.py
deleted file mode 100644 (file)
index 0be45f1..0000000
+++ /dev/null
@@ -1,281 +0,0 @@
-
-import sys, os, stat
-import os.path
-from collections import deque
-
-from twisted.internet import defer, reactor, task
-from twisted.python.failure import Failure
-from twisted.python import runtime
-from twisted.application import service
-
-from allmydata.interfaces import IDirectoryNode
-
-from allmydata.util import log
-from allmydata.util.fileutil import abspath_expanduser_unicode, precondition_abspath
-from allmydata.util.encodingutil import listdir_unicode, to_filepath, \
-     unicode_from_filepath, quote_local_unicode_path, FilenameEncodingError
-from allmydata.immutable.upload import FileName, Data
-from allmydata import backupdb, magicpath
-
-
-IN_EXCL_UNLINK = 0x04000000L
-
-def get_inotify_module():
-    try:
-        if sys.platform == "win32":
-            from allmydata.windows import inotify
-        elif runtime.platform.supportsINotify():
-            from twisted.internet import inotify
-        else:
-            raise NotImplementedError("filesystem notification needed for drop-upload is not supported.\n"
-                                      "This currently requires Linux or Windows.")
-        return inotify
-    except (ImportError, AttributeError) as e:
-        log.msg(e)
-        if sys.platform == "win32":
-            raise NotImplementedError("filesystem notification needed for drop-upload is not supported.\n"
-                                      "Windows support requires at least Vista, and has only been tested on Windows 7.")
-        raise
-
-
-class DropUploader(service.MultiService):
-    name = 'drop-upload'
-
-    def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None,
-                 pending_delay=1.0):
-        precondition_abspath(local_dir)
-
-        service.MultiService.__init__(self)
-        self._local_dir = abspath_expanduser_unicode(local_dir)
-        self._upload_lazy_tail = defer.succeed(None)
-        self._pending = set()
-        self._client = client
-        self._stats_provider = client.stats_provider
-        self._convergence = client.convergence
-        self._local_path = to_filepath(self._local_dir)
-        self._dbfile = dbfile
-
-        self._upload_deque = deque()
-        self.is_upload_ready = False
-
-        self._inotify = inotify or get_inotify_module()
-
-        if not self._local_path.exists():
-            raise AssertionError("The '[drop_upload] local.directory' parameter was %s "
-                                 "but there is no directory at that location."
-                                 % quote_local_unicode_path(local_dir))
-        if not self._local_path.isdir():
-            raise AssertionError("The '[drop_upload] local.directory' parameter was %s "
-                                 "but the thing at that location is not a directory."
-                                 % quote_local_unicode_path(local_dir))
-
-        # TODO: allow a path rather than a cap URI.
-        self._parent = self._client.create_node_from_uri(upload_dircap)
-        if not IDirectoryNode.providedBy(self._parent):
-            raise AssertionError("The URI in 'private/drop_upload_dircap' does not refer to a directory.")
-        if self._parent.is_unknown() or self._parent.is_readonly():
-            raise AssertionError("The URI in 'private/drop_upload_dircap' is not a writecap to a directory.")
-
-        self._processed_callback = lambda ign: None
-        self._ignore_count = 0
-
-        self._notifier = inotify.INotify()
-        if hasattr(self._notifier, 'set_pending_delay'):
-            self._notifier.set_pending_delay(pending_delay)
-
-        # We don't watch for IN_CREATE, because that would cause us to read and upload a
-        # possibly-incomplete file before the application has closed it. There should always
-        # be an IN_CLOSE_WRITE after an IN_CREATE (I think).
-        # TODO: what about IN_MOVE_SELF, IN_MOVED_FROM, or IN_UNMOUNT?
-        #
-        self.mask = ( inotify.IN_CLOSE_WRITE
-                    | inotify.IN_MOVED_TO
-                    | inotify.IN_MOVED_FROM
-                    | inotify.IN_DELETE
-                    | inotify.IN_ONLYDIR
-                    | IN_EXCL_UNLINK
-                    )
-        self._notifier.watch(self._local_path, mask=self.mask, callbacks=[self._notify],
-                             recursive=True)
-
-    def _check_db_file(self, childpath):
-        # returns True if the file must be uploaded.
-        assert self._db != None
-        r = self._db.check_file(childpath)
-        filecap = r.was_uploaded()
-        if filecap is False:
-            return True
-
-    def _scan(self, localpath):
-        if not os.path.isdir(localpath):
-            raise AssertionError("Programmer error: _scan() must be passed a directory path.")
-        quoted_path = quote_local_unicode_path(localpath)
-        try:
-            children = listdir_unicode(localpath)
-        except EnvironmentError:
-            raise(Exception("WARNING: magic folder: permission denied on directory %s" % (quoted_path,)))
-        except FilenameEncodingError:
-            raise(Exception("WARNING: magic folder: could not list directory %s due to a filename encoding error" % (quoted_path,)))
-
-        for child in children:
-            assert isinstance(child, unicode), child
-            childpath = os.path.join(localpath, child)
-            # note: symlinks to directories are both islink() and isdir()
-            isdir = os.path.isdir(childpath)
-            isfile = os.path.isfile(childpath)
-            islink = os.path.islink(childpath)
-
-            if islink:
-                self.warn("WARNING: cannot backup symlink %s" % quote_local_unicode_path(childpath))
-            elif isdir:
-                # process directories unconditionally
-                self._append_to_deque(childpath)
-
-                # recurse on the child directory
-                self._scan(childpath)
-            elif isfile:
-                must_upload = self._check_db_file(childpath)
-                if must_upload:
-                    self._append_to_deque(childpath)
-            else:
-                self.warn("WARNING: cannot backup special file %s" % quote_local_unicode_path(childpath))
-
-    def startService(self):
-        self._db = backupdb.get_backupdb(self._dbfile)
-        if self._db is None:
-            return Failure(Exception('ERROR: Unable to load magic folder db.'))
-
-        service.MultiService.startService(self)
-        d = self._notifier.startReading()
-
-        self._scan(self._local_dir)
-
-        self._stats_provider.count('drop_upload.dirs_monitored', 1)
-        return d
-
-    def upload_ready(self):
-        """upload_ready is used to signal us to start
-        processing the upload items...
-        """
-        self.is_upload_ready = True
-        self._turn_deque()
-
-    def _append_to_deque(self, path):
-        self._upload_deque.append(path)
-        self._pending.add(path)
-        self._stats_provider.count('drop_upload.objects_queued', 1)
-        if self.is_upload_ready:
-            reactor.callLater(0, self._turn_deque)
-
-    def _turn_deque(self):
-        try:
-            path = self._upload_deque.pop()
-        except IndexError:
-            self._log("magic folder upload deque is now empty")
-            self._upload_lazy_tail = defer.succeed(None)
-            return
-        self._upload_lazy_tail.addCallback(lambda ign: task.deferLater(reactor, 0, self._process, path))
-        self._upload_lazy_tail.addCallback(lambda ign: self._turn_deque())
-
-    def _notify(self, opaque, path, events_mask):
-        self._log("inotify event %r, %r, %r\n" % (opaque, path, ', '.join(self._inotify.humanReadableMask(events_mask))))
-        path_u = unicode_from_filepath(path)
-        if path_u not in self._pending:
-            self._append_to_deque(path_u)
-
-    def _process(self, path):
-        d = defer.succeed(None)
-
-        def _add_file(name):
-            u = FileName(path, self._convergence)
-            return self._parent.add_file(name, u, overwrite=True)
-
-        def _add_dir(name):
-            self._notifier.watch(to_filepath(path), mask=self.mask, callbacks=[self._notify], recursive=True)
-            u = Data("", self._convergence)
-            name += "@_"
-            d2 = self._parent.add_file(name, u, overwrite=True)
-            def _succeeded(ign):
-                self._log("created subdirectory %r" % (path,))
-                self._stats_provider.count('drop_upload.directories_created', 1)
-            def _failed(f):
-                self._log("failed to create subdirectory %r" % (path,))
-                return f
-            d2.addCallbacks(_succeeded, _failed)
-            d2.addCallback(lambda ign: self._scan(path))
-            return d2
-
-        def _maybe_upload(val):
-            self._pending.remove(path)
-            relpath = os.path.relpath(path, self._local_dir)
-            name = magicpath.path2magic(relpath)
-
-            if not os.path.exists(path):
-                self._log("drop-upload: notified object %r disappeared "
-                          "(this is normal for temporary objects)" % (path,))
-                self._stats_provider.count('drop_upload.objects_disappeared', 1)
-                return None
-            elif os.path.islink(path):
-                raise Exception("symlink not being processed")
-
-            if os.path.isdir(path):
-                return _add_dir(name)
-            elif os.path.isfile(path):
-                d2 = _add_file(name)
-                def add_db_entry(filenode):
-                    filecap = filenode.get_uri()
-                    s = os.stat(path)
-                    size = s[stat.ST_SIZE]
-                    ctime = s[stat.ST_CTIME]
-                    mtime = s[stat.ST_MTIME]
-                    self._db.did_upload_file(filecap, path, mtime, ctime, size)
-                    self._stats_provider.count('drop_upload.files_uploaded', 1)
-                d2.addCallback(add_db_entry)
-                return d2
-            else:
-                raise Exception("non-directory/non-regular file not being processed")
-
-        d.addCallback(_maybe_upload)
-
-        def _succeeded(res):
-            self._stats_provider.count('drop_upload.objects_queued', -1)
-            self._stats_provider.count('drop_upload.objects_succeeded', 1)
-            return res
-        def _failed(f):
-            self._stats_provider.count('drop_upload.objects_queued', -1)
-            self._stats_provider.count('drop_upload.objects_failed', 1)
-            self._log("%r while processing %r" % (f, path))
-            return f
-        d.addCallbacks(_succeeded, _failed)
-        d.addBoth(self._do_processed_callback)
-        return d
-
-    def _do_processed_callback(self, res):
-        if self._ignore_count == 0:
-            self._processed_callback(res)
-        else:
-            self._ignore_count -= 1
-        return None  # intentionally suppress failures, which have already been logged
-
-    def set_processed_callback(self, callback, ignore_count=0):
-        """
-        This sets a function that will be called after a notification has been processed
-        (successfully or unsuccessfully).
-        """
-        self._processed_callback = callback
-        self._ignore_count = ignore_count
-
-    def finish(self, for_tests=False):
-        self._notifier.stopReading()
-        self._stats_provider.count('drop_upload.dirs_monitored', -1)
-        if for_tests and hasattr(self._notifier, 'wait_until_stopped'):
-            return self._notifier.wait_until_stopped()
-        else:
-            return defer.succeed(None)
-
-    def remove_service(self):
-        return service.MultiService.disownServiceParent(self)
-
-    def _log(self, msg):
-        self._client.log("drop-upload: " + msg)
-        #open("events", "ab+").write(msg)
diff --git a/src/allmydata/frontends/magic_folder.py b/src/allmydata/frontends/magic_folder.py
new file mode 100644 (file)
index 0000000..79dff17
--- /dev/null
@@ -0,0 +1,281 @@
+
+import sys, os, stat
+import os.path
+from collections import deque
+
+from twisted.internet import defer, reactor, task
+from twisted.python.failure import Failure
+from twisted.python import runtime
+from twisted.application import service
+
+from allmydata.interfaces import IDirectoryNode
+
+from allmydata.util import log
+from allmydata.util.fileutil import abspath_expanduser_unicode, precondition_abspath
+from allmydata.util.encodingutil import listdir_unicode, to_filepath, \
+     unicode_from_filepath, quote_local_unicode_path, FilenameEncodingError
+from allmydata.immutable.upload import FileName, Data
+from allmydata import backupdb, magicpath
+
+
+IN_EXCL_UNLINK = 0x04000000L
+
+def get_inotify_module():
+    try:
+        if sys.platform == "win32":
+            from allmydata.windows import inotify
+        elif runtime.platform.supportsINotify():
+            from twisted.internet import inotify
+        else:
+            raise NotImplementedError("filesystem notification needed for drop-upload is not supported.\n"
+                                      "This currently requires Linux or Windows.")
+        return inotify
+    except (ImportError, AttributeError) as e:
+        log.msg(e)
+        if sys.platform == "win32":
+            raise NotImplementedError("filesystem notification needed for drop-upload is not supported.\n"
+                                      "Windows support requires at least Vista, and has only been tested on Windows 7.")
+        raise
+
+
+class MagicFolder(service.MultiService):
+    name = 'magic-folder'
+
+    def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
+                 pending_delay=1.0):
+        precondition_abspath(local_dir)
+
+        service.MultiService.__init__(self)
+        self._local_dir = abspath_expanduser_unicode(local_dir)
+        self._upload_lazy_tail = defer.succeed(None)
+        self._pending = set()
+        self._client = client
+        self._stats_provider = client.stats_provider
+        self._convergence = client.convergence
+        self._local_path = to_filepath(self._local_dir)
+        self._dbfile = dbfile
+
+        self._upload_deque = deque()
+        self.is_upload_ready = False
+
+        self._inotify = inotify or get_inotify_module()
+
+        if not self._local_path.exists():
+            raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
+                                 "but there is no directory at that location."
+                                 % quote_local_unicode_path(local_dir))
+        if not self._local_path.isdir():
+            raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
+                                 "but the thing at that location is not a directory."
+                                 % quote_local_unicode_path(local_dir))
+
+        # TODO: allow a path rather than a cap URI.
+        self._upload_dirnode = self._client.create_node_from_uri(upload_dircap)
+        if not IDirectoryNode.providedBy(self._upload_dirnode):
+            raise AssertionError("The URI in 'private/magic_folder_dircap' does not refer to a directory.")
+        if self._upload_dirnode.is_unknown() or self._upload_dirnode.is_readonly():
+            raise AssertionError("The URI in 'private/magic_folder_dircap' is not a writecap to a directory.")
+
+        self._processed_callback = lambda ign: None
+        self._ignore_count = 0
+
+        self._notifier = inotify.INotify()
+        if hasattr(self._notifier, 'set_pending_delay'):
+            self._notifier.set_pending_delay(pending_delay)
+
+        # We don't watch for IN_CREATE, because that would cause us to read and upload a
+        # possibly-incomplete file before the application has closed it. There should always
+        # be an IN_CLOSE_WRITE after an IN_CREATE (I think).
+        # TODO: what about IN_MOVE_SELF, IN_MOVED_FROM, or IN_UNMOUNT?
+        #
+        self.mask = ( inotify.IN_CLOSE_WRITE
+                    | inotify.IN_MOVED_TO
+                    | inotify.IN_MOVED_FROM
+                    | inotify.IN_DELETE
+                    | inotify.IN_ONLYDIR
+                    | IN_EXCL_UNLINK
+                    )
+        self._notifier.watch(self._local_path, mask=self.mask, callbacks=[self._notify],
+                             recursive=True)
+
+    def _check_db_file(self, childpath):
+        # returns True if the file must be uploaded.
+        assert self._db != None
+        r = self._db.check_file(childpath)
+        filecap = r.was_uploaded()
+        if filecap is False:
+            return True
+
+    def _scan(self, localpath):
+        if not os.path.isdir(localpath):
+            raise AssertionError("Programmer error: _scan() must be passed a directory path.")
+        quoted_path = quote_local_unicode_path(localpath)
+        try:
+            children = listdir_unicode(localpath)
+        except EnvironmentError:
+            raise(Exception("WARNING: magic folder: permission denied on directory %s" % (quoted_path,)))
+        except FilenameEncodingError:
+            raise(Exception("WARNING: magic folder: could not list directory %s due to a filename encoding error" % (quoted_path,)))
+
+        for child in children:
+            assert isinstance(child, unicode), child
+            childpath = os.path.join(localpath, child)
+            # note: symlinks to directories are both islink() and isdir()
+            isdir = os.path.isdir(childpath)
+            isfile = os.path.isfile(childpath)
+            islink = os.path.islink(childpath)
+
+            if islink:
+                self.warn("WARNING: cannot backup symlink %s" % quote_local_unicode_path(childpath))
+            elif isdir:
+                # process directories unconditionally
+                self._append_to_deque(childpath)
+
+                # recurse on the child directory
+                self._scan(childpath)
+            elif isfile:
+                must_upload = self._check_db_file(childpath)
+                if must_upload:
+                    self._append_to_deque(childpath)
+            else:
+                self.warn("WARNING: cannot backup special file %s" % quote_local_unicode_path(childpath))
+
+    def startService(self):
+        self._db = backupdb.get_backupdb(self._dbfile)
+        if self._db is None:
+            return Failure(Exception('ERROR: Unable to load magic folder db.'))
+
+        service.MultiService.startService(self)
+        d = self._notifier.startReading()
+
+        self._scan(self._local_dir)
+
+        self._stats_provider.count('magic_folder.dirs_monitored', 1)
+        return d
+
+    def upload_ready(self):
+        """upload_ready is used to signal us to start
+        processing the upload items...
+        """
+        self.is_upload_ready = True
+        self._turn_deque()
+
+    def _append_to_deque(self, path):
+        self._upload_deque.append(path)
+        self._pending.add(path)
+        self._stats_provider.count('magic_folder.objects_queued', 1)
+        if self.is_upload_ready:
+            reactor.callLater(0, self._turn_deque)
+
+    def _turn_deque(self):
+        try:
+            path = self._upload_deque.pop()
+        except IndexError:
+            self._log("magic folder upload deque is now empty")
+            self._upload_lazy_tail = defer.succeed(None)
+            return
+        self._upload_lazy_tail.addCallback(lambda ign: task.deferLater(reactor, 0, self._process, path))
+        self._upload_lazy_tail.addCallback(lambda ign: self._turn_deque())
+
+    def _notify(self, opaque, path, events_mask):
+        self._log("inotify event %r, %r, %r\n" % (opaque, path, ', '.join(self._inotify.humanReadableMask(events_mask))))
+        path_u = unicode_from_filepath(path)
+        if path_u not in self._pending:
+            self._append_to_deque(path_u)
+
+    def _process(self, path):
+        d = defer.succeed(None)
+
+        def _add_file(name):
+            u = FileName(path, self._convergence)
+            return self._upload_dirnode.add_file(name, u, overwrite=True)
+
+        def _add_dir(name):
+            self._notifier.watch(to_filepath(path), mask=self.mask, callbacks=[self._notify], recursive=True)
+            u = Data("", self._convergence)
+            name += "@_"
+            d2 = self._upload_dirnode.add_file(name, u, overwrite=True)
+            def _succeeded(ign):
+                self._log("created subdirectory %r" % (path,))
+                self._stats_provider.count('magic_folder.directories_created', 1)
+            def _failed(f):
+                self._log("failed to create subdirectory %r" % (path,))
+                return f
+            d2.addCallbacks(_succeeded, _failed)
+            d2.addCallback(lambda ign: self._scan(path))
+            return d2
+
+        def _maybe_upload(val):
+            self._pending.remove(path)
+            relpath = os.path.relpath(path, self._local_dir)
+            name = magicpath.path2magic(relpath)
+
+            if not os.path.exists(path):
+                self._log("drop-upload: notified object %r disappeared "
+                          "(this is normal for temporary objects)" % (path,))
+                self._stats_provider.count('magic_folder.objects_disappeared', 1)
+                return None
+            elif os.path.islink(path):
+                raise Exception("symlink not being processed")
+
+            if os.path.isdir(path):
+                return _add_dir(name)
+            elif os.path.isfile(path):
+                d2 = _add_file(name)
+                def add_db_entry(filenode):
+                    filecap = filenode.get_uri()
+                    s = os.stat(path)
+                    size = s[stat.ST_SIZE]
+                    ctime = s[stat.ST_CTIME]
+                    mtime = s[stat.ST_MTIME]
+                    self._db.did_upload_file(filecap, path, mtime, ctime, size)
+                    self._stats_provider.count('magic_folder.files_uploaded', 1)
+                d2.addCallback(add_db_entry)
+                return d2
+            else:
+                raise Exception("non-directory/non-regular file not being processed")
+
+        d.addCallback(_maybe_upload)
+
+        def _succeeded(res):
+            self._stats_provider.count('magic_folder.objects_queued', -1)
+            self._stats_provider.count('magic_folder.objects_succeeded', 1)
+            return res
+        def _failed(f):
+            self._stats_provider.count('magic_folder.objects_queued', -1)
+            self._stats_provider.count('magic_folder.objects_failed', 1)
+            self._log("%r while processing %r" % (f, path))
+            return f
+        d.addCallbacks(_succeeded, _failed)
+        d.addBoth(self._do_processed_callback)
+        return d
+
+    def _do_processed_callback(self, res):
+        if self._ignore_count == 0:
+            self._processed_callback(res)
+        else:
+            self._ignore_count -= 1
+        return None  # intentionally suppress failures, which have already been logged
+
+    def set_processed_callback(self, callback, ignore_count=0):
+        """
+        This sets a function that will be called after a notification has been processed
+        (successfully or unsuccessfully).
+        """
+        self._processed_callback = callback
+        self._ignore_count = ignore_count
+
+    def finish(self, for_tests=False):
+        self._notifier.stopReading()
+        self._stats_provider.count('magic_folder.dirs_monitored', -1)
+        if for_tests and hasattr(self._notifier, 'wait_until_stopped'):
+            return self._notifier.wait_until_stopped()
+        else:
+            return defer.succeed(None)
+
+    def remove_service(self):
+        return service.MultiService.disownServiceParent(self)
+
+    def _log(self, msg):
+        self._client.log("drop-upload: " + msg)
+        #open("events", "ab+").write(msg)
index 0547d7a0af1a0e8db2fdca7574868ec73702ec40..74ed4f7420949cb0683eec94cac8b93ac573ea0a 100644 (file)
@@ -4,7 +4,7 @@ from twisted.trial import unittest
 from twisted.application import service
 
 import allmydata
-import allmydata.frontends.drop_upload
+import allmydata.frontends.magic_folder
 import allmydata.util.log
 
 from allmydata.node import Node, OldConfigError, OldConfigOptionError, MissingConfigEntry, UnescapedHashError
@@ -302,20 +302,21 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
         _check("helper.furl = None", None)
         _check("helper.furl = pb://blah\n", "pb://blah")
 
-    def test_create_drop_uploader(self):
-        class MockDropUploader(service.MultiService):
-            name = 'drop-upload'
+    def test_create_magic_folder(self):
+        class MockMagicFolder(service.MultiService):
+            name = 'magic-folder'
 
-            def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None,
+            def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
                          pending_delay=1.0):
                 service.MultiService.__init__(self)
                 self.client = client
                 self.upload_dircap = upload_dircap
+                self.collective_dircap = collective_dircap
                 self.local_dir = local_dir
                 self.dbfile = dbfile
                 self.inotify = inotify
 
-        self.patch(allmydata.frontends.drop_upload, 'DropUploader', MockDropUploader)
+        self.patch(allmydata.frontends.magic_folder, 'MagicFolder', MockMagicFolder)
 
         upload_dircap = "URI:DIR2:blah"
         local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir")
@@ -323,10 +324,10 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
         config = (BASECONFIG +
                   "[storage]\n" +
                   "enabled = false\n" +
-                  "[drop_upload]\n" +
+                  "[magic_folder]\n" +
                   "enabled = true\n")
 
-        basedir1 = "test_client.Basic.test_create_drop_uploader1"
+        basedir1 = "test_client.Basic.test_create_magic_folder1"
         os.mkdir(basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
@@ -334,48 +335,49 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
         self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config)
-        fileutil.write(os.path.join(basedir1, "private", "drop_upload_dircap"), "URI:DIR2:blah")
-        fileutil.write(os.path.join(basedir1, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow")
+        fileutil.write(os.path.join(basedir1, "private", "magic_folder_dircap"), "URI:DIR2:blah")
+        fileutil.write(os.path.join(basedir1, "private", "collective_dircap"), "URI:DIR2:meow")
         self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
-                       config + "upload.dircap = " + upload_dircap + "\n")
+                       config.replace("[magic_folder]\n", "[drop_upload]\n"))
         self.failUnlessRaises(OldConfigOptionError, client.Client, basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
                        config + "local.directory = " + local_dir_utf8 + "\n")
         c1 = client.Client(basedir1)
-        uploader = c1.getServiceNamed('drop-upload')
-        self.failUnless(isinstance(uploader, MockDropUploader), uploader)
-        self.failUnlessReallyEqual(uploader.client, c1)
-        self.failUnlessReallyEqual(uploader.upload_dircap, upload_dircap)
-        self.failUnlessReallyEqual(os.path.basename(uploader.local_dir), local_dir_u)
-        self.failUnless(uploader.inotify is None, uploader.inotify)
-        self.failUnless(uploader.running)
+        magicfolder = c1.getServiceNamed('magic-folder')
+        self.failUnless(isinstance(magicfolder, MockMagicFolder), magicfolder)
+        self.failUnlessReallyEqual(magicfolder.client, c1)
+        self.failUnlessReallyEqual(magicfolder.upload_dircap, upload_dircap)
+        self.failUnlessReallyEqual(os.path.basename(magicfolder.local_dir), local_dir_u)
+        self.failUnless(magicfolder.inotify is None, magicfolder.inotify)
+        self.failUnless(magicfolder.running)
 
         class Boom(Exception):
             pass
-        def BoomDropUploader(client, upload_dircap, local_dir_utf8, inotify=None):
+        def BoomMagicFolder(self, client, upload_dircap, collective_dircap, local_dir, dbfile,
+                            inotify=None, pending_delay=1.0):
             raise Boom()
 
         logged_messages = []
         def mock_log(*args, **kwargs):
             logged_messages.append("%r %r" % (args, kwargs))
         self.patch(allmydata.util.log, 'msg', mock_log)
-        self.patch(allmydata.frontends.drop_upload, 'DropUploader', BoomDropUploader)
+        self.patch(allmydata.frontends.magic_folder, 'MagicFolder', BoomMagicFolder)
 
-        basedir2 = "test_client.Basic.test_create_drop_uploader2"
+        basedir2 = "test_client.Basic.test_create_magic_folder2"
         os.mkdir(basedir2)
         os.mkdir(os.path.join(basedir2, "private"))
         fileutil.write(os.path.join(basedir2, "tahoe.cfg"),
                        BASECONFIG +
-                       "[drop_upload]\n" +
+                       "[magic_folder]\n" +
                        "enabled = true\n" +
                        "local.directory = " + local_dir_utf8 + "\n")
-        fileutil.write(os.path.join(basedir2, "private", "drop_upload_dircap"), "URI:DIR2:blah")
-        fileutil.write(os.path.join(basedir2, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow")
+        fileutil.write(os.path.join(basedir2, "private", "magic_folder_dircap"), "URI:DIR2:blah")
+        fileutil.write(os.path.join(basedir2, "private", "collective_dircap"), "URI:DIR2:meow")
         c2 = client.Client(basedir2)
-        self.failUnlessRaises(KeyError, c2.getServiceNamed, 'drop-upload')
+        self.failUnlessRaises(KeyError, c2.getServiceNamed, 'magic-folder')
         self.failUnless([True for arg in logged_messages if "Boom" in arg],
                         logged_messages)
 
diff --git a/src/allmydata/test/test_drop_upload.py b/src/allmydata/test/test_drop_upload.py
deleted file mode 100644 (file)
index 6be6d33..0000000
+++ /dev/null
@@ -1,384 +0,0 @@
-
-import os, sys, stat, time
-
-from twisted.trial import unittest
-from twisted.internet import defer
-
-from allmydata.interfaces import IDirectoryNode, NoSuchChildError
-
-from allmydata.util import fake_inotify, fileutil
-from allmydata.util.encodingutil import get_filesystem_encoding, to_filepath
-from allmydata.util.consumer import download_to_data
-from allmydata.test.no_network import GridTestMixin
-from allmydata.test.common_util import ReallyEqualMixin, NonASCIIPathMixin
-from allmydata.test.common import ShouldFailMixin
-
-from allmydata.frontends import drop_upload
-from allmydata.frontends.drop_upload import DropUploader
-from allmydata import backupdb
-from allmydata.util.fileutil import abspath_expanduser_unicode
-
-
-class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonASCIIPathMixin):
-    """
-    These tests will be run both with a mock notifier, and (on platforms that support it)
-    with the real INotify.
-    """
-
-    def setUp(self):
-        GridTestMixin.setUp(self)
-        temp = self.mktemp()
-        self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding()))
-        self.uploader = None
-        self.dir_node = None
-
-    def _get_count(self, name):
-        return self.stats_provider.get_stats()["counters"].get(name, 0)
-
-    def _createdb(self):
-        dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir)
-        bdb = backupdb.get_backupdb(dbfile)
-        self.failUnless(bdb, "unable to create backupdb from %r" % (dbfile,))
-        self.failUnlessEqual(bdb.VERSION, 2)
-        return bdb
-
-    def _made_upload_dir(self, n):
-        if self.dir_node == None:
-            self.dir_node = n
-        else:
-            n = self.dir_node
-        self.failUnless(IDirectoryNode.providedBy(n))
-        self.upload_dirnode = n
-        self.upload_dircap = n.get_uri()
-        self.parent_dircap = "abc123"
-
-    def _create_uploader(self, ign):
-        dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir)
-        self.uploader = DropUploader(self.client, self.upload_dircap, self.parent_dircap, self.local_dir,
-                                     dbfile, inotify=self.inotify, pending_delay=0.2)
-        self.uploader.setServiceParent(self.client)
-        self.uploader.upload_ready()
-
-    # Prevent unclean reactor errors.
-    def _cleanup(self, res):
-        d = defer.succeed(None)
-        if self.uploader is not None:
-            d.addCallback(lambda ign: self.uploader.finish(for_tests=True))
-        d.addCallback(lambda ign: res)
-        return d
-
-    def test_db_basic(self):
-        fileutil.make_dirs(self.basedir)
-        self._createdb()
-
-    def test_db_persistence(self):
-        """Test that a file upload creates an entry in the database."""
-
-        fileutil.make_dirs(self.basedir)
-        db = self._createdb()
-
-        path = abspath_expanduser_unicode(u"myFile1", base=self.basedir)
-        db.did_upload_file('URI:LIT:1', path, 0, 0, 33)
-
-        c = db.cursor
-        c.execute("SELECT size,mtime,ctime,fileid"
-                  " FROM local_files"
-                  " WHERE path=?",
-                  (path,))
-        row = db.cursor.fetchone()
-        self.failIfEqual(row, None)
-
-        # Second test uses db.check_file instead of SQL query directly
-        # to confirm the previous upload entry in the db.
-        path = abspath_expanduser_unicode(u"myFile2", base=self.basedir)
-        fileutil.write(path, "meow\n")
-        s = os.stat(path)
-        size = s[stat.ST_SIZE]
-        ctime = s[stat.ST_CTIME]
-        mtime = s[stat.ST_MTIME]
-        db.did_upload_file('URI:LIT:2', path, mtime, ctime, size)
-        r = db.check_file(path)
-        self.failUnless(r.was_uploaded())
-
-    def test_uploader_start_service(self):
-        self.set_up_grid()
-
-        self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"),
-                                                    base=self.basedir)
-        self.mkdir_nonascii(self.local_dir)
-
-        self.client = self.g.clients[0]
-        self.stats_provider = self.client.stats_provider
-
-        d = self.client.create_dirnode()
-        d.addCallback(self._made_upload_dir)
-        d.addCallback(self._create_uploader)
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.dirs_monitored'), 1))
-        d.addBoth(self._cleanup)
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.dirs_monitored'), 0))
-        return d
-
-    def test_move_tree(self):
-        self.set_up_grid()
-
-        self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"),
-                                                    base=self.basedir)
-        self.mkdir_nonascii(self.local_dir)
-
-        self.client = self.g.clients[0]
-        self.stats_provider = self.client.stats_provider
-
-        empty_tree_name = self.unicode_or_fallback(u"empty_tr\u00EAe", u"empty_tree")
-        empty_tree_dir = abspath_expanduser_unicode(empty_tree_name, base=self.basedir)
-        new_empty_tree_dir = abspath_expanduser_unicode(empty_tree_name, base=self.local_dir)
-
-        small_tree_name = self.unicode_or_fallback(u"small_tr\u00EAe", u"empty_tree")
-        small_tree_dir = abspath_expanduser_unicode(small_tree_name, base=self.basedir)
-        new_small_tree_dir = abspath_expanduser_unicode(small_tree_name, base=self.local_dir)
-
-        d = self.client.create_dirnode()
-        d.addCallback(self._made_upload_dir)
-
-        d.addCallback(self._create_uploader)
-
-        def _check_move_empty_tree(res):
-            self.mkdir_nonascii(empty_tree_dir)
-            d2 = defer.Deferred()
-            self.uploader.set_processed_callback(d2.callback, ignore_count=0)
-            os.rename(empty_tree_dir, new_empty_tree_dir)
-            self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_TO)
-            return d2
-        d.addCallback(_check_move_empty_tree)
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 1))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 0))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 1))
-
-        def _check_move_small_tree(res):
-            self.mkdir_nonascii(small_tree_dir)
-            fileutil.write(abspath_expanduser_unicode(u"what", base=small_tree_dir), "say when")
-            d2 = defer.Deferred()
-            self.uploader.set_processed_callback(d2.callback, ignore_count=1)
-            os.rename(small_tree_dir, new_small_tree_dir)
-            self.notify(to_filepath(new_small_tree_dir), self.inotify.IN_MOVED_TO)
-            return d2
-        d.addCallback(_check_move_small_tree)
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 3))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 1))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2))
-
-        def _check_moved_tree_is_watched(res):
-            d2 = defer.Deferred()
-            self.uploader.set_processed_callback(d2.callback, ignore_count=0)
-            fileutil.write(abspath_expanduser_unicode(u"another", base=new_small_tree_dir), "file")
-            self.notify(to_filepath(abspath_expanduser_unicode(u"another", base=new_small_tree_dir)), self.inotify.IN_CLOSE_WRITE)
-            return d2
-        d.addCallback(_check_moved_tree_is_watched)
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 4))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 2))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2))
-
-        # Files that are moved out of the upload directory should no longer be watched.
-        def _move_dir_away(ign):
-            os.rename(new_empty_tree_dir, empty_tree_dir)
-            # Wuh? Why don't we get this event for the real test?
-            #self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_FROM)
-        d.addCallback(_move_dir_away)
-        def create_file(val):
-            test_file = abspath_expanduser_unicode(u"what", base=empty_tree_dir)
-            fileutil.write(test_file, "meow")
-            return
-        d.addCallback(create_file)
-        d.addCallback(lambda ign: time.sleep(1))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 4))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 2))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2))
-
-        d.addBoth(self._cleanup)
-        return d
-
-    def test_persistence(self):
-        """
-        Perform an upload of a given file and then stop the client.
-        Start a new client and uploader... and verify that the file is NOT uploaded
-        a second time. This test is meant to test the database persistence along with
-        the startup and shutdown code paths of the uploader.
-        """
-        self.set_up_grid()
-        self.local_dir = abspath_expanduser_unicode(u"test_persistence", base=self.basedir)
-        self.mkdir_nonascii(self.local_dir)
-
-        self.client = self.g.clients[0]
-        self.stats_provider = self.client.stats_provider
-        d = self.client.create_dirnode()
-        d.addCallback(self._made_upload_dir)
-        d.addCallback(self._create_uploader)
-
-        def create_file(val):
-            d2 = defer.Deferred()
-            self.uploader.set_processed_callback(d2.callback)
-            test_file = abspath_expanduser_unicode(u"what", base=self.local_dir)
-            fileutil.write(test_file, "meow")
-            self.notify(to_filepath(test_file), self.inotify.IN_CLOSE_WRITE)
-            return d2
-        d.addCallback(create_file)
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 1))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
-        d.addCallback(self._cleanup)
-
-        def _restart(ign):
-            self.set_up_grid()
-            self.client = self.g.clients[0]
-            self.stats_provider = self.client.stats_provider
-        d.addCallback(_restart)
-        d.addCallback(self._create_uploader)
-        d.addCallback(lambda ign: time.sleep(3))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 0))
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
-        d.addBoth(self._cleanup)
-        return d
-
-    def test_drop_upload(self):
-        self.set_up_grid()
-        self.local_dir = os.path.join(self.basedir, self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir"))
-        self.mkdir_nonascii(self.local_dir)
-
-        self.client = self.g.clients[0]
-        self.stats_provider = self.client.stats_provider
-
-        d = self.client.create_dirnode()
-
-        d.addCallback(self._made_upload_dir)
-        d.addCallback(self._create_uploader)
-
-        # Write something short enough for a LIT file.
-        d.addCallback(lambda ign: self._check_file(u"short", "test"))
-
-        # Write to the same file again with different data.
-        d.addCallback(lambda ign: self._check_file(u"short", "different"))
-
-        # Test that temporary files are not uploaded.
-        d.addCallback(lambda ign: self._check_file(u"tempfile", "test", temporary=True))
-
-        # Test that we tolerate creation of a subdirectory.
-        d.addCallback(lambda ign: os.mkdir(os.path.join(self.local_dir, u"directory")))
-
-        # Write something longer, and also try to test a Unicode name if the fs can represent it.
-        name_u = self.unicode_or_fallback(u"l\u00F8ng", u"long")
-        d.addCallback(lambda ign: self._check_file(name_u, "test"*100))
-
-        # TODO: test that causes an upload failure.
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_failed'), 0))
-
-        d.addBoth(self._cleanup)
-        return d
-
-    def _check_file(self, name_u, data, temporary=False):
-        previously_uploaded = self._get_count('drop_upload.objects_succeeded')
-        previously_disappeared = self._get_count('drop_upload.objects_disappeared')
-
-        d = defer.Deferred()
-
-        # Note: this relies on the fact that we only get one IN_CLOSE_WRITE notification per file
-        # (otherwise we would get a defer.AlreadyCalledError). Should we be relying on that?
-        self.uploader.set_processed_callback(d.callback)
-
-        path_u = abspath_expanduser_unicode(name_u, base=self.local_dir)
-        path = to_filepath(path_u)
-
-        # We don't use FilePath.setContent() here because it creates a temporary file that
-        # is renamed into place, which causes events that the test is not expecting.
-        f = open(path_u, "wb")
-        try:
-            if temporary and sys.platform != "win32":
-                os.unlink(path_u)
-            f.write(data)
-        finally:
-            f.close()
-        if temporary and sys.platform == "win32":
-            os.unlink(path_u)
-            self.notify(path, self.inotify.IN_DELETE)
-        fileutil.flush_volume(path_u)
-        self.notify(path, self.inotify.IN_CLOSE_WRITE)
-
-        if temporary:
-            d.addCallback(lambda ign: self.shouldFail(NoSuchChildError, 'temp file not uploaded', None,
-            self.upload_dirnode.get, name_u))
-            d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_disappeared'),
-                                                                 previously_disappeared + 1))
-        else:
-            d.addCallback(lambda ign: self.upload_dirnode.get(name_u))
-            d.addCallback(download_to_data)
-            d.addCallback(lambda actual_data: self.failUnlessReallyEqual(actual_data, data))
-            d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'),
-                                                                 previously_uploaded + 1))
-
-        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
-        return d
-
-
-class MockTest(DropUploadTestMixin, unittest.TestCase):
-    """This can run on any platform, and even if twisted.internet.inotify can't be imported."""
-
-    def setUp(self):
-        DropUploadTestMixin.setUp(self)
-        self.inotify = fake_inotify
-
-    def notify(self, path, mask):
-        self.uploader._notifier.event(path, mask)
-
-    def test_errors(self):
-        self.set_up_grid()
-
-        errors_dir = abspath_expanduser_unicode(u"errors_dir", base=self.basedir)
-        os.mkdir(errors_dir)
-        not_a_dir = abspath_expanduser_unicode(u"NOT_A_DIR", base=self.basedir)
-        fileutil.write(not_a_dir, "")
-        magicfolderdb = abspath_expanduser_unicode(u"magicfolderdb", base=self.basedir)
-        doesnotexist  = abspath_expanduser_unicode(u"doesnotexist", base=self.basedir)
-
-        client = self.g.clients[0]
-        d = client.create_dirnode()
-        def _check_errors(n):
-            self.failUnless(IDirectoryNode.providedBy(n))
-            upload_dircap = n.get_uri()
-            readonly_dircap = n.get_readonly_uri()
-
-            self.shouldFail(AssertionError, 'nonexistent local.directory', 'there is no directory',
-                            DropUploader, client, upload_dircap, '', doesnotexist, magicfolderdb, inotify=fake_inotify)
-            self.shouldFail(AssertionError, 'non-directory local.directory', 'is not a directory',
-                            DropUploader, client, upload_dircap, '', not_a_dir, magicfolderdb, inotify=fake_inotify)
-            self.shouldFail(AssertionError, 'bad upload.dircap', 'does not refer to a directory',
-                            DropUploader, client, 'bad', '', errors_dir, magicfolderdb, inotify=fake_inotify)
-            self.shouldFail(AssertionError, 'non-directory upload.dircap', 'does not refer to a directory',
-                            DropUploader, client, 'URI:LIT:foo', '', errors_dir, magicfolderdb, inotify=fake_inotify)
-            self.shouldFail(AssertionError, 'readonly upload.dircap', 'is not a writecap to a directory',
-                            DropUploader, client, readonly_dircap, '', errors_dir, magicfolderdb, inotify=fake_inotify)
-
-            def _not_implemented():
-                raise NotImplementedError("blah")
-            self.patch(drop_upload, 'get_inotify_module', _not_implemented)
-            self.shouldFail(NotImplementedError, 'unsupported', 'blah',
-                            DropUploader, client, upload_dircap, '', errors_dir, magicfolderdb)
-        d.addCallback(_check_errors)
-        return d
-
-
-class RealTest(DropUploadTestMixin, unittest.TestCase):
-    """This is skipped unless both Twisted and the platform support inotify."""
-
-    def setUp(self):
-        DropUploadTestMixin.setUp(self)
-        self.inotify = drop_upload.get_inotify_module()
-
-    def notify(self, path, mask):
-        # Writing to the filesystem causes the notification.
-        pass
-
-try:
-    drop_upload.get_inotify_module()
-except NotImplementedError:
-    RealTest.skip = "Drop-upload support can only be tested for-real on an OS that supports inotify or equivalent."
diff --git a/src/allmydata/test/test_magic_folder.py b/src/allmydata/test/test_magic_folder.py
new file mode 100644 (file)
index 0000000..80e2c8e
--- /dev/null
@@ -0,0 +1,384 @@
+
+import os, sys, stat, time
+
+from twisted.trial import unittest
+from twisted.internet import defer
+
+from allmydata.interfaces import IDirectoryNode, NoSuchChildError
+
+from allmydata.util import fake_inotify, fileutil
+from allmydata.util.encodingutil import get_filesystem_encoding, to_filepath
+from allmydata.util.consumer import download_to_data
+from allmydata.test.no_network import GridTestMixin
+from allmydata.test.common_util import ReallyEqualMixin, NonASCIIPathMixin
+from allmydata.test.common import ShouldFailMixin
+
+from allmydata.frontends import magic_folder
+from allmydata.frontends.magic_folder import MagicFolder
+from allmydata import backupdb
+from allmydata.util.fileutil import abspath_expanduser_unicode
+
+
+class MagicFolderTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonASCIIPathMixin):
+    """
+    These tests will be run both with a mock notifier, and (on platforms that support it)
+    with the real INotify.
+    """
+
+    def setUp(self):
+        GridTestMixin.setUp(self)
+        temp = self.mktemp()
+        self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding()))
+        self.magicfolder = None
+        self.dir_node = None
+
+    def _get_count(self, name):
+        return self.stats_provider.get_stats()["counters"].get(name, 0)
+
+    def _createdb(self):
+        dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir)
+        bdb = backupdb.get_backupdb(dbfile)
+        self.failUnless(bdb, "unable to create backupdb from %r" % (dbfile,))
+        self.failUnlessEqual(bdb.VERSION, 2)
+        return bdb
+
+    def _made_upload_dir(self, n):
+        if self.dir_node == None:
+            self.dir_node = n
+        else:
+            n = self.dir_node
+        self.failUnless(IDirectoryNode.providedBy(n))
+        self.upload_dirnode = n
+        self.upload_dircap = n.get_uri()
+        self.collective_dircap = "abc123"
+
+    def _create_magicfolder(self, ign):
+        dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir)
+        self.magicfolder = MagicFolder(self.client, self.upload_dircap, self.collective_dircap, self.local_dir,
+                                       dbfile, inotify=self.inotify, pending_delay=0.2)
+        self.magicfolder.setServiceParent(self.client)
+        self.magicfolder.upload_ready()
+
+    # Prevent unclean reactor errors.
+    def _cleanup(self, res):
+        d = defer.succeed(None)
+        if self.magicfolder is not None:
+            d.addCallback(lambda ign: self.magicfolder.finish(for_tests=True))
+        d.addCallback(lambda ign: res)
+        return d
+
+    def test_db_basic(self):
+        fileutil.make_dirs(self.basedir)
+        self._createdb()
+
+    def test_db_persistence(self):
+        """Test that a file upload creates an entry in the database."""
+
+        fileutil.make_dirs(self.basedir)
+        db = self._createdb()
+
+        path = abspath_expanduser_unicode(u"myFile1", base=self.basedir)
+        db.did_upload_file('URI:LIT:1', path, 0, 0, 33)
+
+        c = db.cursor
+        c.execute("SELECT size,mtime,ctime,fileid"
+                  " FROM local_files"
+                  " WHERE path=?",
+                  (path,))
+        row = db.cursor.fetchone()
+        self.failIfEqual(row, None)
+
+        # Second test uses db.check_file instead of SQL query directly
+        # to confirm the previous upload entry in the db.
+        path = abspath_expanduser_unicode(u"myFile2", base=self.basedir)
+        fileutil.write(path, "meow\n")
+        s = os.stat(path)
+        size = s[stat.ST_SIZE]
+        ctime = s[stat.ST_CTIME]
+        mtime = s[stat.ST_MTIME]
+        db.did_upload_file('URI:LIT:2', path, mtime, ctime, size)
+        r = db.check_file(path)
+        self.failUnless(r.was_uploaded())
+
+    def test_magicfolder_start_service(self):
+        self.set_up_grid()
+
+        self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"),
+                                                    base=self.basedir)
+        self.mkdir_nonascii(self.local_dir)
+
+        self.client = self.g.clients[0]
+        self.stats_provider = self.client.stats_provider
+
+        d = self.client.create_dirnode()
+        d.addCallback(self._made_upload_dir)
+        d.addCallback(self._create_magicfolder)
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.dirs_monitored'), 1))
+        d.addBoth(self._cleanup)
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.dirs_monitored'), 0))
+        return d
+
+    def test_move_tree(self):
+        self.set_up_grid()
+
+        self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"),
+                                                    base=self.basedir)
+        self.mkdir_nonascii(self.local_dir)
+
+        self.client = self.g.clients[0]
+        self.stats_provider = self.client.stats_provider
+
+        empty_tree_name = self.unicode_or_fallback(u"empty_tr\u00EAe", u"empty_tree")
+        empty_tree_dir = abspath_expanduser_unicode(empty_tree_name, base=self.basedir)
+        new_empty_tree_dir = abspath_expanduser_unicode(empty_tree_name, base=self.local_dir)
+
+        small_tree_name = self.unicode_or_fallback(u"small_tr\u00EAe", u"empty_tree")
+        small_tree_dir = abspath_expanduser_unicode(small_tree_name, base=self.basedir)
+        new_small_tree_dir = abspath_expanduser_unicode(small_tree_name, base=self.local_dir)
+
+        d = self.client.create_dirnode()
+        d.addCallback(self._made_upload_dir)
+
+        d.addCallback(self._create_magicfolder)
+
+        def _check_move_empty_tree(res):
+            self.mkdir_nonascii(empty_tree_dir)
+            d2 = defer.Deferred()
+            self.magicfolder.set_processed_callback(d2.callback, ignore_count=0)
+            os.rename(empty_tree_dir, new_empty_tree_dir)
+            self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_TO)
+            return d2
+        d.addCallback(_check_move_empty_tree)
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 1))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 0))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 1))
+
+        def _check_move_small_tree(res):
+            self.mkdir_nonascii(small_tree_dir)
+            fileutil.write(abspath_expanduser_unicode(u"what", base=small_tree_dir), "say when")
+            d2 = defer.Deferred()
+            self.magicfolder.set_processed_callback(d2.callback, ignore_count=1)
+            os.rename(small_tree_dir, new_small_tree_dir)
+            self.notify(to_filepath(new_small_tree_dir), self.inotify.IN_MOVED_TO)
+            return d2
+        d.addCallback(_check_move_small_tree)
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 3))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 1))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
+
+        def _check_moved_tree_is_watched(res):
+            d2 = defer.Deferred()
+            self.magicfolder.set_processed_callback(d2.callback, ignore_count=0)
+            fileutil.write(abspath_expanduser_unicode(u"another", base=new_small_tree_dir), "file")
+            self.notify(to_filepath(abspath_expanduser_unicode(u"another", base=new_small_tree_dir)), self.inotify.IN_CLOSE_WRITE)
+            return d2
+        d.addCallback(_check_moved_tree_is_watched)
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 4))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 2))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
+
+        # Files that are moved out of the upload directory should no longer be watched.
+        def _move_dir_away(ign):
+            os.rename(new_empty_tree_dir, empty_tree_dir)
+            # Wuh? Why don't we get this event for the real test?
+            #self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_FROM)
+        d.addCallback(_move_dir_away)
+        def create_file(val):
+            test_file = abspath_expanduser_unicode(u"what", base=empty_tree_dir)
+            fileutil.write(test_file, "meow")
+            return
+        d.addCallback(create_file)
+        d.addCallback(lambda ign: time.sleep(1))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 4))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 2))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
+
+        d.addBoth(self._cleanup)
+        return d
+
+    def test_persistence(self):
+        """
+        Perform an upload of a given file and then stop the client.
+        Start a new client and magic-folder service... and verify that the file is NOT uploaded
+        a second time. This test is meant to test the database persistence along with
+        the startup and shutdown code paths of the magic-folder service.
+        """
+        self.set_up_grid()
+        self.local_dir = abspath_expanduser_unicode(u"test_persistence", base=self.basedir)
+        self.mkdir_nonascii(self.local_dir)
+
+        self.client = self.g.clients[0]
+        self.stats_provider = self.client.stats_provider
+        d = self.client.create_dirnode()
+        d.addCallback(self._made_upload_dir)
+        d.addCallback(self._create_magicfolder)
+
+        def create_file(val):
+            d2 = defer.Deferred()
+            self.magicfolder.set_processed_callback(d2.callback)
+            test_file = abspath_expanduser_unicode(u"what", base=self.local_dir)
+            fileutil.write(test_file, "meow")
+            self.notify(to_filepath(test_file), self.inotify.IN_CLOSE_WRITE)
+            return d2
+        d.addCallback(create_file)
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 1))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
+        d.addCallback(self._cleanup)
+
+        def _restart(ign):
+            self.set_up_grid()
+            self.client = self.g.clients[0]
+            self.stats_provider = self.client.stats_provider
+        d.addCallback(_restart)
+        d.addCallback(self._create_magicfolder)
+        d.addCallback(lambda ign: time.sleep(3))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 0))
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
+        d.addBoth(self._cleanup)
+        return d
+
+    def test_magic_folder(self):
+        self.set_up_grid()
+        self.local_dir = os.path.join(self.basedir, self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir"))
+        self.mkdir_nonascii(self.local_dir)
+
+        self.client = self.g.clients[0]
+        self.stats_provider = self.client.stats_provider
+
+        d = self.client.create_dirnode()
+
+        d.addCallback(self._made_upload_dir)
+        d.addCallback(self._create_magicfolder)
+
+        # Write something short enough for a LIT file.
+        d.addCallback(lambda ign: self._check_file(u"short", "test"))
+
+        # Write to the same file again with different data.
+        d.addCallback(lambda ign: self._check_file(u"short", "different"))
+
+        # Test that temporary files are not uploaded.
+        d.addCallback(lambda ign: self._check_file(u"tempfile", "test", temporary=True))
+
+        # Test that we tolerate creation of a subdirectory.
+        d.addCallback(lambda ign: os.mkdir(os.path.join(self.local_dir, u"directory")))
+
+        # Write something longer, and also try to test a Unicode name if the fs can represent it.
+        name_u = self.unicode_or_fallback(u"l\u00F8ng", u"long")
+        d.addCallback(lambda ign: self._check_file(name_u, "test"*100))
+
+        # TODO: test that causes an upload failure.
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_failed'), 0))
+
+        d.addBoth(self._cleanup)
+        return d
+
+    def _check_file(self, name_u, data, temporary=False):
+        previously_uploaded = self._get_count('magic_folder.objects_succeeded')
+        previously_disappeared = self._get_count('magic_folder.objects_disappeared')
+
+        d = defer.Deferred()
+
+        # Note: this relies on the fact that we only get one IN_CLOSE_WRITE notification per file
+        # (otherwise we would get a defer.AlreadyCalledError). Should we be relying on that?
+        self.magicfolder.set_processed_callback(d.callback)
+
+        path_u = abspath_expanduser_unicode(name_u, base=self.local_dir)
+        path = to_filepath(path_u)
+
+        # We don't use FilePath.setContent() here because it creates a temporary file that
+        # is renamed into place, which causes events that the test is not expecting.
+        f = open(path_u, "wb")
+        try:
+            if temporary and sys.platform != "win32":
+                os.unlink(path_u)
+            f.write(data)
+        finally:
+            f.close()
+        if temporary and sys.platform == "win32":
+            os.unlink(path_u)
+            self.notify(path, self.inotify.IN_DELETE)
+        fileutil.flush_volume(path_u)
+        self.notify(path, self.inotify.IN_CLOSE_WRITE)
+
+        if temporary:
+            d.addCallback(lambda ign: self.shouldFail(NoSuchChildError, 'temp file not uploaded', None,
+            self.upload_dirnode.get, name_u))
+            d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_disappeared'),
+                                                                 previously_disappeared + 1))
+        else:
+            d.addCallback(lambda ign: self.upload_dirnode.get(name_u))
+            d.addCallback(download_to_data)
+            d.addCallback(lambda actual_data: self.failUnlessReallyEqual(actual_data, data))
+            d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'),
+                                                                 previously_uploaded + 1))
+
+        d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
+        return d
+
+
+class MockTest(MagicFolderTestMixin, unittest.TestCase):
+    """This can run on any platform, and even if twisted.internet.inotify can't be imported."""
+
+    def setUp(self):
+        MagicFolderTestMixin.setUp(self)
+        self.inotify = fake_inotify
+
+    def notify(self, path, mask):
+        self.magicfolder._notifier.event(path, mask)
+
+    def test_errors(self):
+        self.set_up_grid()
+
+        errors_dir = abspath_expanduser_unicode(u"errors_dir", base=self.basedir)
+        os.mkdir(errors_dir)
+        not_a_dir = abspath_expanduser_unicode(u"NOT_A_DIR", base=self.basedir)
+        fileutil.write(not_a_dir, "")
+        magicfolderdb = abspath_expanduser_unicode(u"magicfolderdb", base=self.basedir)
+        doesnotexist  = abspath_expanduser_unicode(u"doesnotexist", base=self.basedir)
+
+        client = self.g.clients[0]
+        d = client.create_dirnode()
+        def _check_errors(n):
+            self.failUnless(IDirectoryNode.providedBy(n))
+            upload_dircap = n.get_uri()
+            readonly_dircap = n.get_readonly_uri()
+
+            self.shouldFail(AssertionError, 'nonexistent local.directory', 'there is no directory',
+                            MagicFolder, client, upload_dircap, '', doesnotexist, magicfolderdb, inotify=fake_inotify)
+            self.shouldFail(AssertionError, 'non-directory local.directory', 'is not a directory',
+                            MagicFolder, client, upload_dircap, '', not_a_dir, magicfolderdb, inotify=fake_inotify)
+            self.shouldFail(AssertionError, 'bad upload.dircap', 'does not refer to a directory',
+                            MagicFolder, client, 'bad', '', errors_dir, magicfolderdb, inotify=fake_inotify)
+            self.shouldFail(AssertionError, 'non-directory upload.dircap', 'does not refer to a directory',
+                            MagicFolder, client, 'URI:LIT:foo', '', errors_dir, magicfolderdb, inotify=fake_inotify)
+            self.shouldFail(AssertionError, 'readonly upload.dircap', 'is not a writecap to a directory',
+                            MagicFolder, client, readonly_dircap, '', errors_dir, magicfolderdb, inotify=fake_inotify)
+
+            def _not_implemented():
+                raise NotImplementedError("blah")
+            self.patch(magic_folder, 'get_inotify_module', _not_implemented)
+            self.shouldFail(NotImplementedError, 'unsupported', 'blah',
+                            MagicFolder, client, upload_dircap, '', errors_dir, magicfolderdb)
+        d.addCallback(_check_errors)
+        return d
+
+
+class RealTest(MagicFolderTestMixin, unittest.TestCase):
+    """This is skipped unless both Twisted and the platform support inotify."""
+
+    def setUp(self):
+        MagicFolderTestMixin.setUp(self)
+        self.inotify = magic_folder.get_inotify_module()
+
+    def notify(self, path, mask):
+        # Writing to the filesystem causes the notification.
+        pass
+
+try:
+    magic_folder.get_inotify_module()
+except NotImplementedError:
+    RealTest.skip = "Magic Folder support can only be tested for-real on an OS that supports inotify or equivalent."