From: Daira Hopwood Date: Thu, 2 Apr 2015 14:26:25 +0000 (+0100) Subject: Collaborative design from Daira and David. X-Git-Tag: allmydata-tahoe-1.10.1a1~48 X-Git-Url: https://git.rkrishnan.org/%5B/%5D%20/uri/%22doc.html/vdrive?a=commitdiff_plain;h=14c3bfd12e0b464595a55dccd62442bca6271136;p=tahoe-lafs%2Ftahoe-lafs.git Collaborative design from Daira and David. Signed-off-by: Daira Hopwood --- diff --git a/docs/proposed/magic-folder/filesystem-integration.rst b/docs/proposed/magic-folder/filesystem-integration.rst new file mode 100644 index 00000000..b2642f03 --- /dev/null +++ b/docs/proposed/magic-folder/filesystem-integration.rst @@ -0,0 +1,99 @@ +Magic Folders local filesystem integration design +================================================= + +*Scope* + +This document describes how to integrate the local filesystem with Magic +Folders in an efficient and reliable manner. For now we ignore Remote to +Local synchronization; the design and implementation of this is scheduled +for a later time. We also ignore multiple writers for the same Magic +Folder, which may or may not be supported in future. The design here will +be updated to account for those features in later Objectives. Objective 3 +may require modifying the database schema or operation, and Objective 5 +may modify the User interface. + + +*Local scanning and database* + +When a Magic-Folder-enabled node starts up, it scans all directories +under the local directory and adds every file to a first-in first-out +"scan queue". When processing the scan queue, redundant uploads are +avoided by using the same mechanism the Tahoe backup command uses: we +keep track of previous uploads by recording each file's metadata such as +size, CTIME and MTIME. This information is stored in a database referred +to from now on as the magic folder db. Using this recorded state, we +ensure that when Magic Folder is subsequently started, the local +directory tree can be scanned quickly by comparing current filesystem +metadata with the previously recorded metadata. Each file referenced in +the scan queue is uploaded only if its metadata differs at the time it is +processed. If a change event is detected for a file that is already +queued (and therefore will be processed later), the redundant event is +ignored. + +To implement the magic folder db, we will use an SQLite schema that +initially is the existing Tahoe-LAFS backup schema. This schema may +change in later objectives; this will cause no backward compatibility +problems, because this new feature will be developed on a branch that +makes no compatibility guarantees. However we will have a separate SQLite +database file and separate mutex lock just for Magic Folder. This avoids +usability problems related to mutual exclusion. (If a single file and +lock were used, a backup would block Magic Folder updates for a long +time, and a user would not be able to tell when backups are possible +because Magic Folder acquires a lock at arbitrary times.) + + +*Eventual consistency property* + +It is not possible to prevent local writes to a file while it is being +read in order to upload it. Such writes will result in temporary +inconsistency. Eventual consistency is reached when the queue of pending +uploads is empty. That is, a consistent snapshot will be achieved +eventually when local writes to the target folder cease for a +sufficiently long period of time. + + +*Detecting filesystem changes* + +For the Linux implementation we will use the inotify Linux kernel +subsystem to gather events on the local Magic Folder directory tree. This +implementation was already present in Tahoe-LAFS 1.9.0, but needs to be +changed to gather directory creation and move events, as well as events +indicating that a file has been written. + +For the Windows implementation we will use the ``ReadDirectoryChangesW`` +Win32 API. The prototype implementation simulates the inotify API in +terms of ``ReadDirectoryChangesW``, allowing most of the code to be +shared across platforms. + +When we detect the creation of a new directory below the local Magic +Folder directory, we create it in the Tahoe-LAFS filesystem, and also +scan the new local directory for new files. This scan is necessary to +avoid missing events for creation of files in a new directory before it +can be watched, and to correctly handle cases where an existing directory +is moved to be under the local Magic Folder directory. + + +*User interface* + +The Magic Folders local filesystem integration will initially have a +provisional configuration file-based interface that may not be ideal from +a usability perspective. Creating our local filesystem integration in +this manner will allow us to use and test it indepently of the rest of +the Magic Folder software components. We will focus greater attention on +user interface design as a later milestone in our development roadmap. + +The configuration file, ``tahoe.cfg``, must define a target local +directory to be synchronized. Provisionally, this configuration will +replace the current "drop-upload" section:: + + [magic_folders] + enabled = true + local.directory = "/home/human" + +When a filesystem directory is first "added" to Magic Folders, the user +needs to create the remote Tahoe-LAFS directory using ``tahoe mkdir``, +and configure the Magic-Folder-enabled node with its URI (e.g. by putting +it in a file ``private/magic_folder_dircap``). If there are existing +files in the local directory, they will be uploaded as a result of the +initial scan described earlier. +