--- /dev/null
+
+Welcome to the tahoe fuse interface prototype!
+
+
+Dependencies:
+
+In addition to a working tahoe installation, this interface depends
+on the python-fuse interface. This package is available on Ubuntu
+systems as "python-fuse". It is only known to work with ubuntu
+package version "2.5-5build1". The latest ubuntu package (version
+"1:0.2-pre3-3") appears to not work currently.
+
+Unfortunately this package appears poorly maintained (notice the wildy
+different version strings and changing API semantics), so if you know
+of a good replacement pythonic fuse interface, please let tahoe-dev know
+about it!
+
+
+Configuration:
+
+Currently tahoe-fuse.py uses the same ~/.tahoe/private/root_dir.cap
+file (which is also the CLI default). This is not configurable yet.
+Place a directory cap in this file. (Hint: If you can run "tahoe ls"
+and see a directory listing, this file is properly configured.)
+
+
+Commandline:
+
+The usage is "tahoe-fuse.py <mountpoint>". The mount point needs to
+be an existing directory which should be empty. (If it's not empty
+the contents will be safe, but unavailable while the tahoe-fuse.py
+process is mounted there.)
+
+
+Usage:
+
+To use the interface, use other programs to poke around the
+mountpoint. You should be able to see the same contents as you would
+by using the CLI or WUI for the same directory cap.
+
+
+Runtime Behavior Notes:
+
+Read-only:
+Only reading a tahoe grid is supported, which is reflected in
+the permission modes. With Tahoe 0.7.0, write access should be easier
+to implement, but is not yet present.
+
+In-Memory File Caching:
+Currently requesting a particular file for read causes the entire file to
+be retrieved into tahoe-fuse.py memory before the read operation returns!
+This caching is reused for subsequent reads. Beware large files.
+When transitioning to a finer-grained fuse api, this caching should be
+replaced with straight-forward calls to the wapi. In my opinion, the
+Tahoe node should do all the caching tricks, so that extensions such as
+tahoe-fuse.py can be simple and thin.
+
+Backgrounding Behavior:
+When using the 2.5-5build1 ubuntu package, and no other arguments
+besides a mountpoint to tahoe-fuse.py, the process should remain in
+the foreground and print debug information. Other python-fuse
+versions appear to alter this behavior and may fork the process to
+the background and obscure the log output. Bonus points to whomever
+discovers the fate of these poor log messages in this case.
+
+"Investigative Logging":
+This prototype is designed to aide in further fuse development, so
+currently *every* fuse interface call figures out the process from
+which the file system request originates, then it figures out that
+processes commandline (this uses the /proc file system). This is handy
+for interactive inspection of what kinds of behavior invokes which
+file system operations, but may not work for you. To disable this
+inspection, edit the source and comment out all of the "@debugcall"
+[FIXME: double check python ref name] method decorators by inserting a
+'#' so it looks like "#@debugcall" (without quotes).
+
+Not-to-spec:
+The current version was not implemented according to any spec and
+makes quite a few dubious "guesses" for what data to pass the fuse
+interface. You may see bizarre values, which may potentialy confuse
+any processes visiting the files under the mount point.
+
+Serial, blocking operations:
+Most fuse operations result in one or more http calls to the WAPI.
+These are serial and blocking (at least for the tested python-fuse
+version 2.5-5build1), so access to this file system is quite
+inefficient.
+
+
+Good luck!
--- /dev/null
+#! /usr/bin/env python
+'''
+Tahoe thin-client fuse module.
+
+See the accompanying README for configuration/usage details.
+
+Goals:
+
+- Delegate to Tahoe webapi as much as possible.
+- Thin rather than clever. (Even when that means clunky.)
+
+
+Warts:
+
+- Reads cache entire file contents, violating the thinness goal. Can we GET spans of files?
+- Single threaded.
+
+
+Road-map:
+1. Add unit tests where possible with little code modification.
+2. Make unit tests pass for a variety of python-fuse module versions.
+3. Modify the design to make possible unit test coverage of larger portions of code.
+
+Wishlist:
+- Perhaps integrate cli aliases or root_dir.cap.
+- Research pkg_resources; see if it can replace the try-import-except-import-error pattern.
+- Switch to logging instead of homebrew logging.
+'''
+
+
+#import bindann
+#bindann.install_exception_handler()
+
+import sys, stat, os, errno, urllib, time
+
+try:
+ import simplejson
+except ImportError, e:
+ raise SystemExit('''\
+Could not import simplejson, which is bundled with Tahoe. Please
+update your PYTHONPATH environment variable to include the tahoe
+"support/lib/python<VERSION>/site-packages" directory.
+
+If you run this from the Tahoe source directory, use this command:
+PYTHONPATH="$PYTHONPATH:./support/lib/python%d.%d/site-packages/" python %s
+''' % (sys.version_info[:2] + (' '.join(sys.argv),)))
+
+
+try:
+ import fuse
+except ImportError, e:
+ raise SystemExit('''\
+Could not import fuse, the pythonic fuse bindings. This dependency
+of tahoe-fuse.py is *not* bundled with tahoe. Please install it.
+On debian/ubuntu systems run: sudo apt-get install python-fuse
+''')
+
+# FIXME: Check for non-working fuse versions here.
+# FIXME: Make this work for all common python-fuse versions.
+
+# FIXME: Currently uses the old, silly path-based (non-stateful) interface:
+fuse.fuse_python_api = (0, 1) # Use the silly path-based api for now.
+
+
+### Config:
+TahoeConfigDir = '~/.tahoe'
+MagicDevNumber = 42
+UnknownSize = -1
+
+
+def main():
+ basedir = os.path.expanduser(TahoeConfigDir)
+
+ for i, arg in enumerate(sys.argv):
+ if arg == '--basedir':
+ try:
+ basedir = sys.argv[i+1]
+ sys.argv[i:i+2] = []
+ except IndexError:
+ sys.argv = [sys.argv[0], '--help']
+
+
+ log_init(basedir)
+ log('Commandline: %r', sys.argv)
+
+ fs = TahoeFS(basedir)
+ fs.main()
+
+
+### Utilities for debug:
+_logfile = None # Private to log* functions.
+
+def log_init(confdir):
+ global _logfile
+
+ logpath = os.path.join(confdir, 'logs', 'tahoe_fuse.log')
+ _logfile = open(logpath, 'a')
+ log('Log opened at: %s\n', time.strftime('%Y-%m-%d %H:%M:%S'))
+
+
+def log(msg, *args):
+ _logfile.write((msg % args) + '\n')
+ _logfile.flush()
+
+
+def trace_calls(m):
+ def dbmeth(self, *a, **kw):
+ pid = self.GetContext()['pid']
+ log('[%d %r]\n%s%r%r', pid, get_cmdline(pid), m.__name__, a, kw)
+ try:
+ r = m(self, *a, **kw)
+ if (type(r) is int) and (r < 0):
+ log('-> -%s\n', errno.errorcode[-r],)
+ else:
+ repstr = repr(r)[:256]
+ log('-> %s\n', repstr)
+ return r
+ except:
+ sys.excepthook(*sys.exc_info())
+
+ return dbmeth
+
+
+def get_cmdline(pid):
+ f = open('/proc/%d/cmdline' % pid, 'r')
+ args = f.read().split('\0')
+ f.close()
+ assert args[-1] == ''
+ return args[:-1]
+
+
+class SystemError (Exception):
+ def __init__(self, eno):
+ self.eno = eno
+ Exception.__init__(self, errno.errorcode[eno])
+
+ @staticmethod
+ def wrap_returns(meth):
+ def wrapper(*args, **kw):
+ try:
+ return meth(*args, **kw)
+ except SystemError, e:
+ return -e.eno
+ wrapper.__name__ = meth.__name__
+ return wrapper
+
+
+### Heart of the Matter:
+class TahoeFS (fuse.Fuse):
+ def __init__(self, confdir):
+ log('Initializing with confdir = %r', confdir)
+ fuse.Fuse.__init__(self)
+ self.confdir = confdir
+
+ self.flags = 0 # FIXME: What goes here?
+ self.multithreaded = 0
+
+ # silly path-based file handles.
+ self.filecontents = {} # {path -> contents}
+
+ self._init_url()
+ self._init_rootdir()
+
+ def _init_url(self):
+ f = open(os.path.join(self.confdir, 'webport'), 'r')
+ contents = f.read()
+ f.close()
+
+ fields = contents.split(':')
+ proto, port = fields[:2]
+ assert proto == 'tcp'
+ port = int(port)
+ self.url = 'http://localhost:%d' % (port,)
+
+ def _init_rootdir(self):
+ # For now we just use the same default as the CLI:
+ rootdirfn = os.path.join(self.confdir, 'private', 'root_dir.cap')
+ try:
+ f = open(rootdirfn, 'r')
+ cap = f.read().strip()
+ f.close()
+ except EnvironmentError, le:
+ # FIXME: This user-friendly help message may be platform-dependent because it checks the exception description.
+ if le.args[1].find('No such file or directory') != -1:
+ raise SystemExit('%s requires a directory capability in %s, but it was not found.\nPlease see "The CLI" in "docs/using.html".\n' % (sys.argv[0], rootdirfn))
+ else:
+ raise le
+
+ self.rootdir = TahoeDir(self.url, canonicalize_cap(cap))
+
+ def _get_node(self, path):
+ assert path.startswith('/')
+ if path == '/':
+ return self.rootdir.resolve_path([])
+ else:
+ parts = path.split('/')[1:]
+ return self.rootdir.resolve_path(parts)
+
+ def _get_contents(self, path):
+ node = self._get_node(path)
+ contents = node.open().read()
+ self.filecontents[path] = contents
+ return contents
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def getattr(self, path):
+ node = self._get_node(path)
+ return node.getattr()
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def getdir(self, path):
+ """
+ return: [(name, typeflag), ... ]
+ """
+ node = self._get_node(path)
+ return node.getdir()
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def mythread(self):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def chmod(self, path, mode):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def chown(self, path, uid, gid):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def fsync(self, path, isFsyncFile):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def link(self, target, link):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def mkdir(self, path, mode):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def mknod(self, path, mode, dev_ignored):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def open(self, path, mode):
+ IgnoredFlags = os.O_RDONLY | os.O_NONBLOCK | os.O_SYNC | os.O_LARGEFILE
+ # Note: IgnoredFlags are all ignored!
+ for fname in dir(os):
+ if fname.startswith('O_'):
+ flag = getattr(os, fname)
+ if flag & IgnoredFlags:
+ continue
+ elif mode & flag:
+ log('Flag not supported: %s', fname)
+ raise SystemError(errno.ENOSYS)
+
+ self._get_contents(path)
+ return 0
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def read(self, path, length, offset):
+ return self._get_contents(path)[offset:length]
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def release(self, path):
+ del self.filecontents[path]
+ return 0
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def readlink(self, path):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def rename(self, oldpath, newpath):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def rmdir(self, path):
+ return -errno.ENOSYS
+
+ #@trace_calls
+ @SystemError.wrap_returns
+ def statfs(self):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def symlink ( self, targetPath, linkPath ):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def truncate(self, path, size):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def unlink(self, path):
+ return -errno.ENOSYS
+
+ @trace_calls
+ @SystemError.wrap_returns
+ def utime(self, path, times):
+ return -errno.ENOSYS
+
+
+class TahoeNode (object):
+ NextInode = 0
+
+ @staticmethod
+ def make(baseurl, uri):
+ typefield = uri.split(':', 2)[1]
+ # FIXME: is this check correct?
+ if uri.find('URI:DIR2') != -1:
+ return TahoeDir(baseurl, uri)
+ else:
+ return TahoeFile(baseurl, uri)
+
+ def __init__(self, baseurl, uri):
+ self.burl = baseurl
+ self.uri = uri
+ self.fullurl = '%s/uri/%s' % (self.burl, self.uri)
+ self.inode = TahoeNode.NextInode
+ TahoeNode.NextInode += 1
+
+ def getattr(self):
+ """
+ - st_mode (protection bits)
+ - st_ino (inode number)
+ - st_dev (device)
+ - st_nlink (number of hard links)
+ - st_uid (user ID of owner)
+ - st_gid (group ID of owner)
+ - st_size (size of file, in bytes)
+ - st_atime (time of most recent access)
+ - st_mtime (time of most recent content modification)
+ - st_ctime (platform dependent; time of most recent metadata change on Unix,
+ or the time of creation on Windows).
+ """
+ # FIXME: Return metadata that isn't completely fabricated.
+ return (self.get_mode(),
+ self.inode,
+ MagicDevNumber,
+ self.get_linkcount(),
+ os.getuid(),
+ os.getgid(),
+ self.get_size(),
+ 0,
+ 0,
+ 0)
+
+ def get_metadata(self):
+ f = self.open('?t=json')
+ json = f.read()
+ f.close()
+ return simplejson.loads(json)
+
+ def open(self, postfix=''):
+ url = self.fullurl + postfix
+ log('*** Fetching: %r', url)
+ return urllib.urlopen(url)
+
+
+class TahoeFile (TahoeNode):
+ def __init__(self, baseurl, uri):
+ #assert uri.split(':', 2)[1] in ('CHK', 'LIT'), `uri` # fails as of 0.7.0
+ TahoeNode.__init__(self, baseurl, uri)
+
+ # nonfuse:
+ def get_mode(self):
+ return stat.S_IFREG | 0400 # Read only regular file.
+
+ def get_linkcount(self):
+ return 1
+
+ def get_size(self):
+ rawsize = self.get_metadata()[1]['size']
+ if type(rawsize) is not int: # FIXME: What about sizes which do not fit in python int?
+ assert rawsize == u'?', `rawsize`
+ return UnknownSize
+ else:
+ return rawsize
+
+ def resolve_path(self, path):
+ assert path == []
+ return self
+
+
+class TahoeDir (TahoeNode):
+ def __init__(self, baseurl, uri):
+ TahoeNode.__init__(self, baseurl, uri)
+
+ self.mode = stat.S_IFDIR | 0500 # Read only directory.
+
+ # FUSE:
+ def getdir(self):
+ d = [('.', self.get_mode()), ('..', self.get_mode())]
+ for name, child in self.get_children().items():
+ if name: # Just ignore this crazy case!
+ d.append((name, child.get_mode()))
+ return d
+
+ # nonfuse:
+ def get_mode(self):
+ return stat.S_IFDIR | 0500 # Read only directory.
+
+ def get_linkcount(self):
+ return len(self.getdir())
+
+ def get_size(self):
+ return 2 ** 12 # FIXME: What do we return here? len(self.get_metadata())
+
+ def resolve_path(self, path):
+ assert type(path) is list
+
+ if path:
+ head = path[0]
+ child = self.get_child(head)
+ return child.resolve_path(path[1:])
+ else:
+ return self
+
+ def get_child(self, name):
+ c = self.get_children()
+ return c[name]
+
+ def get_children(self):
+ flag, md = self.get_metadata()
+ assert flag == 'dirnode'
+
+ c = {}
+ for name, (childflag, childmd) in md['children'].items():
+ if childflag == 'dirnode':
+ cls = TahoeDir
+ else:
+ cls = TahoeFile
+
+ c[str(name)] = cls(self.burl, childmd['ro_uri'])
+ return c
+
+
+def canonicalize_cap(cap):
+ cap = urllib.unquote(cap)
+ i = cap.find('URI:')
+ assert i != -1, 'A cap must contain "URI:...", but this does not: ' + cap
+ return cap[i:]
+
+
+if __name__ == '__main__':
+ main()
+
--- /dev/null
+This announcement is archived in the tahoe-dev mailing list archive:
+
+http://allmydata.org/pipermail/tahoe-dev/2008-March/000465.html
+
+[tahoe-dev] Another FUSE interface
+Armin Rigo arigo at tunes.org
+Sat Mar 29 04:35:36 PDT 2008
+
+ * Previous message: [tahoe-dev] announcing allmydata.org "Tahoe", v1.0
+ * Next message: [tahoe-dev] convergent encryption reconsidered -- salting and key-strengthening
+ * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
+
+Hi all,
+
+I implemented for fun another Tahoe-to-FUSE interface using my own set
+of FUSE bindings. If you are interested, you can check out the
+following subversion directory:
+
+ http://codespeak.net/svn/user/arigo/hack/pyfuse
+
+tahoe.py is a 100-lines, half-an-hour-job interface to Tahoe, limited to
+read-only at the moment. The rest of the directory contains PyFuse, and
+many other small usage examples. PyFuse is a pure Python FUSE daemon
+(no messy linking issues, no dependencies).
+
+
+A bientot,
+
+Armin Rigo
+
+ * Previous message: [tahoe-dev] announcing allmydata.org "Tahoe", v1.0
+ * Next message: [tahoe-dev] convergent encryption reconsidered -- salting and key-strengthening
+ * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
+
+More information about the tahoe-dev mailing list
+
--- /dev/null
+#! /usr/bin/env python
+'''
+Unit and system tests for tahoe-fuse.
+'''
+
+# Note: It's always a SetupFailure, not a TestFailure if a webapi
+# operation fails, because this does not indicate a fuse interface
+# failure.
+
+# TODO: Unmount after tests regardless of failure or success!
+
+# TODO: Test mismatches between tahoe and fuse/posix. What about nodes
+# with crazy names ('\0', unicode, '/', '..')? Huuuuge files?
+# Huuuuge directories... As tahoe approaches production quality, it'd
+# be nice if the fuse interface did so also by hardening against such cases.
+
+# FIXME: Only create / launch necessary nodes. Do we still need an introducer and three nodes?
+
+# FIXME: This framework might be replaceable with twisted.trial,
+# especially the "layer" design, which is a bit cumbersome when
+# using recursion to manage multiple clients.
+
+# FIXME: Identify all race conditions (hint: starting clients, versus
+# using the grid fs).
+
+import sys, os, shutil, unittest, subprocess
+import tempfile, re, time, signal, random, httplib
+import traceback
+
+# Import fuse implementations:
+FuseDir = os.path.join('.', 'contrib', 'fuse')
+if not os.path.isdir(FuseDir):
+ raise SystemExit('''
+Could not find directory "%s". Please run this script from the tahoe
+source base directory.
+''' % (FuseDir,))
+
+sys.path.append(os.path.join(FuseDir, 'impl_a'))
+import tahoe_fuse as impl_a
+
+sys.path.append(os.path.join(FuseDir, 'impl_b'))
+import pyfuse.tahoe as impl_b
+
+
+### Main flow control:
+def main(args = sys.argv):
+ target = 'all'
+ if len(args) > 1:
+ target = args.pop(1)
+
+ if target not in ('all', 'unit', 'system'):
+ raise SystemExit(Usage)
+
+ if target in ('all', 'unit'):
+ run_unit_tests()
+
+ if target in ('all', 'system'):
+ run_system_test()
+
+
+def run_unit_tests():
+ print 'Running Unit Tests.'
+ try:
+ unittest.main()
+ except SystemExit, se:
+ pass
+ print 'Unit Tests complete.\n'
+
+
+def run_system_test():
+ SystemTest().run()
+
+
+### System Testing:
+class SystemTest (object):
+ def __init__(self):
+ # These members represent configuration:
+ self.fullcleanup = False # FIXME: Make this a commandline option.
+
+ # These members represent test state:
+ self.cliexec = None
+ self.testroot = None
+
+ # This test state is specific to the first client:
+ self.port = None
+ self.clientbase = None
+
+ ## Top-level flow control:
+ # These "*_layer" methods call eachother in a linear fashion, using
+ # exception unwinding to do cleanup properly. Each "layer" invokes
+ # a deeper layer, and each layer does its own cleanup upon exit.
+
+ def run(self, fullcleanup = False):
+ '''
+ If full_cleanup, delete all temporary state.
+ Else: If there is an error do not delete basedirs.
+
+ Set to False if you wish to analyze a failure.
+ '''
+ self.fullcleanup = fullcleanup
+ print '\n*** Setting up system tests.'
+ try:
+ results = self.init_cli_layer()
+ print '\n*** System Tests complete:'
+ for implpath, failures, total in reuslts:
+ print 'Implementation %r: %d failed out of %d.' % (implpath, failures, total)
+ except self.SetupFailure, sfail:
+ print
+ print sfail
+ print '\n*** System Tests were not successfully completed.'
+
+ def init_cli_layer(self):
+ '''This layer finds the appropriate tahoe executable.'''
+ self.cliexec = os.path.join('.', 'bin', 'tahoe')
+ version = self.run_tahoe('--version')
+ print 'Using %r with version:\n%s' % (self.cliexec, version.rstrip())
+
+ return self.create_testroot_layer()
+
+ def create_testroot_layer(self):
+ print 'Creating test base directory.'
+ self.testroot = tempfile.mkdtemp(prefix='tahoe_fuse_test_')
+ try:
+ return self.launch_introducer_layer()
+ finally:
+ if self.fullcleanup:
+ print 'Cleaning up test root directory.'
+ try:
+ shutil.rmtree(self.testroot)
+ except Exception, e:
+ print 'Exception removing test root directory: %r' % (self.testroot, )
+ print 'Ignoring cleanup exception: %r' % (e,)
+ else:
+ print 'Leaving test root directory: %r' % (self.testroot, )
+
+
+ def launch_introducer_layer(self):
+ print 'Launching introducer.'
+ introbase = os.path.join(self.testroot, 'introducer')
+
+ # NOTE: We assume if tahoe exits with non-zero status, no separate
+ # tahoe child process is still running.
+ createoutput = self.run_tahoe('create-introducer', '--basedir', introbase)
+
+ self.check_tahoe_output(createoutput, ExpectedCreationOutput, introbase)
+
+ startoutput = self.run_tahoe('start', '--basedir', introbase)
+ try:
+ self.check_tahoe_output(startoutput, ExpectedStartOutput, introbase)
+
+ return self.launch_clients_layer(introbase)
+
+ finally:
+ print 'Stopping introducer node.'
+ self.stop_node(introbase)
+
+ TotalClientsNeeded = 3
+ def launch_clients_layer(self, introbase, clientnum = 0):
+ if clientnum >= self.TotalClientsNeeded:
+ return self.create_test_dirnode_layer()
+
+ tmpl = 'Launching client %d of %d.'
+ print tmpl % (clientnum,
+ self.TotalClientsNeeded)
+
+ base = os.path.join(self.testroot, 'client_%d' % (clientnum,))
+
+ output = self.run_tahoe('create-client', '--basedir', base)
+ self.check_tahoe_output(output, ExpectedCreationOutput, base)
+
+ webportpath = os.path.join(base, 'webport')
+ if clientnum == 0:
+ # The first client is special:
+ self.clientbase = base
+ self.port = random.randrange(1024, 2**15)
+
+ f = open(webportpath, 'w')
+ f.write('tcp:%d:interface=127.0.0.1\n' % self.port)
+ f.close()
+ else:
+ os.remove(webportpath)
+
+
+ introfurl = os.path.join(introbase, 'introducer.furl')
+
+ self.polling_operation(lambda : os.path.isfile(introfurl))
+ shutil.copy(introfurl, base)
+
+ # NOTE: We assume if tahoe exist with non-zero status, no separate
+ # tahoe child process is still running.
+ startoutput = self.run_tahoe('start', '--basedir', base)
+ try:
+ self.check_tahoe_output(startoutput, ExpectedStartOutput, base)
+
+ return self.launch_clients_layer(introbase, clientnum+1)
+
+ finally:
+ print 'Stopping client node %d.' % (clientnum,)
+ self.stop_node(base)
+
+ def create_test_dirnode_layer(self):
+ print 'Creating test dirnode.'
+
+ cap = self.create_dirnode()
+
+ f = open(os.path.join(self.clientbase, 'private', 'root_dir.cap'), 'w')
+ f.write(cap)
+ f.close()
+
+ return self.mount_fuse_layer(cap)
+
+ def mount_fuse_layer(self, fusebasecap):
+ mpbase = os.path.join(self.testroot, 'mountpoint')
+ os.mkdir(mpbase)
+
+ results = []
+
+ # Mount and test each implementation:
+ for implnum, implmod in enumerate([impl_a, impl_b]):
+ implpath = implmod.__file__
+ print '\n*** Mounting and Testing implementation #%d: %r' % (implnum, implpath)
+
+ print 'Mounting implementation #%d: %r' % (implnum, implpath)
+ mountpath = os.path.join(mpbase, 'impl_%d' % (implnum,))
+ os.mkdir(mountpath)
+
+ exitcode, output = gather_output(['python',
+ implpath,
+ mountpath,
+ '--basedir', self.clientbase])
+
+ if exitcode != 0 or output:
+ tmpl = '%r failed to launch:\n'
+ tmpl += 'Exit Status: %r\n'
+ tmpl += 'Output:\n%s\n'
+ raise self.SetupFailure(tmpl, implpath, exitcode, output)
+
+ try:
+ failures, total = self.run_test_layer(fusebasecap, mountpath)
+ print '\n*** Test results for implementation %r: %d failed out of %d.' % (implpath, failures, total)
+ results.append((implpath, failures, total))
+
+ finally:
+ print 'Unmounting implementation #%d' % (implnum,)
+ args = ['fusermount', '-u', mountpath]
+ ec, out = gather_output(args)
+ if exitcode != 0 or output:
+ tmpl = 'fusermount failed to unmount:\n'
+ tmpl += 'Arguments: %r\n'
+ tmpl += 'Exit Status: %r\n'
+ tmpl += 'Output:\n%s\n'
+ raise self.SetupFailure(tmpl, args, ec, out)
+
+ return results
+
+ def run_test_layer(self, fbcap, mp):
+ total = failures = 0
+ testnames = [n for n in sorted(dir(self)) if n.startswith('test_')]
+ for name in testnames:
+ total += 1
+ print '\n*** Running test #%d: %s' % (total, name)
+ try:
+ testcap = self.create_dirnode()
+ self.attach_node(fbcap, testcap, name)
+
+ method = getattr(self, name)
+ method(testcap, testdir = os.path.join(mp, name))
+ print 'Test succeeded.'
+ except self.TestFailure, f:
+ print f
+ failures += 1
+ except:
+ print 'Error in test code... Cleaning up.'
+ raise
+
+ return (failures, total)
+
+
+ # Tests:
+ def test_directory_existence(self, testcap, testdir):
+ if not os.path.isdir(testdir):
+ raise self.TestFailure('Attached test directory not found: %r', testdir)
+
+ def test_empty_directory_listing(self, testcap, testdir):
+ listing = os.listdir(testdir)
+ if listing:
+ raise self.TestFailure('Expected empty directory, found: %r', listing)
+
+ def test_directory_listing(self, testcap, testdir):
+ names = []
+ filesizes = {}
+
+ for i in range(3):
+ fname = 'file_%d' % (i,)
+ names.append(fname)
+ body = 'Hello World #%d!' % (i,)
+ filesizes[fname] = len(body)
+
+ cap = self.webapi_call('PUT', '/uri', body)
+ self.attach_node(testcap, cap, fname)
+
+ dname = 'dir_%d' % (i,)
+ names.append(dname)
+
+ cap = self.create_dirnode()
+ self.attach_node(testcap, cap, dname)
+
+ names.sort()
+
+ listing = os.listdir(testdir)
+ listing.sort()
+ if listing != names:
+ tmpl = 'Expected directory list containing %r but fuse gave %r'
+ raise self.TestFailure(tmpl, names, listing)
+
+ for file, size in filesizes.items():
+ st = os.stat(os.path.join(testdir, file))
+ if st.st_size != size:
+ tmpl = 'Expected %r size of %r but fuse returned %r'
+ raise self.TestFailure(tmpl, file, size, st.st_size)
+
+ def test_file_contents(self, testcap, testdir):
+ name = 'hw.txt'
+ body = 'Hello World!'
+
+ cap = self.webapi_call('PUT', '/uri', body)
+ self.attach_node(testcap, cap, name)
+
+ path = os.path.join(testdir, name)
+ try:
+ found = open(path, 'r').read()
+ except Exception, err:
+ tmpl = 'Could not read file contents of %r: %r'
+ raise self.TestFailure(tmpl, path, err)
+
+ if found != body:
+ tmpl = 'Expected file contents %r but found %r'
+ raise self.TestFailure(tmpl, body, found)
+
+
+ # Utilities:
+ def run_tahoe(self, *args):
+ realargs = ('tahoe',) + args
+ status, output = gather_output(realargs, executable=self.cliexec)
+ if status != 0:
+ tmpl = 'The tahoe cli exited with nonzero status.\n'
+ tmpl += 'Executable: %r\n'
+ tmpl += 'Command arguments: %r\n'
+ tmpl += 'Exit status: %r\n'
+ tmpl += 'Output:\n%s\n[End of tahoe output.]\n'
+ raise self.SetupFailure(tmpl,
+ self.cliexec,
+ realargs,
+ status,
+ output)
+ return output
+
+ def check_tahoe_output(self, output, expected, expdir):
+ m = re.match(expected, output, re.M)
+ if m is None:
+ tmpl = 'The output of tahoe did not match the expectation:\n'
+ tmpl += 'Expected regex: %s\n'
+ tmpl += 'Actual output: %r\n'
+ self.warn(tmpl, expected, output)
+
+ elif expdir != m.group('path'):
+ tmpl = 'The output of tahoe refers to an unexpected directory:\n'
+ tmpl += 'Expected directory: %r\n'
+ tmpl += 'Actual directory: %r\n'
+ self.warn(tmpl, expdir, m.group(1))
+
+ def stop_node(self, basedir):
+ try:
+ self.run_tahoe('stop', '--basedir', basedir)
+ except Exception, e:
+ print 'Failed to stop tahoe node.'
+ print 'Ignoring cleanup exception:'
+ # Indent the exception description:
+ desc = str(e).rstrip()
+ print ' ' + desc.replace('\n', '\n ')
+
+ def webapi_call(self, method, path, body=None, **options):
+ if options:
+ path = path + '?' + ('&'.join(['%s=%s' % kv for kv in options.items()]))
+
+ conn = httplib.HTTPConnection('127.0.0.1', self.port)
+ conn.request(method, path, body = body)
+ resp = conn.getresponse()
+
+ if resp.status != 200:
+ tmpl = 'A webapi operation failed.\n'
+ tmpl += 'Request: %r %r\n'
+ tmpl += 'Body:\n%s\n'
+ tmpl += 'Response:\nStatus %r\nBody:\n%s'
+ raise self.SetupFailure(tmpl,
+ method, path,
+ body or '',
+ resp.status, body)
+
+ return resp.read()
+
+ def create_dirnode(self):
+ return self.webapi_call('PUT', '/uri', t='mkdir').strip()
+
+ def attach_node(self, dircap, childcap, childname):
+ body = self.webapi_call('PUT',
+ '/uri/%s/%s' % (dircap, childname),
+ body = childcap,
+ t = 'uri',
+ replace = 'false')
+ assert body.strip() == childcap, `status, dircap, childcap, childname`
+
+ def polling_operation(self, operation, timeout = 10.0, pollinterval = 0.2):
+ totaltime = timeout # Fudging for edge-case SetupFailure description...
+
+ totalattempts = int(timeout / pollinterval)
+
+ starttime = time.time()
+ for attempt in range(totalattempts):
+ opstart = time.time()
+
+ try:
+ result = operation()
+ except KeyboardInterrupt, e:
+ raise
+ except Exception, e:
+ result = False
+
+ totaltime = time.time() - starttime
+
+ if result is not False:
+ #tmpl = '(Polling took over %.2f seconds.)'
+ #print tmpl % (totaltime,)
+ return result
+
+ elif totaltime > timeout:
+ break
+
+ else:
+ opdelay = time.time() - opstart
+ realinterval = max(0., pollinterval - opdelay)
+
+ #tmpl = '(Poll attempt %d failed after %.2f seconds, sleeping %.2f seconds.)'
+ #print tmpl % (attempt+1, opdelay, realinterval)
+ time.sleep(realinterval)
+
+ tmpl = 'Timeout after waiting for creation of introducer.furl.\n'
+ tmpl += 'Waited %.2f seconds (%d polls).'
+ raise self.SetupFailure(tmpl, totaltime, attempt+1)
+
+ def warn(self, tmpl, *args):
+ print ('Test Warning: ' + tmpl) % args
+
+
+ # SystemTest Exceptions:
+ class Failure (Exception):
+ def __init__(self, tmpl, *args):
+ msg = self.Prefix + (tmpl % args)
+ Exception.__init__(self, msg)
+
+ class SetupFailure (Failure):
+ Prefix = 'Setup Failure - The test framework encountered an error:\n'
+
+ class TestFailure (Failure):
+ Prefix = 'TestFailure: '
+
+
+### Unit Tests:
+class Impl_A_UnitTests (unittest.TestCase):
+ '''Tests small stand-alone functions.'''
+ def test_canonicalize_cap(self):
+ iopairs = [('http://127.0.0.1:8123/uri/URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y',
+ 'URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y'),
+ ('http://127.0.0.1:8123/uri/URI%3ACHK%3Ak7ktp1qr7szmt98s1y3ha61d9w%3A8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy%3A3%3A12%3A242?filename=welcome.html',
+ 'URI:CHK:k7ktp1qr7szmt98s1y3ha61d9w:8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy:3:12:242?filename=welcome.html')]
+
+ for input, output in iopairs:
+ result = impl_a.canonicalize_cap(input)
+ self.failUnlessEqual(output, result, 'input == %r' % (input,))
+
+
+
+### Misc:
+def gather_output(*args, **kwargs):
+ '''
+ This expects the child does not require input and that it closes
+ stdout/err eventually.
+ '''
+ p = subprocess.Popen(stdout = subprocess.PIPE,
+ stderr = subprocess.STDOUT,
+ *args,
+ **kwargs)
+ output = p.stdout.read()
+ exitcode = p.wait()
+ return (exitcode, output)
+
+
+ExpectedCreationOutput = r'(introducer|client) created in (?P<path>.*?)\n'
+ExpectedStartOutput = r'STARTING (?P<path>.*?)\n(introducer|client) node probably started'
+
+
+Usage = '''
+Usage: %s [target]
+
+Run tests for the given target.
+
+target is one of: unit, system, or all
+''' % (sys.argv[0],)
+
+
+
+if __name__ == '__main__':
+ main()
+++ /dev/null
-
-Welcome to the tahoe fuse interface prototype!
-
-
-Dependencies:
-
-In addition to a working tahoe installation, this interface depends
-on the python-fuse interface. This package is available on Ubuntu
-systems as "python-fuse". It is only known to work with ubuntu
-package version "2.5-5build1". The latest ubuntu package (version
-"1:0.2-pre3-3") appears to not work currently.
-
-Unfortunately this package appears poorly maintained (notice the wildy
-different version strings and changing API semantics), so if you know
-of a good replacement pythonic fuse interface, please let tahoe-dev know
-about it!
-
-
-Configuration:
-
-Currently tahoe-fuse.py uses the same ~/.tahoe/private/root_dir.cap
-file (which is also the CLI default). This is not configurable yet.
-Place a directory cap in this file. (Hint: If you can run "tahoe ls"
-and see a directory listing, this file is properly configured.)
-
-
-Commandline:
-
-The usage is "tahoe-fuse.py <mountpoint>". The mount point needs to
-be an existing directory which should be empty. (If it's not empty
-the contents will be safe, but unavailable while the tahoe-fuse.py
-process is mounted there.)
-
-
-Usage:
-
-To use the interface, use other programs to poke around the
-mountpoint. You should be able to see the same contents as you would
-by using the CLI or WUI for the same directory cap.
-
-
-Runtime Behavior Notes:
-
-Read-only:
-Only reading a tahoe grid is supported, which is reflected in
-the permission modes. With Tahoe 0.7.0, write access should be easier
-to implement, but is not yet present.
-
-In-Memory File Caching:
-Currently requesting a particular file for read causes the entire file to
-be retrieved into tahoe-fuse.py memory before the read operation returns!
-This caching is reused for subsequent reads. Beware large files.
-When transitioning to a finer-grained fuse api, this caching should be
-replaced with straight-forward calls to the wapi. In my opinion, the
-Tahoe node should do all the caching tricks, so that extensions such as
-tahoe-fuse.py can be simple and thin.
-
-Backgrounding Behavior:
-When using the 2.5-5build1 ubuntu package, and no other arguments
-besides a mountpoint to tahoe-fuse.py, the process should remain in
-the foreground and print debug information. Other python-fuse
-versions appear to alter this behavior and may fork the process to
-the background and obscure the log output. Bonus points to whomever
-discovers the fate of these poor log messages in this case.
-
-"Investigative Logging":
-This prototype is designed to aide in further fuse development, so
-currently *every* fuse interface call figures out the process from
-which the file system request originates, then it figures out that
-processes commandline (this uses the /proc file system). This is handy
-for interactive inspection of what kinds of behavior invokes which
-file system operations, but may not work for you. To disable this
-inspection, edit the source and comment out all of the "@debugcall"
-[FIXME: double check python ref name] method decorators by inserting a
-'#' so it looks like "#@debugcall" (without quotes).
-
-Not-to-spec:
-The current version was not implemented according to any spec and
-makes quite a few dubious "guesses" for what data to pass the fuse
-interface. You may see bizarre values, which may potentialy confuse
-any processes visiting the files under the mount point.
-
-Serial, blocking operations:
-Most fuse operations result in one or more http calls to the WAPI.
-These are serial and blocking (at least for the tested python-fuse
-version 2.5-5build1), so access to this file system is quite
-inefficient.
-
-
-Good luck!
+++ /dev/null
-#! /usr/bin/env python
-'''
-Unit and system tests for tahoe-fuse.
-'''
-
-# Note: It's always a SetupFailure, not a TestFailure if a webapi
-# operation fails, because this does not indicate a fuse interface
-# failure.
-
-# TODO: Unmount after tests regardless of failure or success!
-
-# TODO: Test mismatches between tahoe and fuse/posix. What about nodes
-# with crazy names ('\0', unicode, '/', '..')? Huuuuge files?
-# Huuuuge directories... As tahoe approaches production quality, it'd
-# be nice if the fuse interface did so also by hardening against such cases.
-
-# FIXME: Only create / launch necessary nodes. Do we still need an introducer and three nodes?
-
-# FIXME: This framework might be replaceable with twisted.trial,
-# especially the "layer" design, which is a bit cumbersome when
-# using recursion to manage multiple clients.
-
-# FIXME: Identify all race conditions (hint: starting clients, versus
-# using the grid fs).
-
-import sys, os, shutil, unittest, subprocess
-import tempfile, re, time, signal, random, httplib
-import traceback
-
-import tahoe_fuse
-
-
-### Main flow control:
-def main(args = sys.argv[1:]):
- target = 'all'
- if args:
- if len(args) != 1:
- raise SystemExit(Usage)
- target = args[0]
-
- if target not in ('all', 'unit', 'system'):
- raise SystemExit(Usage)
-
- if target in ('all', 'unit'):
- run_unit_tests()
-
- if target in ('all', 'system'):
- run_system_test()
-
-
-def run_unit_tests():
- print 'Running Unit Tests.'
- try:
- unittest.main()
- except SystemExit, se:
- pass
- print 'Unit Tests complete.\n'
-
-
-def run_system_test():
- SystemTest().run()
-
-
-### System Testing:
-class SystemTest (object):
- def __init__(self):
- # These members represent configuration:
- self.fullcleanup = False # FIXME: Make this a commandline option.
-
- # These members represent test state:
- self.cliexec = None
- self.testroot = None
-
- # This test state is specific to the first client:
- self.port = None
- self.clientbase = None
-
- ## Top-level flow control:
- # These "*_layer" methods call eachother in a linear fashion, using
- # exception unwinding to do cleanup properly. Each "layer" invokes
- # a deeper layer, and each layer does its own cleanup upon exit.
-
- def run(self, fullcleanup = False):
- '''
- If full_cleanup, delete all temporary state.
- Else: If there is an error do not delete basedirs.
-
- Set to False if you wish to analyze a failure.
- '''
- self.fullcleanup = fullcleanup
- print '\n*** Setting up system tests.'
- try:
- failures, total = self.init_cli_layer()
- print '\n*** System Tests complete: %d failed out of %d.' % (failures, total)
- except self.SetupFailure, sfail:
- print
- print sfail
- print '\n*** System Tests were not successfully completed.'
-
- def init_cli_layer(self):
- '''This layer finds the appropriate tahoe executable.'''
- runtestpath = os.path.abspath(sys.argv[0])
- path = runtestpath
- for expectedname in ('runtests.py', 'fuse_a', 'contrib'):
- path, name = os.path.split(path)
-
- if name != expectedname:
- reason = 'Unexpected test script path: %r\n'
- reason += 'The system test script must be run from the source directory.'
- raise self.SetupFailure(reason, runtestpath)
-
- self.cliexec = os.path.join(path, 'bin', 'tahoe')
- version = self.run_tahoe('--version')
- print 'Using %r with version:\n%s' % (self.cliexec, version.rstrip())
-
- return self.create_testroot_layer()
-
- def create_testroot_layer(self):
- print 'Creating test base directory.'
- self.testroot = tempfile.mkdtemp(prefix='tahoe_fuse_test_')
- try:
- return self.launch_introducer_layer()
- finally:
- if self.fullcleanup:
- print 'Cleaning up test root directory.'
- try:
- shutil.rmtree(self.testroot)
- except Exception, e:
- print 'Exception removing test root directory: %r' % (self.testroot, )
- print 'Ignoring cleanup exception: %r' % (e,)
- else:
- print 'Leaving test root directory: %r' % (self.testroot, )
-
-
- def launch_introducer_layer(self):
- print 'Launching introducer.'
- introbase = os.path.join(self.testroot, 'introducer')
-
- # NOTE: We assume if tahoe exits with non-zero status, no separate
- # tahoe child process is still running.
- createoutput = self.run_tahoe('create-introducer', '--basedir', introbase)
-
- self.check_tahoe_output(createoutput, ExpectedCreationOutput, introbase)
-
- startoutput = self.run_tahoe('start', '--basedir', introbase)
- try:
- self.check_tahoe_output(startoutput, ExpectedStartOutput, introbase)
-
- return self.launch_clients_layer(introbase)
-
- finally:
- print 'Stopping introducer node.'
- self.stop_node(introbase)
-
- TotalClientsNeeded = 3
- def launch_clients_layer(self, introbase, clientnum = 1):
- if clientnum > self.TotalClientsNeeded:
- return self.create_test_dirnode_layer()
-
- tmpl = 'Launching client %d of %d.'
- print tmpl % (clientnum,
- self.TotalClientsNeeded)
-
- base = os.path.join(self.testroot, 'client_%d' % (clientnum,))
-
- output = self.run_tahoe('create-client', '--basedir', base)
- self.check_tahoe_output(output, ExpectedCreationOutput, base)
-
- webportpath = os.path.join(base, 'webport')
- if clientnum == 1:
- # The first client is special:
- self.clientbase = base
- self.port = random.randrange(1024, 2**15)
-
- f = open(webportpath, 'w')
- f.write('tcp:%d:interface=127.0.0.1\n' % self.port)
- f.close()
- else:
- os.remove(webportpath)
-
-
- introfurl = os.path.join(introbase, 'introducer.furl')
-
- self.polling_operation(lambda : os.path.isfile(introfurl))
- shutil.copy(introfurl, base)
-
- # NOTE: We assume if tahoe exist with non-zero status, no separate
- # tahoe child process is still running.
- startoutput = self.run_tahoe('start', '--basedir', base)
- try:
- self.check_tahoe_output(startoutput, ExpectedStartOutput, base)
-
- return self.launch_clients_layer(introbase, clientnum+1)
-
- finally:
- print 'Stopping client node %d.' % (clientnum,)
- self.stop_node(base)
-
- def create_test_dirnode_layer(self):
- print 'Creating test dirnode.'
-
- cap = self.create_dirnode()
-
- f = open(os.path.join(self.clientbase, 'private', 'root_dir.cap'), 'w')
- f.write(cap)
- f.close()
-
- return self.mount_fuse_layer(cap)
-
- def mount_fuse_layer(self, fusebasecap):
- print 'Mounting fuse interface.'
-
- mp = os.path.join(self.testroot, 'mountpoint')
- os.mkdir(mp)
-
- thispath = os.path.abspath(sys.argv[0])
- thisdir = os.path.dirname(thispath)
- fusescript = os.path.join(thisdir, 'tahoe_fuse.py')
-
- # Even though --help says -f means "foreground operation",
- # my version (ubuntu's python-fuse 1:0.2-pre3-3) still forks to
- # the background... So now we run tahoe_fuse.py and wait for
- # the parent process to return. We use fusermount -u to kill
- # the forked daemon and unmount the test directory.
-
- exitcode, output = gather_output(['python',
- fusescript,
- mp,
- '--basedir', self.clientbase])
-
- if exitcode != 0 or output:
- tmpl = 'tahoe_fuse.py failed to launch:\n'
- tmpl += 'Exit Status: %r\n'
- tmpl += 'Output:\n%s\n'
- raise self.SetupFailure(tmpl, exitcode, output)
-
- try:
- return self.run_test_layer(fusebasecap, mp)
-
- finally:
- print '\n*** Cleaning fuse mount and daemon process'
- args = ['fusermount', '-u', mp]
- ec, out = gather_output(args)
- if exitcode != 0 or output:
- tmpl = 'fusermount failed to unmount tahoe_fuse.py:\n'
- tmpl += 'Arguments: %r\n'
- tmpl += 'Exit Status: %r\n'
- tmpl += 'Output:\n%s\n'
- raise self.SetupFailure(tmpl, args, ec, out)
-
- def run_test_layer(self, fbcap, mountpoint):
- total = failures = 0
- for name in sorted(dir(self)):
- if name.startswith('test_'):
- total += 1
- print '\n*** Running test #%d: %s' % (total, name)
- try:
- testcap = self.create_dirnode()
- self.attach_node(fbcap, testcap, name)
-
- method = getattr(self, name)
- method(testcap, testdir = os.path.join(mountpoint, name))
- print 'Test succeeded.'
- except self.TestFailure, f:
- print f
- failures += 1
- except:
- print 'Error in test code... Cleaning up.'
- raise
-
- return (failures, total)
-
-
- # Tests:
- def test_directory_existence(self, testcap, testdir):
- if not os.path.isdir(testdir):
- raise self.TestFailure('Attached test directory not found: %r', testdir)
-
- def test_empty_directory_listing(self, testcap, testdir):
- listing = os.listdir(testdir)
- if listing:
- raise self.TestFailure('Expected empty directory, found: %r', listing)
-
- def test_directory_listing(self, testcap, testdir):
- names = []
- filesizes = {}
-
- for i in range(3):
- fname = 'file_%d' % (i,)
- names.append(fname)
- body = 'Hello World #%d!' % (i,)
- filesizes[fname] = len(body)
-
- cap = self.webapi_call('PUT', '/uri', body)
- self.attach_node(testcap, cap, fname)
-
- dname = 'dir_%d' % (i,)
- names.append(dname)
-
- cap = self.create_dirnode()
- self.attach_node(testcap, cap, dname)
-
- names.sort()
-
- listing = os.listdir(testdir)
- listing.sort()
- if listing != names:
- tmpl = 'Expected directory list containing %r but fuse gave %r'
- raise self.TestFailure(tmpl, names, listing)
-
- for file, size in filesizes.items():
- st = os.stat(os.path.join(testdir, file))
- if st.st_size != size:
- tmpl = 'Expected %r size of %r but fuse returned %r'
- raise self.TestFailure(tmpl, file, size, st.st_size)
-
- def test_file_contents(self, testcap, testdir):
- name = 'hw.txt'
- body = 'Hello World!'
-
- cap = self.webapi_call('PUT', '/uri', body)
- self.attach_node(testcap, cap, name)
-
- path = os.path.join(testdir, name)
- try:
- found = open(path, 'r').read()
- except Exception, err:
- tmpl = 'Could not read file contents of %r: %r'
- raise self.TestFailure(tmpl, path, err)
-
- if found != body:
- tmpl = 'Expected file contents %r but found %r'
- raise self.TestFailure(tmpl, body, found)
-
-
- # Utilities:
- def run_tahoe(self, *args):
- realargs = ('tahoe',) + args
- status, output = gather_output(realargs, executable=self.cliexec)
- if status != 0:
- tmpl = 'The tahoe cli exited with nonzero status.\n'
- tmpl += 'Executable: %r\n'
- tmpl += 'Command arguments: %r\n'
- tmpl += 'Exit status: %r\n'
- tmpl += 'Output:\n%s\n[End of tahoe output.]\n'
- raise self.SetupFailure(tmpl,
- self.cliexec,
- realargs,
- status,
- output)
- return output
-
- def check_tahoe_output(self, output, expected, expdir):
- m = re.match(expected, output, re.M)
- if m is None:
- tmpl = 'The output of tahoe did not match the expectation:\n'
- tmpl += 'Expected regex: %s\n'
- tmpl += 'Actual output: %r\n'
- self.warn(tmpl, expected, output)
-
- elif expdir != m.group('path'):
- tmpl = 'The output of tahoe refers to an unexpected directory:\n'
- tmpl += 'Expected directory: %r\n'
- tmpl += 'Actual directory: %r\n'
- self.warn(tmpl, expdir, m.group(1))
-
- def stop_node(self, basedir):
- try:
- self.run_tahoe('stop', '--basedir', basedir)
- except Exception, e:
- print 'Failed to stop tahoe node.'
- print 'Ignoring cleanup exception:'
- # Indent the exception description:
- desc = str(e).rstrip()
- print ' ' + desc.replace('\n', '\n ')
-
- def webapi_call(self, method, path, body=None, **options):
- if options:
- path = path + '?' + ('&'.join(['%s=%s' % kv for kv in options.items()]))
-
- conn = httplib.HTTPConnection('127.0.0.1', self.port)
- conn.request(method, path, body = body)
- resp = conn.getresponse()
-
- if resp.status != 200:
- tmpl = 'A webapi operation failed.\n'
- tmpl += 'Request: %r %r\n'
- tmpl += 'Body:\n%s\n'
- tmpl += 'Response:\nStatus %r\nBody:\n%s'
- raise self.SetupFailure(tmpl,
- method, path,
- body or '',
- resp.status, body)
-
- return resp.read()
-
- def create_dirnode(self):
- return self.webapi_call('PUT', '/uri', t='mkdir').strip()
-
- def attach_node(self, dircap, childcap, childname):
- body = self.webapi_call('PUT',
- '/uri/%s/%s' % (dircap, childname),
- body = childcap,
- t = 'uri',
- replace = 'false')
- assert body.strip() == childcap, `status, dircap, childcap, childname`
-
- def polling_operation(self, operation, timeout = 10.0, pollinterval = 0.2):
- totaltime = timeout # Fudging for edge-case SetupFailure description...
-
- totalattempts = int(timeout / pollinterval)
-
- starttime = time.time()
- for attempt in range(totalattempts):
- opstart = time.time()
-
- try:
- result = operation()
- except KeyboardInterrupt, e:
- raise
- except Exception, e:
- result = False
-
- totaltime = time.time() - starttime
-
- if result is not False:
- #tmpl = '(Polling took over %.2f seconds.)'
- #print tmpl % (totaltime,)
- return result
-
- elif totaltime > timeout:
- break
-
- else:
- opdelay = time.time() - opstart
- realinterval = max(0., pollinterval - opdelay)
-
- #tmpl = '(Poll attempt %d failed after %.2f seconds, sleeping %.2f seconds.)'
- #print tmpl % (attempt+1, opdelay, realinterval)
- time.sleep(realinterval)
-
- tmpl = 'Timeout after waiting for creation of introducer.furl.\n'
- tmpl += 'Waited %.2f seconds (%d polls).'
- raise self.SetupFailure(tmpl, totaltime, attempt+1)
-
- def warn(self, tmpl, *args):
- print ('Test Warning: ' + tmpl) % args
-
-
- # SystemTest Exceptions:
- class Failure (Exception):
- def __init__(self, tmpl, *args):
- msg = self.Prefix + (tmpl % args)
- Exception.__init__(self, msg)
-
- class SetupFailure (Failure):
- Prefix = 'Setup Failure - The test framework encountered an error:\n'
-
- class TestFailure (Failure):
- Prefix = 'TestFailure: '
-
-
-### Unit Tests:
-class TestUtilFunctions (unittest.TestCase):
- '''Tests small stand-alone functions.'''
- def test_canonicalize_cap(self):
- iopairs = [('http://127.0.0.1:8123/uri/URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y',
- 'URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y'),
- ('http://127.0.0.1:8123/uri/URI%3ACHK%3Ak7ktp1qr7szmt98s1y3ha61d9w%3A8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy%3A3%3A12%3A242?filename=welcome.html',
- 'URI:CHK:k7ktp1qr7szmt98s1y3ha61d9w:8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy:3:12:242?filename=welcome.html')]
-
- for input, output in iopairs:
- result = tahoe_fuse.canonicalize_cap(input)
- self.failUnlessEqual(output, result, 'input == %r' % (input,))
-
-
-
-### Misc:
-def gather_output(*args, **kwargs):
- '''
- This expects the child does not require input and that it closes
- stdout/err eventually.
- '''
- p = subprocess.Popen(stdout = subprocess.PIPE,
- stderr = subprocess.STDOUT,
- *args,
- **kwargs)
- output = p.stdout.read()
- exitcode = p.wait()
- return (exitcode, output)
-
-
-ExpectedCreationOutput = r'(introducer|client) created in (?P<path>.*?)\n'
-ExpectedStartOutput = r'STARTING (?P<path>.*?)\n(introducer|client) node probably started'
-
-
-Usage = '''
-Usage: %s [target]
-
-Run tests for the given target.
-
-target is one of: unit, system, or all
-''' % (sys.argv[0],)
-
-
-
-if __name__ == '__main__':
- main()
+++ /dev/null
-#! /usr/bin/env python
-'''
-Tahoe thin-client fuse module.
-
-See the accompanying README for configuration/usage details.
-
-Goals:
-
-- Delegate to Tahoe webapi as much as possible.
-- Thin rather than clever. (Even when that means clunky.)
-
-
-Warts:
-
-- Reads cache entire file contents, violating the thinness goal. Can we GET spans of files?
-- Single threaded.
-
-
-Road-map:
-1. Add unit tests where possible with little code modification.
-2. Make unit tests pass for a variety of python-fuse module versions.
-3. Modify the design to make possible unit test coverage of larger portions of code.
-
-Wishlist:
-- Perhaps integrate cli aliases or root_dir.cap.
-- Research pkg_resources; see if it can replace the try-import-except-import-error pattern.
-- Switch to logging instead of homebrew logging.
-'''
-
-
-#import bindann
-#bindann.install_exception_handler()
-
-import sys, stat, os, errno, urllib, time
-
-try:
- import simplejson
-except ImportError, e:
- raise SystemExit('''\
-Could not import simplejson, which is bundled with Tahoe. Please
-update your PYTHONPATH environment variable to include the tahoe
-"support/lib/python<VERSION>/site-packages" directory.
-
-If you run this from the Tahoe source directory, use this command:
-PYTHONPATH="$PYTHONPATH:./support/lib/python%d.%d/site-packages/" python %s
-''' % (sys.version_info[:2] + (' '.join(sys.argv),)))
-
-
-try:
- import fuse
-except ImportError, e:
- raise SystemExit('''\
-Could not import fuse, the pythonic fuse bindings. This dependency
-of tahoe-fuse.py is *not* bundled with tahoe. Please install it.
-On debian/ubuntu systems run: sudo apt-get install python-fuse
-''')
-
-# FIXME: Check for non-working fuse versions here.
-# FIXME: Make this work for all common python-fuse versions.
-
-# FIXME: Currently uses the old, silly path-based (non-stateful) interface:
-fuse.fuse_python_api = (0, 1) # Use the silly path-based api for now.
-
-
-### Config:
-TahoeConfigDir = '~/.tahoe'
-MagicDevNumber = 42
-UnknownSize = -1
-
-
-def main():
- basedir = os.path.expanduser(TahoeConfigDir)
-
- for i, arg in enumerate(sys.argv):
- if arg == '--basedir':
- try:
- basedir = sys.argv[i+1]
- sys.argv[i:i+2] = []
- except IndexError:
- sys.argv = [sys.argv[0], '--help']
-
-
- log_init(basedir)
- log('Commandline: %r', sys.argv)
-
- fs = TahoeFS(basedir)
- fs.main()
-
-
-### Utilities for debug:
-_logfile = None # Private to log* functions.
-
-def log_init(confdir):
- global _logfile
-
- logpath = os.path.join(confdir, 'logs', 'tahoe_fuse.log')
- _logfile = open(logpath, 'a')
- log('Log opened at: %s\n', time.strftime('%Y-%m-%d %H:%M:%S'))
-
-
-def log(msg, *args):
- _logfile.write((msg % args) + '\n')
- _logfile.flush()
-
-
-def trace_calls(m):
- def dbmeth(self, *a, **kw):
- pid = self.GetContext()['pid']
- log('[%d %r]\n%s%r%r', pid, get_cmdline(pid), m.__name__, a, kw)
- try:
- r = m(self, *a, **kw)
- if (type(r) is int) and (r < 0):
- log('-> -%s\n', errno.errorcode[-r],)
- else:
- repstr = repr(r)[:256]
- log('-> %s\n', repstr)
- return r
- except:
- sys.excepthook(*sys.exc_info())
-
- return dbmeth
-
-
-def get_cmdline(pid):
- f = open('/proc/%d/cmdline' % pid, 'r')
- args = f.read().split('\0')
- f.close()
- assert args[-1] == ''
- return args[:-1]
-
-
-class SystemError (Exception):
- def __init__(self, eno):
- self.eno = eno
- Exception.__init__(self, errno.errorcode[eno])
-
- @staticmethod
- def wrap_returns(meth):
- def wrapper(*args, **kw):
- try:
- return meth(*args, **kw)
- except SystemError, e:
- return -e.eno
- wrapper.__name__ = meth.__name__
- return wrapper
-
-
-### Heart of the Matter:
-class TahoeFS (fuse.Fuse):
- def __init__(self, confdir):
- log('Initializing with confdir = %r', confdir)
- fuse.Fuse.__init__(self)
- self.confdir = confdir
-
- self.flags = 0 # FIXME: What goes here?
- self.multithreaded = 0
-
- # silly path-based file handles.
- self.filecontents = {} # {path -> contents}
-
- self._init_url()
- self._init_rootdir()
-
- def _init_url(self):
- f = open(os.path.join(self.confdir, 'webport'), 'r')
- contents = f.read()
- f.close()
-
- fields = contents.split(':')
- proto, port = fields[:2]
- assert proto == 'tcp'
- port = int(port)
- self.url = 'http://localhost:%d' % (port,)
-
- def _init_rootdir(self):
- # For now we just use the same default as the CLI:
- rootdirfn = os.path.join(self.confdir, 'private', 'root_dir.cap')
- try:
- f = open(rootdirfn, 'r')
- cap = f.read().strip()
- f.close()
- except EnvironmentError, le:
- # FIXME: This user-friendly help message may be platform-dependent because it checks the exception description.
- if le.args[1].find('No such file or directory') != -1:
- raise SystemExit('%s requires a directory capability in %s, but it was not found.\nPlease see "The CLI" in "docs/using.html".\n' % (sys.argv[0], rootdirfn))
- else:
- raise le
-
- self.rootdir = TahoeDir(self.url, canonicalize_cap(cap))
-
- def _get_node(self, path):
- assert path.startswith('/')
- if path == '/':
- return self.rootdir.resolve_path([])
- else:
- parts = path.split('/')[1:]
- return self.rootdir.resolve_path(parts)
-
- def _get_contents(self, path):
- node = self._get_node(path)
- contents = node.open().read()
- self.filecontents[path] = contents
- return contents
-
- @trace_calls
- @SystemError.wrap_returns
- def getattr(self, path):
- node = self._get_node(path)
- return node.getattr()
-
- @trace_calls
- @SystemError.wrap_returns
- def getdir(self, path):
- """
- return: [(name, typeflag), ... ]
- """
- node = self._get_node(path)
- return node.getdir()
-
- @trace_calls
- @SystemError.wrap_returns
- def mythread(self):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def chmod(self, path, mode):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def chown(self, path, uid, gid):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def fsync(self, path, isFsyncFile):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def link(self, target, link):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def mkdir(self, path, mode):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def mknod(self, path, mode, dev_ignored):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def open(self, path, mode):
- IgnoredFlags = os.O_RDONLY | os.O_NONBLOCK | os.O_SYNC | os.O_LARGEFILE
- # Note: IgnoredFlags are all ignored!
- for fname in dir(os):
- if fname.startswith('O_'):
- flag = getattr(os, fname)
- if flag & IgnoredFlags:
- continue
- elif mode & flag:
- log('Flag not supported: %s', fname)
- raise SystemError(errno.ENOSYS)
-
- self._get_contents(path)
- return 0
-
- @trace_calls
- @SystemError.wrap_returns
- def read(self, path, length, offset):
- return self._get_contents(path)[offset:length]
-
- @trace_calls
- @SystemError.wrap_returns
- def release(self, path):
- del self.filecontents[path]
- return 0
-
- @trace_calls
- @SystemError.wrap_returns
- def readlink(self, path):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def rename(self, oldpath, newpath):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def rmdir(self, path):
- return -errno.ENOSYS
-
- #@trace_calls
- @SystemError.wrap_returns
- def statfs(self):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def symlink ( self, targetPath, linkPath ):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def truncate(self, path, size):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def unlink(self, path):
- return -errno.ENOSYS
-
- @trace_calls
- @SystemError.wrap_returns
- def utime(self, path, times):
- return -errno.ENOSYS
-
-
-class TahoeNode (object):
- NextInode = 0
-
- @staticmethod
- def make(baseurl, uri):
- typefield = uri.split(':', 2)[1]
- # FIXME: is this check correct?
- if uri.find('URI:DIR2') != -1:
- return TahoeDir(baseurl, uri)
- else:
- return TahoeFile(baseurl, uri)
-
- def __init__(self, baseurl, uri):
- self.burl = baseurl
- self.uri = uri
- self.fullurl = '%s/uri/%s' % (self.burl, self.uri)
- self.inode = TahoeNode.NextInode
- TahoeNode.NextInode += 1
-
- def getattr(self):
- """
- - st_mode (protection bits)
- - st_ino (inode number)
- - st_dev (device)
- - st_nlink (number of hard links)
- - st_uid (user ID of owner)
- - st_gid (group ID of owner)
- - st_size (size of file, in bytes)
- - st_atime (time of most recent access)
- - st_mtime (time of most recent content modification)
- - st_ctime (platform dependent; time of most recent metadata change on Unix,
- or the time of creation on Windows).
- """
- # FIXME: Return metadata that isn't completely fabricated.
- return (self.get_mode(),
- self.inode,
- MagicDevNumber,
- self.get_linkcount(),
- os.getuid(),
- os.getgid(),
- self.get_size(),
- 0,
- 0,
- 0)
-
- def get_metadata(self):
- f = self.open('?t=json')
- json = f.read()
- f.close()
- return simplejson.loads(json)
-
- def open(self, postfix=''):
- url = self.fullurl + postfix
- log('*** Fetching: %r', url)
- return urllib.urlopen(url)
-
-
-class TahoeFile (TahoeNode):
- def __init__(self, baseurl, uri):
- #assert uri.split(':', 2)[1] in ('CHK', 'LIT'), `uri` # fails as of 0.7.0
- TahoeNode.__init__(self, baseurl, uri)
-
- # nonfuse:
- def get_mode(self):
- return stat.S_IFREG | 0400 # Read only regular file.
-
- def get_linkcount(self):
- return 1
-
- def get_size(self):
- rawsize = self.get_metadata()[1]['size']
- if type(rawsize) is not int: # FIXME: What about sizes which do not fit in python int?
- assert rawsize == u'?', `rawsize`
- return UnknownSize
- else:
- return rawsize
-
- def resolve_path(self, path):
- assert path == []
- return self
-
-
-class TahoeDir (TahoeNode):
- def __init__(self, baseurl, uri):
- TahoeNode.__init__(self, baseurl, uri)
-
- self.mode = stat.S_IFDIR | 0500 # Read only directory.
-
- # FUSE:
- def getdir(self):
- d = [('.', self.get_mode()), ('..', self.get_mode())]
- for name, child in self.get_children().items():
- if name: # Just ignore this crazy case!
- d.append((name, child.get_mode()))
- return d
-
- # nonfuse:
- def get_mode(self):
- return stat.S_IFDIR | 0500 # Read only directory.
-
- def get_linkcount(self):
- return len(self.getdir())
-
- def get_size(self):
- return 2 ** 12 # FIXME: What do we return here? len(self.get_metadata())
-
- def resolve_path(self, path):
- assert type(path) is list
-
- if path:
- head = path[0]
- child = self.get_child(head)
- return child.resolve_path(path[1:])
- else:
- return self
-
- def get_child(self, name):
- c = self.get_children()
- return c[name]
-
- def get_children(self):
- flag, md = self.get_metadata()
- assert flag == 'dirnode'
-
- c = {}
- for name, (childflag, childmd) in md['children'].items():
- if childflag == 'dirnode':
- cls = TahoeDir
- else:
- cls = TahoeFile
-
- c[str(name)] = cls(self.burl, childmd['ro_uri'])
- return c
-
-
-def canonicalize_cap(cap):
- cap = urllib.unquote(cap)
- i = cap.find('URI:')
- assert i != -1, 'A cap must contain "URI:...", but this does not: ' + cap
- return cap[i:]
-
-
-if __name__ == '__main__':
- main()
-
+++ /dev/null
-This announcement is archived in the tahoe-dev mailing list archive:
-
-http://allmydata.org/pipermail/tahoe-dev/2008-March/000465.html
-
-[tahoe-dev] Another FUSE interface
-Armin Rigo arigo at tunes.org
-Sat Mar 29 04:35:36 PDT 2008
-
- * Previous message: [tahoe-dev] announcing allmydata.org "Tahoe", v1.0
- * Next message: [tahoe-dev] convergent encryption reconsidered -- salting and key-strengthening
- * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
-
-Hi all,
-
-I implemented for fun another Tahoe-to-FUSE interface using my own set
-of FUSE bindings. If you are interested, you can check out the
-following subversion directory:
-
- http://codespeak.net/svn/user/arigo/hack/pyfuse
-
-tahoe.py is a 100-lines, half-an-hour-job interface to Tahoe, limited to
-read-only at the moment. The rest of the directory contains PyFuse, and
-many other small usage examples. PyFuse is a pure Python FUSE daemon
-(no messy linking issues, no dependencies).
-
-
-A bientot,
-
-Armin Rigo
-
- * Previous message: [tahoe-dev] announcing allmydata.org "Tahoe", v1.0
- * Next message: [tahoe-dev] convergent encryption reconsidered -- salting and key-strengthening
- * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
-
-More information about the tahoe-dev mailing list
-
projects / tahoe-lafs / tahoe-lafs.git / commitdiff