From: Zooko O'Whielacronx Date: Mon, 28 Apr 2008 14:05:44 +0000 (-0700) Subject: contrib: add a note about Armin Rigo's fuse implementation X-Git-Tag: allmydata-tahoe-1.1.0~178 X-Git-Url: https://git.rkrishnan.org/pf/content/en/service/something?a=commitdiff_plain;h=c88ea89f1476ebf60d6cdc36f2905f015fa2710d;p=tahoe-lafs%2Ftahoe-lafs.git contrib: add a note about Armin Rigo's fuse implementation --- diff --git a/contrib/fuse/README b/contrib/fuse/README deleted file mode 100644 index f79107cc..00000000 --- a/contrib/fuse/README +++ /dev/null @@ -1,90 +0,0 @@ - -Welcome to the tahoe fuse interface prototype! - - -Dependencies: - -In addition to a working tahoe installation, this interface depends -on the python-fuse interface. This package is available on Ubuntu -systems as "python-fuse". It is only known to work with ubuntu -package version "2.5-5build1". The latest ubuntu package (version -"1:0.2-pre3-3") appears to not work currently. - -Unfortunately this package appears poorly maintained (notice the wildy -different version strings and changing API semantics), so if you know -of a good replacement pythonic fuse interface, please let tahoe-dev know -about it! - - -Configuration: - -Currently tahoe-fuse.py uses the same ~/.tahoe/private/root_dir.cap -file (which is also the CLI default). This is not configurable yet. -Place a directory cap in this file. (Hint: If you can run "tahoe ls" -and see a directory listing, this file is properly configured.) - - -Commandline: - -The usage is "tahoe-fuse.py ". The mount point needs to -be an existing directory which should be empty. (If it's not empty -the contents will be safe, but unavailable while the tahoe-fuse.py -process is mounted there.) - - -Usage: - -To use the interface, use other programs to poke around the -mountpoint. You should be able to see the same contents as you would -by using the CLI or WUI for the same directory cap. - - -Runtime Behavior Notes: - -Read-only: -Only reading a tahoe grid is supported, which is reflected in -the permission modes. With Tahoe 0.7.0, write access should be easier -to implement, but is not yet present. - -In-Memory File Caching: -Currently requesting a particular file for read causes the entire file to -be retrieved into tahoe-fuse.py memory before the read operation returns! -This caching is reused for subsequent reads. Beware large files. -When transitioning to a finer-grained fuse api, this caching should be -replaced with straight-forward calls to the wapi. In my opinion, the -Tahoe node should do all the caching tricks, so that extensions such as -tahoe-fuse.py can be simple and thin. - -Backgrounding Behavior: -When using the 2.5-5build1 ubuntu package, and no other arguments -besides a mountpoint to tahoe-fuse.py, the process should remain in -the foreground and print debug information. Other python-fuse -versions appear to alter this behavior and may fork the process to -the background and obscure the log output. Bonus points to whomever -discovers the fate of these poor log messages in this case. - -"Investigative Logging": -This prototype is designed to aide in further fuse development, so -currently *every* fuse interface call figures out the process from -which the file system request originates, then it figures out that -processes commandline (this uses the /proc file system). This is handy -for interactive inspection of what kinds of behavior invokes which -file system operations, but may not work for you. To disable this -inspection, edit the source and comment out all of the "@debugcall" -[FIXME: double check python ref name] method decorators by inserting a -'#' so it looks like "#@debugcall" (without quotes). - -Not-to-spec: -The current version was not implemented according to any spec and -makes quite a few dubious "guesses" for what data to pass the fuse -interface. You may see bizarre values, which may potentialy confuse -any processes visiting the files under the mount point. - -Serial, blocking operations: -Most fuse operations result in one or more http calls to the WAPI. -These are serial and blocking (at least for the tested python-fuse -version 2.5-5build1), so access to this file system is quite -inefficient. - - -Good luck! diff --git a/contrib/fuse/runtests.py b/contrib/fuse/runtests.py deleted file mode 100644 index dca67006..00000000 --- a/contrib/fuse/runtests.py +++ /dev/null @@ -1,489 +0,0 @@ -#! /usr/bin/env python -''' -Unit and system tests for tahoe-fuse. -''' - -# Note: It's always a SetupFailure, not a TestFailure if a webapi -# operation fails, because this does not indicate a fuse interface -# failure. - -# TODO: Test mismatches between tahoe and fuse/posix. What about nodes -# with crazy names ('\0', unicode, '/', '..')? Huuuuge files? -# Huuuuge directories... As tahoe approaches production quality, it'd -# be nice if the fuse interface did so also by hardening against such cases. - -# FIXME: This framework might be replaceable with twisted.trial, -# especially the "layer" design, which is a bit cumbersome when -# using recursion to manage multiple clients. - -# FIXME: Identify all race conditions (hint: starting clients, versus -# using the grid fs). - -import sys, os, shutil, unittest, subprocess -import tempfile, re, time, signal, random, httplib -import traceback - -import tahoe_fuse - - -### Main flow control: -def main(args = sys.argv[1:]): - target = 'all' - if args: - if len(args) != 1: - raise SystemExit(Usage) - target = args[0] - - if target not in ('all', 'unit', 'system'): - raise SystemExit(Usage) - - if target in ('all', 'unit'): - run_unit_tests() - - if target in ('all', 'system'): - run_system_test() - - -def run_unit_tests(): - print 'Running Unit Tests.' - try: - unittest.main() - except SystemExit, se: - pass - print 'Unit Tests complete.\n' - - -def run_system_test(): - SystemTest().run() - - -### System Testing: -class SystemTest (object): - def __init__(self): - # These members represent configuration: - self.fullcleanup = False # FIXME: Make this a commandline option. - - # These members represent test state: - self.cliexec = None - self.testroot = None - - # This test state is specific to the first client: - self.port = None - self.clientbase = None - - ## Top-level flow control: - # These "*_layer" methods call eachother in a linear fashion, using - # exception unwinding to do cleanup properly. Each "layer" invokes - # a deeper layer, and each layer does its own cleanup upon exit. - - def run(self, fullcleanup = False): - ''' - If full_cleanup, delete all temporary state. - Else: If there is an error do not delete basedirs. - - Set to False if you wish to analyze a failure. - ''' - self.fullcleanup = fullcleanup - print '\n*** Setting up system tests.' - try: - failures, total = self.init_cli_layer() - print '\n*** System Tests complete: %d failed out of %d.' % (failures, total) - except self.SetupFailure, sfail: - print - print sfail - print '\n*** System Tests were not successfully completed.' - - def init_cli_layer(self): - '''This layer finds the appropriate tahoe executable.''' - runtestpath = os.path.abspath(sys.argv[0]) - path = runtestpath - for expectedname in ('runtests.py', 'fuse', 'contrib'): - path, name = os.path.split(path) - - if name != expectedname: - reason = 'Unexpected test script path: %r\n' - reason += 'The system test script must be run from the source directory.' - raise self.SetupFailure(reason, runtestpath) - - self.cliexec = os.path.join(path, 'bin', 'tahoe') - version = self.run_tahoe('--version') - print 'Using %r with version:\n%s' % (self.cliexec, version.rstrip()) - - return self.create_testroot_layer() - - def create_testroot_layer(self): - print 'Creating test base directory.' - self.testroot = tempfile.mkdtemp(prefix='tahoe_fuse_test_') - try: - return self.launch_introducer_layer() - finally: - if self.fullcleanup: - print 'Cleaning up test root directory.' - try: - shutil.rmtree(self.testroot) - except Exception, e: - print 'Exception removing test root directory: %r' % (self.testroot, ) - print 'Ignoring cleanup exception: %r' % (e,) - else: - print 'Leaving test root directory: %r' % (self.testroot, ) - - - def launch_introducer_layer(self): - print 'Launching introducer.' - introbase = os.path.join(self.testroot, 'introducer') - - # NOTE: We assume if tahoe exits with non-zero status, no separate - # tahoe child process is still running. - createoutput = self.run_tahoe('create-introducer', '--basedir', introbase) - - self.check_tahoe_output(createoutput, ExpectedCreationOutput, introbase) - - startoutput = self.run_tahoe('start', '--basedir', introbase) - try: - self.check_tahoe_output(startoutput, ExpectedStartOutput, introbase) - - return self.launch_clients_layer(introbase) - - finally: - print 'Stopping introducer node.' - self.stop_node(introbase) - - TotalClientsNeeded = 3 - def launch_clients_layer(self, introbase, clientnum = 1): - if clientnum > self.TotalClientsNeeded: - return self.create_test_dirnode_layer() - - tmpl = 'Launching client %d of %d.' - print tmpl % (clientnum, - self.TotalClientsNeeded) - - base = os.path.join(self.testroot, 'client_%d' % (clientnum,)) - - output = self.run_tahoe('create-client', '--basedir', base) - self.check_tahoe_output(output, ExpectedCreationOutput, base) - - if clientnum == 1: - # The first client is special: - self.clientbase = base - self.port = random.randrange(1024, 2**15) - - f = open(os.path.join(base, 'webport'), 'w') - f.write('tcp:%d:interface=127.0.0.1\n' % self.port) - f.close() - - introfurl = os.path.join(introbase, 'introducer.furl') - - self.polling_operation(lambda : os.path.isfile(introfurl)) - shutil.copy(introfurl, base) - - # NOTE: We assume if tahoe exist with non-zero status, no separate - # tahoe child process is still running. - startoutput = self.run_tahoe('start', '--basedir', base) - try: - self.check_tahoe_output(startoutput, ExpectedStartOutput, base) - - return self.launch_clients_layer(introbase, clientnum+1) - - finally: - print 'Stopping client node %d.' % (clientnum,) - self.stop_node(base) - - def create_test_dirnode_layer(self): - print 'Creating test dirnode.' - - cap = self.create_dirnode() - - f = open(os.path.join(self.clientbase, 'private', 'root_dir.cap'), 'w') - f.write(cap) - f.close() - - return self.mount_fuse_layer(cap) - - def mount_fuse_layer(self, fusebasecap, fusepause=2.0): - print 'Mounting fuse interface.' - - mp = os.path.join(self.testroot, 'mountpoint') - os.mkdir(mp) - - thispath = os.path.abspath(sys.argv[0]) - thisdir = os.path.dirname(thispath) - fusescript = os.path.join(thisdir, 'tahoe_fuse.py') - try: - proc = subprocess.Popen([fusescript, - mp, - '-f', - '--basedir', self.clientbase]) - - # The mount is verified by the test_layer, but we sleep to - # avoid race conditions against the first few tests. - time.sleep(fusepause) - - return self.run_test_layer(fusebasecap, mp) - - finally: - print '\n*** Cleaning up system test' - - if proc.poll() is None: - print 'Killing fuse interface.' - os.kill(proc.pid, signal.SIGTERM) - print 'Waiting for the fuse interface to exit.' - proc.wait() - - def run_test_layer(self, fbcap, mountpoint): - total = failures = 0 - for name in sorted(dir(self)): - if name.startswith('test_'): - total += 1 - print '\n*** Running test #%d: %s' % (total, name) - try: - testcap = self.create_dirnode() - self.attach_node(fbcap, testcap, name) - - method = getattr(self, name) - method(testcap, testdir = os.path.join(mountpoint, name)) - print 'Test succeeded.' - except self.TestFailure, f: - print f - failures += 1 - except: - print 'Error in test code... Cleaning up.' - raise - - return (failures, total) - - - # Tests: - def test_directory_existence(self, testcap, testdir): - if not os.path.isdir(testdir): - raise self.TestFailure('Attached test directory not found: %r', testdir) - - def test_empty_directory_listing(self, testcap, testdir): - listing = os.listdir(testdir) - if listing: - raise self.TestFailure('Expected empty directory, found: %r', listing) - - def test_directory_listing(self, testcap, testdir): - names = [] - filesizes = {} - - for i in range(3): - fname = 'file_%d' % (i,) - names.append(fname) - body = 'Hello World #%d!' % (i,) - filesizes[fname] = len(body) - - cap = self.webapi_call('PUT', '/uri', body) - self.attach_node(testcap, cap, fname) - - dname = 'dir_%d' % (i,) - names.append(dname) - - cap = self.create_dirnode() - self.attach_node(testcap, cap, dname) - - names.sort() - - listing = os.listdir(testdir) - listing.sort() - if listing != names: - tmpl = 'Expected directory list containing %r but fuse gave %r' - raise self.TestFailure(tmpl, names, listing) - - for file, size in filesizes.items(): - st = os.stat(os.path.join(testdir, file)) - if st.st_size != size: - tmpl = 'Expected %r size of %r but fuse returned %r' - raise self.TestFailure(tmpl, file, size, st.st_size) - - def test_file_contents(self, testcap, testdir): - name = 'hw.txt' - body = 'Hello World!' - - cap = self.webapi_call('PUT', '/uri', body) - self.attach_node(testcap, cap, name) - - path = os.path.join(testdir, name) - try: - found = open(path, 'r').read() - except Exception, err: - tmpl = 'Could not read file contents of %r: %r' - raise self.TestFailure(tmpl, path, err) - - if found != body: - tmpl = 'Expected file contents %r but found %r' - raise self.TestFailure(tmpl, body, found) - - - # Utilities: - def run_tahoe(self, *args): - realargs = ('tahoe',) + args - status, output = gather_output(realargs, executable=self.cliexec) - if status != 0: - tmpl = 'The tahoe cli exited with nonzero status.\n' - tmpl += 'Executable: %r\n' - tmpl += 'Command arguments: %r\n' - tmpl += 'Exit status: %r\n' - tmpl += 'Output:\n%s\n[End of tahoe output.]\n' - raise self.SetupFailure(tmpl, - self.cliexec, - realargs, - status, - output) - return output - - def check_tahoe_output(self, output, expected, expdir): - m = re.match(expected, output, re.M) - if m is None: - tmpl = 'The output of tahoe did not match the expectation:\n' - tmpl += 'Expected regex: %s\n' - tmpl += 'Actual output: %r\n' - self.warn(tmpl, expected, output) - - elif expdir != m.group('path'): - tmpl = 'The output of tahoe refers to an unexpected directory:\n' - tmpl += 'Expected directory: %r\n' - tmpl += 'Actual directory: %r\n' - self.warn(tmpl, expdir, m.group(1)) - - def stop_node(self, basedir): - try: - self.run_tahoe('stop', '--basedir', basedir) - except Exception, e: - print 'Failed to stop tahoe node.' - print 'Ignoring cleanup exception:' - # Indent the exception description: - desc = str(e).rstrip() - print ' ' + desc.replace('\n', '\n ') - - def webapi_call(self, method, path, body=None, **options): - if options: - path = path + '?' + ('&'.join(['%s=%s' % kv for kv in options.items()])) - - conn = httplib.HTTPConnection('127.0.0.1', self.port) - conn.request(method, path, body = body) - resp = conn.getresponse() - - if resp.status != 200: - tmpl = 'A webapi operation failed.\n' - tmpl += 'Request: %r %r\n' - tmpl += 'Body:\n%s\n' - tmpl += 'Response:\nStatus %r\nBody:\n%s' - raise self.SetupFailure(tmpl, - method, path, - body or '', - resp.status, body) - - return resp.read() - - def create_dirnode(self): - return self.webapi_call('PUT', '/uri', t='mkdir').strip() - - def attach_node(self, dircap, childcap, childname): - body = self.webapi_call('PUT', - '/uri/%s/%s' % (dircap, childname), - body = childcap, - t = 'uri', - replace = 'false') - assert body.strip() == childcap, `status, dircap, childcap, childname` - - def polling_operation(self, operation, timeout = 10.0, pollinterval = 0.2): - totaltime = timeout # Fudging for edge-case SetupFailure description... - - totalattempts = int(timeout / pollinterval) - - starttime = time.time() - for attempt in range(totalattempts): - opstart = time.time() - - try: - result = operation() - except KeyboardInterrupt, e: - raise - except Exception, e: - result = False - - totaltime = time.time() - starttime - - if result is not False: - #tmpl = '(Polling took over %.2f seconds.)' - #print tmpl % (totaltime,) - return result - - elif totaltime > timeout: - break - - else: - opdelay = time.time() - opstart - realinterval = max(0., pollinterval - opdelay) - - #tmpl = '(Poll attempt %d failed after %.2f seconds, sleeping %.2f seconds.)' - #print tmpl % (attempt+1, opdelay, realinterval) - time.sleep(realinterval) - - tmpl = 'Timeout after waiting for creation of introducer.furl.\n' - tmpl += 'Waited %.2f seconds (%d polls).' - raise self.SetupFailure(tmpl, totaltime, attempt+1) - - def warn(self, tmpl, *args): - print ('Test Warning: ' + tmpl) % args - - - # SystemTest Exceptions: - class Failure (Exception): - def __init__(self, tmpl, *args): - msg = self.Prefix + (tmpl % args) - Exception.__init__(self, msg) - - class SetupFailure (Failure): - Prefix = 'Setup Failure - The test framework encountered an error:\n' - - class TestFailure (Failure): - Prefix = 'TestFailure: ' - - -### Unit Tests: -class TestUtilFunctions (unittest.TestCase): - '''Tests small stand-alone functions.''' - def test_canonicalize_cap(self): - iopairs = [('http://127.0.0.1:8123/uri/URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y', - 'URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y'), - ('http://127.0.0.1:8123/uri/URI%3ACHK%3Ak7ktp1qr7szmt98s1y3ha61d9w%3A8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy%3A3%3A12%3A242?filename=welcome.html', - 'URI:CHK:k7ktp1qr7szmt98s1y3ha61d9w:8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy:3:12:242?filename=welcome.html')] - - for input, output in iopairs: - result = tahoe_fuse.canonicalize_cap(input) - self.failUnlessEqual(output, result, 'input == %r' % (input,)) - - - -### Misc: -def gather_output(*args, **kwargs): - ''' - This expects the child does not require input and that it closes - stdout/err eventually. - ''' - p = subprocess.Popen(stdout = subprocess.PIPE, - stderr = subprocess.STDOUT, - *args, - **kwargs) - output = p.stdout.read() - exitcode = p.wait() - return (exitcode, output) - - -ExpectedCreationOutput = r'(introducer|client) created in (?P.*?)\n' -ExpectedStartOutput = r'STARTING (?P.*?)\n(introducer|client) node probably started' - - -Usage = ''' -Usage: %s [target] - -Run tests for the given target. - -target is one of: unit, system, or all -''' % (sys.argv[0],) - - - -if __name__ == '__main__': - main() diff --git a/contrib/fuse/tahoe_fuse.py b/contrib/fuse/tahoe_fuse.py deleted file mode 100644 index f9485642..00000000 --- a/contrib/fuse/tahoe_fuse.py +++ /dev/null @@ -1,463 +0,0 @@ -#! /usr/bin/env python -''' -Tahoe thin-client fuse module. - -See the accompanying README for configuration/usage details. - -Goals: - -- Delegate to Tahoe webapi as much as possible. -- Thin rather than clever. (Even when that means clunky.) - - -Warts: - -- Reads cache entire file contents, violating the thinness goal. Can we GET spans of files? -- Single threaded. - - -Road-map: - -1. Add unit tests where possible with little code modification. -2. Make unit tests pass for a variety of python-fuse module versions. -3. Modify the design to make possible unit test coverage of larger portions of code. - -In parallel: -*. Make system tests which launch a client, mount a fuse interface, and excercise the whole stack. - -Wishlist: -- Reuse allmydata.uri to check/canonicalize uris. -- Research pkg_resources; see if it can replace the try-import-except-import-error pattern. -''' - - -#import bindann -#bindann.install_exception_handler() - -import sys, stat, os, errno, urllib, time - -try: - import simplejson -except ImportError, e: - raise SystemExit('''\ -Could not import simplejson, which is bundled with Tahoe. Please -update your PYTHONPATH environment variable to include the tahoe -"support/lib/python/site-packages" directory. -''') - - -try: - import fuse -except ImportError, e: - raise SystemExit('''\ -Could not import fuse, the pythonic fuse bindings. This dependency -of tahoe-fuse.py is *not* bundled with tahoe. Please install it. -On debian/ubuntu systems run: sudo apt-get install python-fuse -''') - -# FIXME: Check for non-working fuse versions here. -# FIXME: Make this work for all common python-fuse versions. - -# FIXME: Currently uses the old, silly path-based (non-stateful) interface: -fuse.fuse_python_api = (0, 1) # Use the silly path-based api for now. - - -### Config: -TahoeConfigDir = '~/.tahoe' -MagicDevNumber = 42 -UnknownSize = -1 - - -def main(): - basedir = os.path.expanduser(TahoeConfigDir) - - for i, arg in enumerate(sys.argv): - if arg == '--basedir': - try: - basedir = sys.argv[i+1] - sys.argv[i:i+2] = [] - except IndexError: - sys.argv = [sys.argv[0], '--help'] - - print 'DEBUG:', sys.argv - - fs = TahoeFS(basedir) - fs.main() - - -### Utilities for debug: -_logfile = None -def log(msg, *args): - global _logfile - if _logfile is None: - confdir = os.path.expanduser(TahoeConfigDir) - path = os.path.join(confdir, 'logs', 'tahoe_fuse.log') - _logfile = open(path, 'a') - _logfile.write('Log opened at: %s\n' % (time.strftime('%Y-%m-%d %H:%M:%S'),)) - _logfile.write((msg % args) + '\n') - _logfile.flush() - - -def trace_calls(m): - def dbmeth(self, *a, **kw): - pid = self.GetContext()['pid'] - log('[%d %r]\n%s%r%r', pid, get_cmdline(pid), m.__name__, a, kw) - try: - r = m(self, *a, **kw) - if (type(r) is int) and (r < 0): - log('-> -%s\n', errno.errorcode[-r],) - else: - repstr = repr(r)[:256] - log('-> %s\n', repstr) - return r - except: - sys.excepthook(*sys.exc_info()) - - return dbmeth - - -def get_cmdline(pid): - f = open('/proc/%d/cmdline' % pid, 'r') - args = f.read().split('\0') - f.close() - assert args[-1] == '' - return args[:-1] - - -class SystemError (Exception): - def __init__(self, eno): - self.eno = eno - Exception.__init__(self, errno.errorcode[eno]) - - @staticmethod - def wrap_returns(meth): - def wrapper(*args, **kw): - try: - return meth(*args, **kw) - except SystemError, e: - return -e.eno - wrapper.__name__ = meth.__name__ - return wrapper - - -### Heart of the Matter: -class TahoeFS (fuse.Fuse): - def __init__(self, confdir): - log('Initializing with confdir = %r', confdir) - fuse.Fuse.__init__(self) - self.confdir = confdir - - self.flags = 0 # FIXME: What goes here? - self.multithreaded = 0 - - # silly path-based file handles. - self.filecontents = {} # {path -> contents} - - self._init_url() - self._init_rootdir() - - def _init_url(self): - f = open(os.path.join(self.confdir, 'webport'), 'r') - contents = f.read() - f.close() - - fields = contents.split(':') - proto, port = fields[:2] - assert proto == 'tcp' - port = int(port) - self.url = 'http://localhost:%d' % (port,) - - def _init_rootdir(self): - # For now we just use the same default as the CLI: - rootdirfn = os.path.join(self.confdir, 'private', 'root_dir.cap') - try: - f = open(rootdirfn, 'r') - cap = f.read().strip() - f.close() - except EnvironmentError, le: - # FIXME: This user-friendly help message may be platform-dependent because it checks the exception description. - if le.args[1].find('No such file or directory') != -1: - raise SystemExit('%s requires a directory capability in %s, but it was not found.\nPlease see "The CLI" in "docs/using.html".\n' % (sys.argv[0], rootdirfn)) - else: - raise le - - self.rootdir = TahoeDir(self.url, canonicalize_cap(cap)) - - def _get_node(self, path): - assert path.startswith('/') - if path == '/': - return self.rootdir.resolve_path([]) - else: - parts = path.split('/')[1:] - return self.rootdir.resolve_path(parts) - - def _get_contents(self, path): - node = self._get_node(path) - contents = node.open().read() - self.filecontents[path] = contents - return contents - - @trace_calls - @SystemError.wrap_returns - def getattr(self, path): - node = self._get_node(path) - return node.getattr() - - @trace_calls - @SystemError.wrap_returns - def getdir(self, path): - """ - return: [(name, typeflag), ... ] - """ - node = self._get_node(path) - return node.getdir() - - @trace_calls - @SystemError.wrap_returns - def mythread(self): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def chmod(self, path, mode): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def chown(self, path, uid, gid): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def fsync(self, path, isFsyncFile): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def link(self, target, link): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def mkdir(self, path, mode): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def mknod(self, path, mode, dev_ignored): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def open(self, path, mode): - IgnoredFlags = os.O_RDONLY | os.O_NONBLOCK | os.O_SYNC | os.O_LARGEFILE - # Note: IgnoredFlags are all ignored! - for fname in dir(os): - if fname.startswith('O_'): - flag = getattr(os, fname) - if flag & IgnoredFlags: - continue - elif mode & flag: - log('Flag not supported: %s', fname) - raise SystemError(errno.ENOSYS) - - self._get_contents(path) - return 0 - - @trace_calls - @SystemError.wrap_returns - def read(self, path, length, offset): - return self._get_contents(path)[offset:length] - - @trace_calls - @SystemError.wrap_returns - def release(self, path): - del self.filecontents[path] - return 0 - - @trace_calls - @SystemError.wrap_returns - def readlink(self, path): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def rename(self, oldpath, newpath): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def rmdir(self, path): - return -errno.ENOSYS - - #@trace_calls - @SystemError.wrap_returns - def statfs(self): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def symlink ( self, targetPath, linkPath ): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def truncate(self, path, size): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def unlink(self, path): - return -errno.ENOSYS - - @trace_calls - @SystemError.wrap_returns - def utime(self, path, times): - return -errno.ENOSYS - - -class TahoeNode (object): - NextInode = 0 - - @staticmethod - def make(baseurl, uri): - typefield = uri.split(':', 2)[1] - # FIXME: is this check correct? - if uri.find('URI:DIR2') != -1: - return TahoeDir(baseurl, uri) - else: - return TahoeFile(baseurl, uri) - - def __init__(self, baseurl, uri): - self.burl = baseurl - self.uri = uri - self.fullurl = '%s/uri/%s' % (self.burl, self.uri) - self.inode = TahoeNode.NextInode - TahoeNode.NextInode += 1 - - def getattr(self): - """ - - st_mode (protection bits) - - st_ino (inode number) - - st_dev (device) - - st_nlink (number of hard links) - - st_uid (user ID of owner) - - st_gid (group ID of owner) - - st_size (size of file, in bytes) - - st_atime (time of most recent access) - - st_mtime (time of most recent content modification) - - st_ctime (platform dependent; time of most recent metadata change on Unix, - or the time of creation on Windows). - """ - # FIXME: Return metadata that isn't completely fabricated. - return (self.get_mode(), - self.inode, - MagicDevNumber, - self.get_linkcount(), - os.getuid(), - os.getgid(), - self.get_size(), - 0, - 0, - 0) - - def get_metadata(self): - f = self.open('?t=json') - json = f.read() - f.close() - return simplejson.loads(json) - - def open(self, postfix=''): - url = self.fullurl + postfix - log('*** Fetching: %r', url) - return urllib.urlopen(url) - - -class TahoeFile (TahoeNode): - def __init__(self, baseurl, uri): - #assert uri.split(':', 2)[1] in ('CHK', 'LIT'), `uri` # fails as of 0.7.0 - TahoeNode.__init__(self, baseurl, uri) - - # nonfuse: - def get_mode(self): - return stat.S_IFREG | 0400 # Read only regular file. - - def get_linkcount(self): - return 1 - - def get_size(self): - rawsize = self.get_metadata()[1]['size'] - if type(rawsize) is not int: # FIXME: What about sizes which do not fit in python int? - assert rawsize == u'?', `rawsize` - return UnknownSize - else: - return rawsize - - def resolve_path(self, path): - assert path == [] - return self - - -class TahoeDir (TahoeNode): - def __init__(self, baseurl, uri): - TahoeNode.__init__(self, baseurl, uri) - - self.mode = stat.S_IFDIR | 0500 # Read only directory. - - # FUSE: - def getdir(self): - d = [('.', self.get_mode()), ('..', self.get_mode())] - for name, child in self.get_children().items(): - if name: # Just ignore this crazy case! - d.append((name, child.get_mode())) - return d - - # nonfuse: - def get_mode(self): - return stat.S_IFDIR | 0500 # Read only directory. - - def get_linkcount(self): - return len(self.getdir()) - - def get_size(self): - return 2 ** 12 # FIXME: What do we return here? len(self.get_metadata()) - - def resolve_path(self, path): - assert type(path) is list - - if path: - head = path[0] - child = self.get_child(head) - return child.resolve_path(path[1:]) - else: - return self - - def get_child(self, name): - c = self.get_children() - return c[name] - - def get_children(self): - flag, md = self.get_metadata() - assert flag == 'dirnode' - - c = {} - for name, (childflag, childmd) in md['children'].items(): - if childflag == 'dirnode': - cls = TahoeDir - else: - cls = TahoeFile - - c[str(name)] = cls(self.burl, childmd['ro_uri']) - return c - - -def canonicalize_cap(cap): - cap = urllib.unquote(cap) - i = cap.find('URI:') - assert i != -1, 'A cap must contain "URI:...", but this does not: ' + cap - return cap[i:] - - -if __name__ == '__main__': - main() - diff --git a/contrib/fuse_a/README b/contrib/fuse_a/README new file mode 100644 index 00000000..f79107cc --- /dev/null +++ b/contrib/fuse_a/README @@ -0,0 +1,90 @@ + +Welcome to the tahoe fuse interface prototype! + + +Dependencies: + +In addition to a working tahoe installation, this interface depends +on the python-fuse interface. This package is available on Ubuntu +systems as "python-fuse". It is only known to work with ubuntu +package version "2.5-5build1". The latest ubuntu package (version +"1:0.2-pre3-3") appears to not work currently. + +Unfortunately this package appears poorly maintained (notice the wildy +different version strings and changing API semantics), so if you know +of a good replacement pythonic fuse interface, please let tahoe-dev know +about it! + + +Configuration: + +Currently tahoe-fuse.py uses the same ~/.tahoe/private/root_dir.cap +file (which is also the CLI default). This is not configurable yet. +Place a directory cap in this file. (Hint: If you can run "tahoe ls" +and see a directory listing, this file is properly configured.) + + +Commandline: + +The usage is "tahoe-fuse.py ". The mount point needs to +be an existing directory which should be empty. (If it's not empty +the contents will be safe, but unavailable while the tahoe-fuse.py +process is mounted there.) + + +Usage: + +To use the interface, use other programs to poke around the +mountpoint. You should be able to see the same contents as you would +by using the CLI or WUI for the same directory cap. + + +Runtime Behavior Notes: + +Read-only: +Only reading a tahoe grid is supported, which is reflected in +the permission modes. With Tahoe 0.7.0, write access should be easier +to implement, but is not yet present. + +In-Memory File Caching: +Currently requesting a particular file for read causes the entire file to +be retrieved into tahoe-fuse.py memory before the read operation returns! +This caching is reused for subsequent reads. Beware large files. +When transitioning to a finer-grained fuse api, this caching should be +replaced with straight-forward calls to the wapi. In my opinion, the +Tahoe node should do all the caching tricks, so that extensions such as +tahoe-fuse.py can be simple and thin. + +Backgrounding Behavior: +When using the 2.5-5build1 ubuntu package, and no other arguments +besides a mountpoint to tahoe-fuse.py, the process should remain in +the foreground and print debug information. Other python-fuse +versions appear to alter this behavior and may fork the process to +the background and obscure the log output. Bonus points to whomever +discovers the fate of these poor log messages in this case. + +"Investigative Logging": +This prototype is designed to aide in further fuse development, so +currently *every* fuse interface call figures out the process from +which the file system request originates, then it figures out that +processes commandline (this uses the /proc file system). This is handy +for interactive inspection of what kinds of behavior invokes which +file system operations, but may not work for you. To disable this +inspection, edit the source and comment out all of the "@debugcall" +[FIXME: double check python ref name] method decorators by inserting a +'#' so it looks like "#@debugcall" (without quotes). + +Not-to-spec: +The current version was not implemented according to any spec and +makes quite a few dubious "guesses" for what data to pass the fuse +interface. You may see bizarre values, which may potentialy confuse +any processes visiting the files under the mount point. + +Serial, blocking operations: +Most fuse operations result in one or more http calls to the WAPI. +These are serial and blocking (at least for the tested python-fuse +version 2.5-5build1), so access to this file system is quite +inefficient. + + +Good luck! diff --git a/contrib/fuse_a/runtests.py b/contrib/fuse_a/runtests.py new file mode 100644 index 00000000..dca67006 --- /dev/null +++ b/contrib/fuse_a/runtests.py @@ -0,0 +1,489 @@ +#! /usr/bin/env python +''' +Unit and system tests for tahoe-fuse. +''' + +# Note: It's always a SetupFailure, not a TestFailure if a webapi +# operation fails, because this does not indicate a fuse interface +# failure. + +# TODO: Test mismatches between tahoe and fuse/posix. What about nodes +# with crazy names ('\0', unicode, '/', '..')? Huuuuge files? +# Huuuuge directories... As tahoe approaches production quality, it'd +# be nice if the fuse interface did so also by hardening against such cases. + +# FIXME: This framework might be replaceable with twisted.trial, +# especially the "layer" design, which is a bit cumbersome when +# using recursion to manage multiple clients. + +# FIXME: Identify all race conditions (hint: starting clients, versus +# using the grid fs). + +import sys, os, shutil, unittest, subprocess +import tempfile, re, time, signal, random, httplib +import traceback + +import tahoe_fuse + + +### Main flow control: +def main(args = sys.argv[1:]): + target = 'all' + if args: + if len(args) != 1: + raise SystemExit(Usage) + target = args[0] + + if target not in ('all', 'unit', 'system'): + raise SystemExit(Usage) + + if target in ('all', 'unit'): + run_unit_tests() + + if target in ('all', 'system'): + run_system_test() + + +def run_unit_tests(): + print 'Running Unit Tests.' + try: + unittest.main() + except SystemExit, se: + pass + print 'Unit Tests complete.\n' + + +def run_system_test(): + SystemTest().run() + + +### System Testing: +class SystemTest (object): + def __init__(self): + # These members represent configuration: + self.fullcleanup = False # FIXME: Make this a commandline option. + + # These members represent test state: + self.cliexec = None + self.testroot = None + + # This test state is specific to the first client: + self.port = None + self.clientbase = None + + ## Top-level flow control: + # These "*_layer" methods call eachother in a linear fashion, using + # exception unwinding to do cleanup properly. Each "layer" invokes + # a deeper layer, and each layer does its own cleanup upon exit. + + def run(self, fullcleanup = False): + ''' + If full_cleanup, delete all temporary state. + Else: If there is an error do not delete basedirs. + + Set to False if you wish to analyze a failure. + ''' + self.fullcleanup = fullcleanup + print '\n*** Setting up system tests.' + try: + failures, total = self.init_cli_layer() + print '\n*** System Tests complete: %d failed out of %d.' % (failures, total) + except self.SetupFailure, sfail: + print + print sfail + print '\n*** System Tests were not successfully completed.' + + def init_cli_layer(self): + '''This layer finds the appropriate tahoe executable.''' + runtestpath = os.path.abspath(sys.argv[0]) + path = runtestpath + for expectedname in ('runtests.py', 'fuse', 'contrib'): + path, name = os.path.split(path) + + if name != expectedname: + reason = 'Unexpected test script path: %r\n' + reason += 'The system test script must be run from the source directory.' + raise self.SetupFailure(reason, runtestpath) + + self.cliexec = os.path.join(path, 'bin', 'tahoe') + version = self.run_tahoe('--version') + print 'Using %r with version:\n%s' % (self.cliexec, version.rstrip()) + + return self.create_testroot_layer() + + def create_testroot_layer(self): + print 'Creating test base directory.' + self.testroot = tempfile.mkdtemp(prefix='tahoe_fuse_test_') + try: + return self.launch_introducer_layer() + finally: + if self.fullcleanup: + print 'Cleaning up test root directory.' + try: + shutil.rmtree(self.testroot) + except Exception, e: + print 'Exception removing test root directory: %r' % (self.testroot, ) + print 'Ignoring cleanup exception: %r' % (e,) + else: + print 'Leaving test root directory: %r' % (self.testroot, ) + + + def launch_introducer_layer(self): + print 'Launching introducer.' + introbase = os.path.join(self.testroot, 'introducer') + + # NOTE: We assume if tahoe exits with non-zero status, no separate + # tahoe child process is still running. + createoutput = self.run_tahoe('create-introducer', '--basedir', introbase) + + self.check_tahoe_output(createoutput, ExpectedCreationOutput, introbase) + + startoutput = self.run_tahoe('start', '--basedir', introbase) + try: + self.check_tahoe_output(startoutput, ExpectedStartOutput, introbase) + + return self.launch_clients_layer(introbase) + + finally: + print 'Stopping introducer node.' + self.stop_node(introbase) + + TotalClientsNeeded = 3 + def launch_clients_layer(self, introbase, clientnum = 1): + if clientnum > self.TotalClientsNeeded: + return self.create_test_dirnode_layer() + + tmpl = 'Launching client %d of %d.' + print tmpl % (clientnum, + self.TotalClientsNeeded) + + base = os.path.join(self.testroot, 'client_%d' % (clientnum,)) + + output = self.run_tahoe('create-client', '--basedir', base) + self.check_tahoe_output(output, ExpectedCreationOutput, base) + + if clientnum == 1: + # The first client is special: + self.clientbase = base + self.port = random.randrange(1024, 2**15) + + f = open(os.path.join(base, 'webport'), 'w') + f.write('tcp:%d:interface=127.0.0.1\n' % self.port) + f.close() + + introfurl = os.path.join(introbase, 'introducer.furl') + + self.polling_operation(lambda : os.path.isfile(introfurl)) + shutil.copy(introfurl, base) + + # NOTE: We assume if tahoe exist with non-zero status, no separate + # tahoe child process is still running. + startoutput = self.run_tahoe('start', '--basedir', base) + try: + self.check_tahoe_output(startoutput, ExpectedStartOutput, base) + + return self.launch_clients_layer(introbase, clientnum+1) + + finally: + print 'Stopping client node %d.' % (clientnum,) + self.stop_node(base) + + def create_test_dirnode_layer(self): + print 'Creating test dirnode.' + + cap = self.create_dirnode() + + f = open(os.path.join(self.clientbase, 'private', 'root_dir.cap'), 'w') + f.write(cap) + f.close() + + return self.mount_fuse_layer(cap) + + def mount_fuse_layer(self, fusebasecap, fusepause=2.0): + print 'Mounting fuse interface.' + + mp = os.path.join(self.testroot, 'mountpoint') + os.mkdir(mp) + + thispath = os.path.abspath(sys.argv[0]) + thisdir = os.path.dirname(thispath) + fusescript = os.path.join(thisdir, 'tahoe_fuse.py') + try: + proc = subprocess.Popen([fusescript, + mp, + '-f', + '--basedir', self.clientbase]) + + # The mount is verified by the test_layer, but we sleep to + # avoid race conditions against the first few tests. + time.sleep(fusepause) + + return self.run_test_layer(fusebasecap, mp) + + finally: + print '\n*** Cleaning up system test' + + if proc.poll() is None: + print 'Killing fuse interface.' + os.kill(proc.pid, signal.SIGTERM) + print 'Waiting for the fuse interface to exit.' + proc.wait() + + def run_test_layer(self, fbcap, mountpoint): + total = failures = 0 + for name in sorted(dir(self)): + if name.startswith('test_'): + total += 1 + print '\n*** Running test #%d: %s' % (total, name) + try: + testcap = self.create_dirnode() + self.attach_node(fbcap, testcap, name) + + method = getattr(self, name) + method(testcap, testdir = os.path.join(mountpoint, name)) + print 'Test succeeded.' + except self.TestFailure, f: + print f + failures += 1 + except: + print 'Error in test code... Cleaning up.' + raise + + return (failures, total) + + + # Tests: + def test_directory_existence(self, testcap, testdir): + if not os.path.isdir(testdir): + raise self.TestFailure('Attached test directory not found: %r', testdir) + + def test_empty_directory_listing(self, testcap, testdir): + listing = os.listdir(testdir) + if listing: + raise self.TestFailure('Expected empty directory, found: %r', listing) + + def test_directory_listing(self, testcap, testdir): + names = [] + filesizes = {} + + for i in range(3): + fname = 'file_%d' % (i,) + names.append(fname) + body = 'Hello World #%d!' % (i,) + filesizes[fname] = len(body) + + cap = self.webapi_call('PUT', '/uri', body) + self.attach_node(testcap, cap, fname) + + dname = 'dir_%d' % (i,) + names.append(dname) + + cap = self.create_dirnode() + self.attach_node(testcap, cap, dname) + + names.sort() + + listing = os.listdir(testdir) + listing.sort() + if listing != names: + tmpl = 'Expected directory list containing %r but fuse gave %r' + raise self.TestFailure(tmpl, names, listing) + + for file, size in filesizes.items(): + st = os.stat(os.path.join(testdir, file)) + if st.st_size != size: + tmpl = 'Expected %r size of %r but fuse returned %r' + raise self.TestFailure(tmpl, file, size, st.st_size) + + def test_file_contents(self, testcap, testdir): + name = 'hw.txt' + body = 'Hello World!' + + cap = self.webapi_call('PUT', '/uri', body) + self.attach_node(testcap, cap, name) + + path = os.path.join(testdir, name) + try: + found = open(path, 'r').read() + except Exception, err: + tmpl = 'Could not read file contents of %r: %r' + raise self.TestFailure(tmpl, path, err) + + if found != body: + tmpl = 'Expected file contents %r but found %r' + raise self.TestFailure(tmpl, body, found) + + + # Utilities: + def run_tahoe(self, *args): + realargs = ('tahoe',) + args + status, output = gather_output(realargs, executable=self.cliexec) + if status != 0: + tmpl = 'The tahoe cli exited with nonzero status.\n' + tmpl += 'Executable: %r\n' + tmpl += 'Command arguments: %r\n' + tmpl += 'Exit status: %r\n' + tmpl += 'Output:\n%s\n[End of tahoe output.]\n' + raise self.SetupFailure(tmpl, + self.cliexec, + realargs, + status, + output) + return output + + def check_tahoe_output(self, output, expected, expdir): + m = re.match(expected, output, re.M) + if m is None: + tmpl = 'The output of tahoe did not match the expectation:\n' + tmpl += 'Expected regex: %s\n' + tmpl += 'Actual output: %r\n' + self.warn(tmpl, expected, output) + + elif expdir != m.group('path'): + tmpl = 'The output of tahoe refers to an unexpected directory:\n' + tmpl += 'Expected directory: %r\n' + tmpl += 'Actual directory: %r\n' + self.warn(tmpl, expdir, m.group(1)) + + def stop_node(self, basedir): + try: + self.run_tahoe('stop', '--basedir', basedir) + except Exception, e: + print 'Failed to stop tahoe node.' + print 'Ignoring cleanup exception:' + # Indent the exception description: + desc = str(e).rstrip() + print ' ' + desc.replace('\n', '\n ') + + def webapi_call(self, method, path, body=None, **options): + if options: + path = path + '?' + ('&'.join(['%s=%s' % kv for kv in options.items()])) + + conn = httplib.HTTPConnection('127.0.0.1', self.port) + conn.request(method, path, body = body) + resp = conn.getresponse() + + if resp.status != 200: + tmpl = 'A webapi operation failed.\n' + tmpl += 'Request: %r %r\n' + tmpl += 'Body:\n%s\n' + tmpl += 'Response:\nStatus %r\nBody:\n%s' + raise self.SetupFailure(tmpl, + method, path, + body or '', + resp.status, body) + + return resp.read() + + def create_dirnode(self): + return self.webapi_call('PUT', '/uri', t='mkdir').strip() + + def attach_node(self, dircap, childcap, childname): + body = self.webapi_call('PUT', + '/uri/%s/%s' % (dircap, childname), + body = childcap, + t = 'uri', + replace = 'false') + assert body.strip() == childcap, `status, dircap, childcap, childname` + + def polling_operation(self, operation, timeout = 10.0, pollinterval = 0.2): + totaltime = timeout # Fudging for edge-case SetupFailure description... + + totalattempts = int(timeout / pollinterval) + + starttime = time.time() + for attempt in range(totalattempts): + opstart = time.time() + + try: + result = operation() + except KeyboardInterrupt, e: + raise + except Exception, e: + result = False + + totaltime = time.time() - starttime + + if result is not False: + #tmpl = '(Polling took over %.2f seconds.)' + #print tmpl % (totaltime,) + return result + + elif totaltime > timeout: + break + + else: + opdelay = time.time() - opstart + realinterval = max(0., pollinterval - opdelay) + + #tmpl = '(Poll attempt %d failed after %.2f seconds, sleeping %.2f seconds.)' + #print tmpl % (attempt+1, opdelay, realinterval) + time.sleep(realinterval) + + tmpl = 'Timeout after waiting for creation of introducer.furl.\n' + tmpl += 'Waited %.2f seconds (%d polls).' + raise self.SetupFailure(tmpl, totaltime, attempt+1) + + def warn(self, tmpl, *args): + print ('Test Warning: ' + tmpl) % args + + + # SystemTest Exceptions: + class Failure (Exception): + def __init__(self, tmpl, *args): + msg = self.Prefix + (tmpl % args) + Exception.__init__(self, msg) + + class SetupFailure (Failure): + Prefix = 'Setup Failure - The test framework encountered an error:\n' + + class TestFailure (Failure): + Prefix = 'TestFailure: ' + + +### Unit Tests: +class TestUtilFunctions (unittest.TestCase): + '''Tests small stand-alone functions.''' + def test_canonicalize_cap(self): + iopairs = [('http://127.0.0.1:8123/uri/URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y', + 'URI:DIR2:yar9nnzsho6czczieeesc65sry:upp1pmypwxits3w9izkszgo1zbdnsyk3nm6h7e19s7os7s6yhh9y'), + ('http://127.0.0.1:8123/uri/URI%3ACHK%3Ak7ktp1qr7szmt98s1y3ha61d9w%3A8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy%3A3%3A12%3A242?filename=welcome.html', + 'URI:CHK:k7ktp1qr7szmt98s1y3ha61d9w:8tiy8drttp65u79pjn7hs31po83e514zifdejidyeo1ee8nsqfyy:3:12:242?filename=welcome.html')] + + for input, output in iopairs: + result = tahoe_fuse.canonicalize_cap(input) + self.failUnlessEqual(output, result, 'input == %r' % (input,)) + + + +### Misc: +def gather_output(*args, **kwargs): + ''' + This expects the child does not require input and that it closes + stdout/err eventually. + ''' + p = subprocess.Popen(stdout = subprocess.PIPE, + stderr = subprocess.STDOUT, + *args, + **kwargs) + output = p.stdout.read() + exitcode = p.wait() + return (exitcode, output) + + +ExpectedCreationOutput = r'(introducer|client) created in (?P.*?)\n' +ExpectedStartOutput = r'STARTING (?P.*?)\n(introducer|client) node probably started' + + +Usage = ''' +Usage: %s [target] + +Run tests for the given target. + +target is one of: unit, system, or all +''' % (sys.argv[0],) + + + +if __name__ == '__main__': + main() diff --git a/contrib/fuse_a/tahoe_fuse.py b/contrib/fuse_a/tahoe_fuse.py new file mode 100644 index 00000000..f9485642 --- /dev/null +++ b/contrib/fuse_a/tahoe_fuse.py @@ -0,0 +1,463 @@ +#! /usr/bin/env python +''' +Tahoe thin-client fuse module. + +See the accompanying README for configuration/usage details. + +Goals: + +- Delegate to Tahoe webapi as much as possible. +- Thin rather than clever. (Even when that means clunky.) + + +Warts: + +- Reads cache entire file contents, violating the thinness goal. Can we GET spans of files? +- Single threaded. + + +Road-map: + +1. Add unit tests where possible with little code modification. +2. Make unit tests pass for a variety of python-fuse module versions. +3. Modify the design to make possible unit test coverage of larger portions of code. + +In parallel: +*. Make system tests which launch a client, mount a fuse interface, and excercise the whole stack. + +Wishlist: +- Reuse allmydata.uri to check/canonicalize uris. +- Research pkg_resources; see if it can replace the try-import-except-import-error pattern. +''' + + +#import bindann +#bindann.install_exception_handler() + +import sys, stat, os, errno, urllib, time + +try: + import simplejson +except ImportError, e: + raise SystemExit('''\ +Could not import simplejson, which is bundled with Tahoe. Please +update your PYTHONPATH environment variable to include the tahoe +"support/lib/python/site-packages" directory. +''') + + +try: + import fuse +except ImportError, e: + raise SystemExit('''\ +Could not import fuse, the pythonic fuse bindings. This dependency +of tahoe-fuse.py is *not* bundled with tahoe. Please install it. +On debian/ubuntu systems run: sudo apt-get install python-fuse +''') + +# FIXME: Check for non-working fuse versions here. +# FIXME: Make this work for all common python-fuse versions. + +# FIXME: Currently uses the old, silly path-based (non-stateful) interface: +fuse.fuse_python_api = (0, 1) # Use the silly path-based api for now. + + +### Config: +TahoeConfigDir = '~/.tahoe' +MagicDevNumber = 42 +UnknownSize = -1 + + +def main(): + basedir = os.path.expanduser(TahoeConfigDir) + + for i, arg in enumerate(sys.argv): + if arg == '--basedir': + try: + basedir = sys.argv[i+1] + sys.argv[i:i+2] = [] + except IndexError: + sys.argv = [sys.argv[0], '--help'] + + print 'DEBUG:', sys.argv + + fs = TahoeFS(basedir) + fs.main() + + +### Utilities for debug: +_logfile = None +def log(msg, *args): + global _logfile + if _logfile is None: + confdir = os.path.expanduser(TahoeConfigDir) + path = os.path.join(confdir, 'logs', 'tahoe_fuse.log') + _logfile = open(path, 'a') + _logfile.write('Log opened at: %s\n' % (time.strftime('%Y-%m-%d %H:%M:%S'),)) + _logfile.write((msg % args) + '\n') + _logfile.flush() + + +def trace_calls(m): + def dbmeth(self, *a, **kw): + pid = self.GetContext()['pid'] + log('[%d %r]\n%s%r%r', pid, get_cmdline(pid), m.__name__, a, kw) + try: + r = m(self, *a, **kw) + if (type(r) is int) and (r < 0): + log('-> -%s\n', errno.errorcode[-r],) + else: + repstr = repr(r)[:256] + log('-> %s\n', repstr) + return r + except: + sys.excepthook(*sys.exc_info()) + + return dbmeth + + +def get_cmdline(pid): + f = open('/proc/%d/cmdline' % pid, 'r') + args = f.read().split('\0') + f.close() + assert args[-1] == '' + return args[:-1] + + +class SystemError (Exception): + def __init__(self, eno): + self.eno = eno + Exception.__init__(self, errno.errorcode[eno]) + + @staticmethod + def wrap_returns(meth): + def wrapper(*args, **kw): + try: + return meth(*args, **kw) + except SystemError, e: + return -e.eno + wrapper.__name__ = meth.__name__ + return wrapper + + +### Heart of the Matter: +class TahoeFS (fuse.Fuse): + def __init__(self, confdir): + log('Initializing with confdir = %r', confdir) + fuse.Fuse.__init__(self) + self.confdir = confdir + + self.flags = 0 # FIXME: What goes here? + self.multithreaded = 0 + + # silly path-based file handles. + self.filecontents = {} # {path -> contents} + + self._init_url() + self._init_rootdir() + + def _init_url(self): + f = open(os.path.join(self.confdir, 'webport'), 'r') + contents = f.read() + f.close() + + fields = contents.split(':') + proto, port = fields[:2] + assert proto == 'tcp' + port = int(port) + self.url = 'http://localhost:%d' % (port,) + + def _init_rootdir(self): + # For now we just use the same default as the CLI: + rootdirfn = os.path.join(self.confdir, 'private', 'root_dir.cap') + try: + f = open(rootdirfn, 'r') + cap = f.read().strip() + f.close() + except EnvironmentError, le: + # FIXME: This user-friendly help message may be platform-dependent because it checks the exception description. + if le.args[1].find('No such file or directory') != -1: + raise SystemExit('%s requires a directory capability in %s, but it was not found.\nPlease see "The CLI" in "docs/using.html".\n' % (sys.argv[0], rootdirfn)) + else: + raise le + + self.rootdir = TahoeDir(self.url, canonicalize_cap(cap)) + + def _get_node(self, path): + assert path.startswith('/') + if path == '/': + return self.rootdir.resolve_path([]) + else: + parts = path.split('/')[1:] + return self.rootdir.resolve_path(parts) + + def _get_contents(self, path): + node = self._get_node(path) + contents = node.open().read() + self.filecontents[path] = contents + return contents + + @trace_calls + @SystemError.wrap_returns + def getattr(self, path): + node = self._get_node(path) + return node.getattr() + + @trace_calls + @SystemError.wrap_returns + def getdir(self, path): + """ + return: [(name, typeflag), ... ] + """ + node = self._get_node(path) + return node.getdir() + + @trace_calls + @SystemError.wrap_returns + def mythread(self): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def chmod(self, path, mode): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def chown(self, path, uid, gid): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def fsync(self, path, isFsyncFile): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def link(self, target, link): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def mkdir(self, path, mode): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def mknod(self, path, mode, dev_ignored): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def open(self, path, mode): + IgnoredFlags = os.O_RDONLY | os.O_NONBLOCK | os.O_SYNC | os.O_LARGEFILE + # Note: IgnoredFlags are all ignored! + for fname in dir(os): + if fname.startswith('O_'): + flag = getattr(os, fname) + if flag & IgnoredFlags: + continue + elif mode & flag: + log('Flag not supported: %s', fname) + raise SystemError(errno.ENOSYS) + + self._get_contents(path) + return 0 + + @trace_calls + @SystemError.wrap_returns + def read(self, path, length, offset): + return self._get_contents(path)[offset:length] + + @trace_calls + @SystemError.wrap_returns + def release(self, path): + del self.filecontents[path] + return 0 + + @trace_calls + @SystemError.wrap_returns + def readlink(self, path): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def rename(self, oldpath, newpath): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def rmdir(self, path): + return -errno.ENOSYS + + #@trace_calls + @SystemError.wrap_returns + def statfs(self): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def symlink ( self, targetPath, linkPath ): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def truncate(self, path, size): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def unlink(self, path): + return -errno.ENOSYS + + @trace_calls + @SystemError.wrap_returns + def utime(self, path, times): + return -errno.ENOSYS + + +class TahoeNode (object): + NextInode = 0 + + @staticmethod + def make(baseurl, uri): + typefield = uri.split(':', 2)[1] + # FIXME: is this check correct? + if uri.find('URI:DIR2') != -1: + return TahoeDir(baseurl, uri) + else: + return TahoeFile(baseurl, uri) + + def __init__(self, baseurl, uri): + self.burl = baseurl + self.uri = uri + self.fullurl = '%s/uri/%s' % (self.burl, self.uri) + self.inode = TahoeNode.NextInode + TahoeNode.NextInode += 1 + + def getattr(self): + """ + - st_mode (protection bits) + - st_ino (inode number) + - st_dev (device) + - st_nlink (number of hard links) + - st_uid (user ID of owner) + - st_gid (group ID of owner) + - st_size (size of file, in bytes) + - st_atime (time of most recent access) + - st_mtime (time of most recent content modification) + - st_ctime (platform dependent; time of most recent metadata change on Unix, + or the time of creation on Windows). + """ + # FIXME: Return metadata that isn't completely fabricated. + return (self.get_mode(), + self.inode, + MagicDevNumber, + self.get_linkcount(), + os.getuid(), + os.getgid(), + self.get_size(), + 0, + 0, + 0) + + def get_metadata(self): + f = self.open('?t=json') + json = f.read() + f.close() + return simplejson.loads(json) + + def open(self, postfix=''): + url = self.fullurl + postfix + log('*** Fetching: %r', url) + return urllib.urlopen(url) + + +class TahoeFile (TahoeNode): + def __init__(self, baseurl, uri): + #assert uri.split(':', 2)[1] in ('CHK', 'LIT'), `uri` # fails as of 0.7.0 + TahoeNode.__init__(self, baseurl, uri) + + # nonfuse: + def get_mode(self): + return stat.S_IFREG | 0400 # Read only regular file. + + def get_linkcount(self): + return 1 + + def get_size(self): + rawsize = self.get_metadata()[1]['size'] + if type(rawsize) is not int: # FIXME: What about sizes which do not fit in python int? + assert rawsize == u'?', `rawsize` + return UnknownSize + else: + return rawsize + + def resolve_path(self, path): + assert path == [] + return self + + +class TahoeDir (TahoeNode): + def __init__(self, baseurl, uri): + TahoeNode.__init__(self, baseurl, uri) + + self.mode = stat.S_IFDIR | 0500 # Read only directory. + + # FUSE: + def getdir(self): + d = [('.', self.get_mode()), ('..', self.get_mode())] + for name, child in self.get_children().items(): + if name: # Just ignore this crazy case! + d.append((name, child.get_mode())) + return d + + # nonfuse: + def get_mode(self): + return stat.S_IFDIR | 0500 # Read only directory. + + def get_linkcount(self): + return len(self.getdir()) + + def get_size(self): + return 2 ** 12 # FIXME: What do we return here? len(self.get_metadata()) + + def resolve_path(self, path): + assert type(path) is list + + if path: + head = path[0] + child = self.get_child(head) + return child.resolve_path(path[1:]) + else: + return self + + def get_child(self, name): + c = self.get_children() + return c[name] + + def get_children(self): + flag, md = self.get_metadata() + assert flag == 'dirnode' + + c = {} + for name, (childflag, childmd) in md['children'].items(): + if childflag == 'dirnode': + cls = TahoeDir + else: + cls = TahoeFile + + c[str(name)] = cls(self.burl, childmd['ro_uri']) + return c + + +def canonicalize_cap(cap): + cap = urllib.unquote(cap) + i = cap.find('URI:') + assert i != -1, 'A cap must contain "URI:...", but this does not: ' + cap + return cap[i:] + + +if __name__ == '__main__': + main() + diff --git a/contrib/fuse_b/announce.txt b/contrib/fuse_b/announce.txt new file mode 100644 index 00000000..69db54be --- /dev/null +++ b/contrib/fuse_b/announce.txt @@ -0,0 +1,36 @@ +This announcement is archived in the tahoe-dev mailing list archive: + +http://allmydata.org/pipermail/tahoe-dev/2008-March/000465.html + +[tahoe-dev] Another FUSE interface +Armin Rigo arigo at tunes.org +Sat Mar 29 04:35:36 PDT 2008 + + * Previous message: [tahoe-dev] announcing allmydata.org "Tahoe", v1.0 + * Next message: [tahoe-dev] convergent encryption reconsidered -- salting and key-strengthening + * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] + +Hi all, + +I implemented for fun another Tahoe-to-FUSE interface using my own set +of FUSE bindings. If you are interested, you can check out the +following subversion directory: + + http://codespeak.net/svn/user/arigo/hack/pyfuse + +tahoe.py is a 100-lines, half-an-hour-job interface to Tahoe, limited to +read-only at the moment. The rest of the directory contains PyFuse, and +many other small usage examples. PyFuse is a pure Python FUSE daemon +(no messy linking issues, no dependencies). + + +A bientot, + +Armin Rigo + + * Previous message: [tahoe-dev] announcing allmydata.org "Tahoe", v1.0 + * Next message: [tahoe-dev] convergent encryption reconsidered -- salting and key-strengthening + * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] + +More information about the tahoe-dev mailing list +