* use new decentralized directories everywhere instead of old centralized directories
* provide UI to them through the web server
* provide UI to them through the CLI
* update unit tests to simulate decentralized mutable directories in order to test other components that rely on them
* remove the notion of a "vdrive server" and a client thereof
* remove the notion of a "public vdrive", which was a directory that was centrally published/subscribed automatically by the tahoe node (you can accomplish this manually by making a directory and posting the URL to it on your web site, for example)
* add a notion of "wait_for_numpeers" when you need to publish data to peers, which is how many peers should be attached before you start. The default is 1.
* add __repr__ for filesystem nodes (note: these reprs contain a few bits of the secret key!)
* fix a few bugs where we used to equate "mutable" with "not read-only". Nowadays all directories are mutable, but some might be read-only (to you).
* fix a few bugs where code wasn't aware of the new general-purpose metadata dict the comes with each filesystem edge
* sundry fixes to unit tests to adjust to the new directories, e.g. don't assume that every share on disk belongs to a chk file.
A brief map to where the code lives in this distribution:
src/allmydata: the code for this project. When installed, this provides the
- 'allmydata' package. This includes a few pieces copied from
+ 'allmydata' package. This includes a few pieces copied from
the PyCrypto package, in allmydata/Crypto/* .
Within src/allmydata/ :
node.py: the base Node, which handles connection establishment and
application startup
- client.py, introducer_and_vdrive.py:
+ client.py, introducer.py:
these are two specialized subclasses of Node, for users and the central
- introducer/vdrive handler, respectively. Each works by assembling a
- collection of services underneath a top-level Node instance.
+ introducer, respectively. Each works by assembling a collection of services
+ underneath a top-level Node instance.
- introducer.py: node introduction handlers, client is used by client.py,
- server is used by introducer_and_vdrive.py
+ introducer.py: node introduction handlers, client and server
storageserver.py: provides storage services to other nodes
download.py: download server selection, share retrieval, decoding
- dirnode.py: implements the directory nodes. One part runs on the
- global vdrive server, the other runs inside a client
- (starting with vdrive.py)
-
- vdrive.py: provides a client-side service that accesses the global
- shared virtual drive and the per-user private drive.
+ dirnode2.py: implements the distributed directory nodes.
webish.py, web/*.xhtml: provides the web frontend, using a Nevow server
test/*.py: unit tests
-Both the client and the central introducer-and-vdrive node runs as a tree of
+Both the client and the central introducer node runs as a tree of
(twisted.application.service) Service instances. The Foolscap "Tub" is one of
these. Client nodes have an Uploader service and a Downloader service that
-turn data into URIs and back again. They also have a VirtualDrive service
-which provides access to the single global shared filesystem.
+turn data into URIs and back again.
The Uploader is given an "upload source" (which could be an open filehandle,
a filename on local disk, or even a string), and returns a Deferred that
== Client Configuration ==
-introducer.furl and vdrive.furl (mandatory): These FURLs tell the client how
-to connect to the introducer/vdrive server. Each Tahoe grid is defined by
-this pair. They are created by the introducer/vdrive-server node and written
-into its base directory when it starts, whereupon they should be published to
-everyone who wishes to attach a client to that grid
+introducer.furl (mandatory): This FURL tells the client how to connect to the
+introducer. Each Tahoe grid is defined by an introducer. The introducer's
+furl is created by the introducer node and written into its base directory
+when it starts, whereupon it should be published to everyone who wishes to
+attach a client to that grid
webport (optional): This controls where the client's webserver should listen,
providing vdrive access as defined in webapi.txt . This file should contain a
symlink "authorized_keys.8022" to your ~/.ssh/authorized_keys file, and it
will accept the same keys as the rest of your account.
-sizelimit: If present, this file establishes an upper bound (in bytes) on the
-amount of storage consumed by share data (data that your node holds on behalf
-of clients that are uploading files to the grid). To avoid providing more
-than 100MB of data to other clients, write "100000000" into this file. Note
-that this is a fairly loose bound, and the node may occasionally use slightly
-more storage than this. To enforce a stronger (and possibly more reliable)
-limit, use a symlink to place the 'storage/' directory on a separate
-size-limited filesystem, and/or use per-user OS/filesystem quotas.
+sizelimit (optional): If present, this file establishes an upper bound (in
+bytes) on the amount of storage consumed by share data (data that your node
+holds on behalf of clients that are uploading files to the grid). To avoid
+providing more than 100MB of data to other clients, write "100000000" into
+this file. Note that this is a fairly loose bound, and the node may
+occasionally use slightly more storage than this. To enforce a stronger (and
+possibly more reliable) limit, use a symlink to place the 'storage/'
+directory on a separate size-limited filesystem, and/or use per-user
+OS/filesystem quotas.
+
+my_private_dir.uri (optional): When you create a new tahoe client, this
+file is created with no contents (as an empty file). When the node starts
+up, it will inspect this file. If the file doesn't exist then nothing will
+be done. If the file exists, then the node will try to read the contents of
+the file and parse the contents as a read-write URI to a mutable directory.
+If the file exists but doesn't contain a well-formed read-write URI to a
+mutable directory (which is the case if the file is empty), then the node
+will create a new decentralized mutable directory and write its URI into this
+file. The start.html page will contain a URL pointing to this directory if
+it exists.
+
== Node State ==
certificate allows the node to have a cryptographically-strong identifier
(the Foolscap "TubID"), and to establish secure connections to other nodes.
-global_root.uri: The first time the client contacts the vdrive-server, it
-retrieves the dirnode URI of the global root directory, and writes it into
-this file. On subsequent runs, this URI is used each time the user accesses
-the global vdrive.
-
-my_vdrive.uri: The first time the client contacts the vdrive-server, it will
-create a brand new directory to use as the non-shared private vdrive root,
-and it stores the dirnode URI of this directory in this file. On subsequent
-runs, it will read the URI from this file to provide access to the private
-vdrive.
-
storage/ : Nodes which host StorageServers will create this directory to hold
shares of files on behalf of other clients. There will be a directory
underneath it for each StorageIndex for which this node is holding shares.
centralized storage meshes to gather operational logs in a single place.
-== Introducer/vdrive-server configuration ==
+== Introducer configuration ==
-Introducer/vdrive-server nodes use the same 'advertised_ip_addresses' file
-as client nodes. They also use 'authorized_keys.SSHPORT'.
+Introducer nodes use the same 'advertised_ip_addresses' file as client
+nodes. They also use 'authorized_keys.SSHPORT'.
encoding_parameters (optional): This file sets the encoding parameters that
will be distributed to all client nodes and used when they encode files
The default value of encoding_parameters is "3 7 10".
-== Introducer/vdrive-server state ==
+== Introducer state ==
-The Introducer / Virtual-Drive Server node maintains some different state
-than regular client nodes. Both of these services are currently hosted inside
-the same node, although keeping the FURLs in separate files will make it
-easier to split these services in the future.
+The Introducer node maintains some different state than regular client
+nodes.
introducer.furl : This is generated the first time the introducer node is
started, and used again on subsequent runs, to give the introduction service
a persistent long-term identity. This file should be published and copied
into new client nodes before they are started for the first time.
-vdrive.furl : This is also generated the first time the node is started, and
-re-used on later runs. This FURL provides access to the vdrive service, used
-both to create+access all dirnodes and to learn about the global shared root
-vdrive.
-
introducer.port : this serves exactly the same purpose as 'client.port', but
has a different name to make it clear what kind of node is being run.
-vdrive/ : this directory is created by the vdrive service to hold the
-encrypted contents of dirnodes on behalf of all clients. It contains one file
-per dirnode, plus a file named 'root' which contains the binary storage index
-of the global shared root vdrive.
-
introducer.tac : this file is like client.tac but defines an
-introducer/vdrive-server node instead of a client node.
+introducer node instead of a client node.
== Other files ==
+++ /dev/null
-pb://xextf3eap44o3wi27mf7ehiur6wvhzr6@tahoecs.allmydata.com:56677/vdrive
=== Mutable File URIs ===
+TODO: update this documentation for v0.7.0 which does have decentralized mutable files and decentralized directories
The current release does not provide for mutable files, hence all file URIs
correspond to immutable data. Future releases will probably add mutable
files, creating a new class of Mutable File URIs. These URIs will contain the
layer) needs to keep track of "directory nodes", or "dirnodes" for short.
source:docs/dirnodes.txt describes how these work.
+TODO: update this documentation for v0.7.0 which has decentralized mutable files and decentralized directories
In the current release, each dirnode is stored (in encrypted form) on a
single "vdrive server". The Foolscap FURL that points at this server is kept
inside the "dirnode URI", as well as the read-key or write-key used in the
return os.urandom(20)
class Node:
- def __init__(self, nid, introducer_and_vdrive, simulator):
+ def __init__(self, nid, introducer, simulator):
self.nid = nid
- self.introducer_and_vdrive = introducer_and_vdrive
+ self.introducer = introducer
self.simulator = simulator
self.shares = {}
self.capacity = random.randrange(1000)
def permute_peers(self, fileid):
permuted = [(sha(fileid+n.nid),n)
- for n in self.introducer_and_vdrive.get_all_nodes()]
+ for n in self.introducer.get_all_nodes()]
permuted.sort()
return permuted
node.delete_share(fileid)
return False
self.files.append((fileid, numshares))
- self.introducer_and_vdrive.please_preserve(fileid, size, tried, last_givento)
+ self.introducer.please_preserve(fileid, size, tried, last_givento)
return (True, tried)
def accept_share(self, fileid, sharesize):
which = random.choice(self.files)
self.files.remove(which)
fileid,numshares = which
- self.introducer_and_vdrive.delete(fileid)
+ self.introducer.delete(fileid)
return True
class IntroducerAndVdrive:
self.rrd = RRD("/tmp/utilization.rrd", ds=[ds], rra=[rra], start=self.time)
self.rrd.create()
- self.introducer_and_vdrive = q = IntroducerAndVdrive(self)
+ self.introducer = q = IntroducerAndVdrive(self)
self.all_nodes = [Node(randomid(), q, self)
for i in range(self.NUM_NODES)]
q.all_nodes = self.all_nodes
avg_tried = "NONE"
else:
avg_tried = sum(self.published_files) / len(self.published_files)
- print time, etype, self.added_data, self.failed_files, self.lost_data_bytes, avg_tried, len(self.introducer_and_vdrive.living_files), self.introducer_and_vdrive.utilization
+ print time, etype, self.added_data, self.failed_files, self.lost_data_bytes, avg_tried, len(self.introducer.living_files), self.introducer.utilization
global s
s = None
c = SimpleDirnodeChecker(tub)
d = c.check(uri_to_check)
else:
- raise ValueError("I don't know how to check '%s'" % (uri_to_check,))
+ return defer.succeed(True) # TODO I don't know how to check, but I'm pretending to succeed.
def _done(res):
# TODO: handle exceptions too, record something useful about them
c = SimpleDirnodeChecker(tub)
return c.check(uri_to_verify)
else:
- raise ValueError("I don't know how to verify '%s'" %
- (uri_to_verify,))
+ return defer.succeed(True) # TODO I don't know how to verify, but I'm pretending to succeed.
def checker_results_for(self, uri_to_check):
if uri_to_check and self.results:
from allmydata.checker import Checker
from allmydata.control import ControlServer
from allmydata.introducer import IntroducerClient
-from allmydata.vdrive import VirtualDrive
-from allmydata.util import hashutil, idlib, testutil
-
-from allmydata.dirnode import FileNode
+from allmydata.util import hashutil, idlib, testutil, observer
+from allmydata.util.assertutil import precondition
+from allmydata.filenode import FileNode
from allmydata.dirnode2 import NewDirectoryNode
from allmydata.mutable import MutableFileNode
-from allmydata.interfaces import IURI, INewDirectoryURI, IDirnodeURI, \
- IFileURI, IMutableFileURI
+from allmydata.interfaces import IURI, INewDirectoryURI, \
+ IReadonlyNewDirectoryURI, IFileURI, IMutableFileURI
+from allmydata import uri
class Client(node.Node, Referenceable, testutil.PollMixin):
implements(RIClient)
STOREDIR = 'storage'
NODETYPE = "client"
SUICIDE_PREVENTION_HOTLINE_FILE = "suicide_prevention_hotline"
+ PRIVATE_DIRECTORY_URI = "my_private_dir.uri"
# we're pretty narrow-minded right now
OLDEST_SUPPORTED_VERSION = allmydata.__version__
self.add_service(Uploader())
self.add_service(Downloader())
self.add_service(Checker())
- self.add_service(VirtualDrive())
- webport = self.get_config("webport")
- if webport:
- self.init_web(webport) # strports string
+ self.private_directory_uri = None
+ self._private_uri_observers = None
+ self._start_page_observers = None
self.introducer_furl = self.get_config("introducer.furl", required=True)
hotline = TimerService(1.0, self._check_hotline, hotline_file)
hotline.setServiceParent(self)
+ webport = self.get_config("webport")
+ if webport:
+ self.init_web(webport) # strports string
+
+ def _init_start_page(self, privdiruri):
+ ws = self.getServiceNamed("webish")
+ startfile = os.path.join(self.basedir, "start.html")
+ nodeurl_file = os.path.join(self.basedir, "node.url")
+ return ws.create_start_html(privdiruri, startfile, nodeurl_file)
+
+ def init_start_page(self):
+ from twisted.internet import defer
+ defer.setDebugging(True)
+ if not self._start_page_observers:
+ self._start_page_observers = observer.OneShotObserverList()
+ d = self.get_private_uri()
+ d.addCallback(self._init_start_page)
+ d.addCallback(self._start_page_observers.fire)
+ d.addErrback(log.err)
+ return self._start_page_observers.when_fired()
+
def init_secret(self):
def make_secret():
return idlib.b2a(os.urandom(16)) + "\n"
if self.get_config("push_to_ourselves") is not None:
self.push_to_ourselves = True
+ def _maybe_create_private_directory(self):
+ """
+ If 'my_private_dir.uri' exists, then I try to read a mutable
+ directory URI from it. If it exists but doesn't contain a well-formed
+ read-write mutable directory URI, then I create a new mutable
+ directory and write its URI into that file.
+ """
+ privdirfile = os.path.join(self.basedir, self.PRIVATE_DIRECTORY_URI)
+ if os.path.exists(privdirfile):
+ try:
+ theuri = open(privdirfile, "r").read().strip()
+ if not uri.is_string_newdirnode_rw(theuri):
+ raise EnvironmentError("not a well-formed mutable directory uri")
+ except EnvironmentError, le:
+ d = self.when_tub_ready()
+ def _when_tub_ready(res):
+ return self.create_empty_dirnode(wait_for_numpeers=1)
+ d.addCallback(_when_tub_ready)
+ def _when_created(newdirnode):
+ log.msg("created new private directory: %s" % (newdirnode,))
+ privdiruri = newdirnode.get_uri()
+ self.private_directory_uri = privdiruri
+ open(privdirfile, "w+").write(privdiruri)
+ self._private_uri_observers.fire(privdiruri)
+ d.addCallback(_when_created)
+ d.addErrback(self._private_uri_observers.fire)
+ else:
+ self.private_directory_uri = theuri
+ log.msg("loaded private directory: %s" % (self.private_directory_uri,))
+ self._private_uri_observers.fire(self.private_directory_uri)
+ else:
+ # If there is no such file then this is how the node is configured
+ # to not create a private directory.
+ self._private_uri_observers.fire(None)
+
+ def get_private_uri(self):
+ """
+ Eventually fires with the URI (as a string) to this client's private
+ directory, or with None if this client has been configured not to
+ create one.
+ """
+ if self._private_uri_observers is None:
+ self._private_uri_observers = observer.OneShotObserverList()
+ self._maybe_create_private_directory()
+ return self._private_uri_observers.when_fired()
+
def init_web(self, webport):
+ self.log("init_web(webport=%s)", args=(webport,))
+
from allmydata.webish import WebishServer
- # this must be called after the VirtualDrive is attached
ws = WebishServer(webport)
if self.get_config("webport_allow_localfile") is not None:
ws.allow_local_access(True)
self.add_service(ws)
- vd = self.getServiceNamed("vdrive")
- startfile = os.path.join(self.basedir, "start.html")
- nodeurl_file = os.path.join(self.basedir, "node.url")
- d = vd.when_private_root_available()
- d.addCallback(ws.create_start_html, startfile, nodeurl_file)
-
+ self.init_start_page()
def _check_hotline(self, hotline_file):
if os.path.exists(hotline_file):
# dirnode. The other three create brand-new filenodes/dirnodes.
def create_node_from_uri(self, u):
- # this returns synchronously. As a result, it cannot be used to
- # create old-style dirnodes, since those contain a RemoteReference.
- # This means that new-style dirnodes cannot contain old-style
- # dirnodes as children.
+ # this returns synchronously.
u = IURI(u)
+ if IReadonlyNewDirectoryURI.providedBy(u):
+ # new-style read-only dirnodes
+ return NewDirectoryNode(self).init_from_uri(u)
if INewDirectoryURI.providedBy(u):
# new-style dirnodes
return NewDirectoryNode(self).init_from_uri(u)
- if IDirnodeURI.providedBy(u):
- ## handles old-style dirnodes, both mutable and immutable
- #return dirnode.create_directory_node(self, u)
- raise RuntimeError("not possible, sorry")
if IFileURI.providedBy(u):
# CHK
return FileNode(u, self)
- assert IMutableFileURI.providedBy(u)
+ assert IMutableFileURI.providedBy(u), u
return MutableFileNode(self).init_from_uri(u)
- def create_empty_dirnode(self):
+ def create_empty_dirnode(self, wait_for_numpeers):
n = NewDirectoryNode(self)
- d = n.create()
+ d = n.create(wait_for_numpeers=wait_for_numpeers)
d.addCallback(lambda res: n)
return d
- def create_mutable_file(self, contents=""):
+ def create_mutable_file(self, contents="", wait_for_numpeers=None):
n = MutableFileNode(self)
- d = n.create(contents)
+ d = n.create(contents, wait_for_numpeers=wait_for_numpeers)
d.addCallback(lambda res: n)
return d
- def upload(self, uploadable):
+ def upload(self, uploadable, wait_for_numpeers):
uploader = self.getServiceNamed("uploader")
- return uploader.upload(uploadable)
+ return uploader.upload(uploadable, wait_for_numpeers=wait_for_numpeers)
+++ /dev/null
-
-import os.path
-from zope.interface import implements
-from twisted.application import service
-from twisted.internet import defer
-from foolscap import Referenceable
-from allmydata import uri
-from allmydata.interfaces import RIVirtualDriveServer, \
- IDirectoryNode, IFileNode, IFileURI, IDirnodeURI, IURI, \
- BadWriteEnablerError, NotMutableError
-from allmydata.util import bencode, idlib, hashutil, fileutil
-from allmydata.Crypto.Cipher import AES
-
-# VirtualDriveServer is the side that hosts directory nodes
-
-class NoPublicRootError(Exception):
- pass
-
-class VirtualDriveServer(service.MultiService, Referenceable):
- implements(RIVirtualDriveServer)
- name = "filetable"
-
- def __init__(self, basedir, offer_public_root=True):
- service.MultiService.__init__(self)
- self._basedir = os.path.abspath(basedir)
- fileutil.make_dirs(self._basedir)
- self._root = None
- if offer_public_root:
- rootfile = os.path.join(self._basedir, "root")
- if not os.path.exists(rootfile):
- u = uri.DirnodeURI("fakefurl", hashutil.random_key())
- self.create_directory(u.storage_index, u.write_enabler)
- f = open(rootfile, "wb")
- f.write(u.writekey)
- f.close()
- self._root = u.writekey
- else:
- f = open(rootfile, "rb")
- self._root = f.read()
-
- def set_furl(self, myfurl):
- self._myfurl = myfurl
-
- def get_public_root_uri(self):
- if self._root:
- u = uri.DirnodeURI(self._myfurl, self._root)
- return u.to_string()
- raise NoPublicRootError
- remote_get_public_root_uri = get_public_root_uri
-
- def create_directory(self, index, write_enabler):
- data = [write_enabler, []]
- self._write_to_file(index, data)
- return index
- remote_create_directory = create_directory
-
- # the file on disk consists of the write_enabler token and a list of
- # (H(name), E(name), E(write), E(read)) tuples.
-
- def _read_from_file(self, index):
- name = idlib.b2a(index)
- data = open(os.path.join(self._basedir, name), "rb").read()
- return bencode.bdecode(data)
-
- def _write_to_file(self, index, data):
- name = idlib.b2a(index)
- f = open(os.path.join(self._basedir, name), "wb")
- f.write(bencode.bencode(data))
- f.close()
-
-
- def get(self, index, key):
- data = self._read_from_file(index)
- for (H_key, E_key, E_write, E_read) in data[1]:
- if H_key == key:
- return (E_write, E_read)
- raise KeyError("unable to find key %s" % idlib.b2a(key))
- remote_get = get
-
- def list(self, index):
- data = self._read_from_file(index)
- response = [ (E_key, E_write, E_read)
- for (H_key, E_key, E_write, E_read) in data[1] ]
- return response
- remote_list = list
-
- def delete(self, index, write_enabler, key):
- data = self._read_from_file(index)
- if data[0] != write_enabler:
- raise BadWriteEnablerError
- for i,(H_key, E_key, E_write, E_read) in enumerate(data[1]):
- if H_key == key:
- del data[1][i]
- self._write_to_file(index, data)
- return
- raise KeyError("unable to find key %s" % idlib.b2a(key))
- remote_delete = delete
-
- def set(self, index, write_enabler, key, name, write, read):
- data = self._read_from_file(index)
- if data[0] != write_enabler:
- raise BadWriteEnablerError
- # first, see if the key is already present
- for i,(H_key, E_key, E_write, E_read) in enumerate(data[1]):
- if H_key == key:
- # it is, we need to remove it first. Recurse to complete the
- # operation.
- self.delete(index, write_enabler, key)
- return self.set(index, write_enabler, key,
- name, write, read)
- # now just append the data
- data[1].append( (key, name, write, read) )
- self._write_to_file(index, data)
- remote_set = set
-
-# whereas ImmutableDirectoryNodes and their support mechanisms live on the
-# client side
-
-def create_directory_node(client, diruri):
- u = IURI(diruri)
- assert IDirnodeURI.providedBy(u)
- d = client.tub.getReference(u.furl)
- def _got(rref):
- if isinstance(u, uri.DirnodeURI):
- return MutableDirectoryNode(u, client, rref)
- else: # uri.ReadOnlyDirnodeURI
- return ImmutableDirectoryNode(u, client, rref)
- d.addCallback(_got)
- return d
-
-IV_LENGTH = 14
-def encrypt(key, data):
- IV = os.urandom(IV_LENGTH)
- counterstart = IV + "\x00"*(16-IV_LENGTH)
- assert len(counterstart) == 16, len(counterstart)
- cryptor = AES.new(key=key, mode=AES.MODE_CTR, counterstart=counterstart)
- crypttext = cryptor.encrypt(data)
- mac = hashutil.hmac(key, IV + crypttext)
- assert len(mac) == 32
- return IV + crypttext + mac
-
-class IntegrityCheckError(Exception):
- pass
-
-def decrypt(key, data):
- assert len(data) >= (32+IV_LENGTH), len(data)
- IV, crypttext, mac = data[:IV_LENGTH], data[IV_LENGTH:-32], data[-32:]
- if mac != hashutil.hmac(key, IV+crypttext):
- raise IntegrityCheckError("HMAC does not match, crypttext is corrupted")
- counterstart = IV + "\x00"*(16-IV_LENGTH)
- assert len(counterstart) == 16, len(counterstart)
- cryptor = AES.new(key=key, mode=AES.MODE_CTR, counterstart=counterstart)
- plaintext = cryptor.decrypt(crypttext)
- return plaintext
-
-
-class ImmutableDirectoryNode:
- implements(IDirectoryNode)
-
- def __init__(self, myuri, client, rref):
- u = IDirnodeURI(myuri)
- assert u.is_readonly()
- self._uri = u.to_string()
- self._client = client
- self._tub = client.tub
- self._rref = rref
-
- self._readkey = u.readkey
- self._writekey = u.writekey
- self._write_enabler = u.write_enabler
- self._index = u.storage_index
- self._mutable = False
-
- def dump(self):
- return ["URI: %s" % self._uri,
- "rk: %s" % idlib.b2a(self._readkey),
- "index: %s" % idlib.b2a(self._index),
- ]
-
- def is_mutable(self):
- return self._mutable
-
- def get_uri(self):
- return self._uri
-
- def get_immutable_uri(self):
- # return the dirnode URI for a read-only form of this directory
- return IDirnodeURI(self._uri).get_readonly().to_string()
-
- def __hash__(self):
- return hash((self.__class__, self._uri))
- def __cmp__(self, them):
- if cmp(type(self), type(them)):
- return cmp(type(self), type(them))
- if cmp(self.__class__, them.__class__):
- return cmp(self.__class__, them.__class__)
- return cmp(self._uri, them._uri)
-
- def _encrypt(self, key, data):
- return encrypt(key, data)
-
- def _decrypt(self, key, data):
- return decrypt(key, data)
-
- def _decrypt_child(self, E_write, E_read):
- if E_write and self._writekey:
- # we prefer read-write children when we can get them
- return self._decrypt(self._writekey, E_write)
- else:
- return self._decrypt(self._readkey, E_read)
-
- def list(self):
- d = self._rref.callRemote("list", self._index)
- entries = {}
- def _got(res):
- dl = []
- for (E_name, E_write, E_read) in res:
- name = self._decrypt(self._readkey, E_name)
- child_uri = self._decrypt_child(E_write, E_read)
- d2 = self._create_node(child_uri)
- def _created(node, name):
- entries[name] = node
- d2.addCallback(_created, name)
- dl.append(d2)
- return defer.DeferredList(dl)
- d.addCallback(_got)
- d.addCallback(lambda res: entries)
- return d
-
- def _hash_name(self, name):
- return hashutil.dir_name_hash(self._readkey, name)
-
- def has_child(self, name):
- d = self.get(name)
- def _good(res):
- return True
- def _err(f):
- f.trap(KeyError)
- return False
- d.addCallbacks(_good, _err)
- return d
-
- def get(self, name):
- H_name = self._hash_name(name)
- d = self._rref.callRemote("get", self._index, H_name)
- def _check_index_error(f):
- f.trap(KeyError)
- raise KeyError("get(index=%s): unable to find child named '%s'"
- % (idlib.b2a(self._index), name))
- d.addErrback(_check_index_error)
- d.addCallback(lambda (E_write, E_read):
- self._decrypt_child(E_write, E_read))
- d.addCallback(self._create_node)
- return d
-
- def _set(self, name, write_child, read_child):
- if not self._mutable:
- return defer.fail(NotMutableError())
- H_name = self._hash_name(name)
- E_name = self._encrypt(self._readkey, name)
- E_write = ""
- if self._writekey and write_child:
- assert isinstance(write_child, str)
- E_write = self._encrypt(self._writekey, write_child)
- assert isinstance(read_child, str)
- E_read = self._encrypt(self._readkey, read_child)
- d = self._rref.callRemote("set", self._index, self._write_enabler,
- H_name, E_name, E_write, E_read)
- return d
-
- def set_uri(self, name, child_uri):
- write, read = self._split_uri(child_uri)
- return self._set(name, write, read)
-
- def set_node(self, name, child):
- d = self.set_uri(name, child.get_uri())
- d.addCallback(lambda res: child)
- return d
-
- def delete(self, name):
- if not self._mutable:
- return defer.fail(NotMutableError())
- H_name = self._hash_name(name)
- d = self._rref.callRemote("delete", self._index, self._write_enabler,
- H_name)
- return d
-
- def _create_node(self, child_uri):
- u = IURI(child_uri)
- if IDirnodeURI.providedBy(u):
- return create_directory_node(self._client, u)
- else:
- return defer.succeed(self._client.create_node_from_uri(child_uri))
-
- def _split_uri(self, child_uri):
- u = IURI(child_uri)
- if u.is_mutable() and not u.is_readonly():
- write = u.to_string()
- else:
- write = None
- read = u.get_readonly().to_string()
- return (write, read)
-
- def create_empty_directory(self, name):
- if not self._mutable:
- return defer.fail(NotMutableError())
- child_writekey = hashutil.random_key()
- furl = IDirnodeURI(self._uri).furl
- u = uri.DirnodeURI(furl, child_writekey)
- child = MutableDirectoryNode(u, self._client, self._rref)
- d = self._rref.callRemote("create_directory",
- child._index, child._write_enabler)
- d.addCallback(lambda index: self.set_node(name, child))
- return d
-
- def add_file(self, name, uploadable):
- if not self._mutable:
- return defer.fail(NotMutableError())
- uploader = self._client.getServiceNamed("uploader")
- d = uploader.upload(uploadable)
- d.addCallback(lambda uri: self.set_node(name,
- FileNode(uri, self._client)))
- return d
-
- def move_child_to(self, current_child_name,
- new_parent, new_child_name=None):
- if not (self._mutable and new_parent.is_mutable()):
- return defer.fail(NotMutableError())
- if new_child_name is None:
- new_child_name = current_child_name
- d = self.get(current_child_name)
- d.addCallback(lambda child: new_parent.set_node(new_child_name, child))
- d.addCallback(lambda child: self.delete(current_child_name))
- return d
-
- def build_manifest(self):
- # given a dirnode, construct a frozenset of verifier-capabilities for
- # all the nodes it references.
-
- # this is just a tree-walker, except that following each edge
- # requires a Deferred.
-
- manifest = set()
- manifest.add(self.get_verifier())
-
- d = self._build_manifest_from_node(self, manifest)
- def _done(res):
- # LIT nodes have no verifier-capability: their data is stored
- # inside the URI itself, so there is no need to refresh anything.
- # They indicate this by returning None from their get_verifier
- # method. We need to remove any such Nones from our set. We also
- # want to convert all these caps into strings.
- return frozenset([cap.to_string()
- for cap in manifest
- if cap is not None])
- d.addCallback(_done)
- return d
-
- def _build_manifest_from_node(self, node, manifest):
- d = node.list()
- def _got_list(res):
- dl = []
- for name, child in res.iteritems():
- verifier = child.get_verifier()
- if verifier not in manifest:
- manifest.add(verifier)
- if IDirectoryNode.providedBy(child):
- dl.append(self._build_manifest_from_node(child,
- manifest))
- if dl:
- return defer.DeferredList(dl)
- d.addCallback(_got_list)
- return d
-
- def get_verifier(self):
- return IDirnodeURI(self._uri).get_verifier()
-
- def check(self):
- verifier = self.get_verifier()
- return self._client.getServiceNamed("checker").check(verifier)
-
- def get_child_at_path(self, path):
- if not path:
- return defer.succeed(self)
- if isinstance(path, (str, unicode)):
- path = path.split("/")
- childname = path[0]
- remaining_path = path[1:]
- d = self.get(childname)
- if remaining_path:
- def _got(node):
- return node.get_child_at_path(remaining_path)
- d.addCallback(_got)
- return d
-
-class MutableDirectoryNode(ImmutableDirectoryNode):
- implements(IDirectoryNode)
-
- def __init__(self, myuri, client, rref):
- u = IDirnodeURI(myuri)
- assert not u.is_readonly()
- self._uri = u.to_string()
- self._client = client
- self._tub = client.tub
- self._rref = rref
-
- self._readkey = u.readkey
- self._writekey = u.writekey
- self._write_enabler = u.write_enabler
- self._index = u.storage_index
- self._mutable = True
-
-def create_directory(client, furl):
- u = uri.DirnodeURI(furl, hashutil.random_key())
- d = client.tub.getReference(furl)
- def _got_vdrive_server(vdrive_server):
- node = MutableDirectoryNode(u, client, vdrive_server)
- d2 = vdrive_server.callRemote("create_directory",
- u.storage_index, u.write_enabler)
- d2.addCallback(lambda res: node)
- return d2
- d.addCallback(_got_vdrive_server)
- return d
-
-class FileNode:
- implements(IFileNode)
-
- def __init__(self, uri, client):
- u = IFileURI(uri)
- self.uri = u.to_string()
- self._client = client
-
- def get_uri(self):
- return self.uri
-
- def is_readonly(self):
- return True
-
- def get_size(self):
- return IFileURI(self.uri).get_size()
-
- def __hash__(self):
- return hash((self.__class__, self.uri))
- def __cmp__(self, them):
- if cmp(type(self), type(them)):
- return cmp(type(self), type(them))
- if cmp(self.__class__, them.__class__):
- return cmp(self.__class__, them.__class__)
- return cmp(self.uri, them.uri)
-
- def get_verifier(self):
- return IFileURI(self.uri).get_verifier()
-
- def check(self):
- verifier = self.get_verifier()
- return self._client.getServiceNamed("checker").check(verifier)
-
- def download(self, target):
- downloader = self._client.getServiceNamed("downloader")
- return downloader.download(self.uri, target)
-
- def download_to_data(self):
- downloader = self._client.getServiceNamed("downloader")
- return downloader.download_to_data(self.uri)
-
from zope.interface import implements
from twisted.internet import defer
+from twisted.python import log
import simplejson
+from allmydata.mutable import NotMutableError
from allmydata.interfaces import IMutableFileNode, IDirectoryNode,\
- INewDirectoryURI, IFileNode, NotMutableError, \
+ INewDirectoryURI, IURI, IFileNode, \
IVerifierURI
from allmydata.util import hashutil
from allmydata.util.hashutil import netstring
-from allmydata.dirnode import IntegrityCheckError
from allmydata.uri import NewDirectoryURI
from allmydata.Crypto.Cipher import AES
def __init__(self, client):
self._client = client
+ def __repr__(self):
+ return "<%s %s %s>" % (self.__class__.__name__, self.is_readonly() and "RO" or "RW", hasattr(self, '_uri') and self._uri.abbrev())
def init_from_uri(self, myuri):
- u = INewDirectoryURI(myuri)
- self._uri = u
+ self._uri = IURI(myuri)
self._node = self.filenode_class(self._client)
- self._node.init_from_uri(u.get_filenode_uri())
+ self._node.init_from_uri(self._uri.get_filenode_uri())
return self
- def create(self):
+ def create(self, wait_for_numpeers=None):
+ """
+ Returns a deferred that eventually fires with self once the directory
+ has been created (distributed across a set of storage servers).
+ """
# first we create a MutableFileNode with empty_contents, then use its
# URI to create our own.
self._node = self.filenode_class(self._client)
empty_contents = self._pack_contents({})
- d = self._node.create(empty_contents)
+ d = self._node.create(empty_contents, wait_for_numpeers=wait_for_numpeers)
d.addCallback(self._filenode_created)
return d
def _filenode_created(self, res):
self._uri = NewDirectoryURI(self._node._uri)
- return None
+ return self
def _read(self):
d = self._node.download_to_data()
mac = encwrcap[-32:]
key = hashutil.mutable_rwcap_key_hash(IV, self._node.get_writekey())
if mac != hashutil.hmac(key, IV+crypttext):
- raise IntegrityCheckError("HMAC does not match, crypttext is corrupted")
+ raise hashutil.IntegrityCheckError("HMAC does not match, crypttext is corrupted")
counterstart = "\x00"*16
cryptor = AES.new(key=key, mode=AES.MODE_CTR, counterstart=counterstart)
plaintext = cryptor.decrypt(crypttext)
# an empty directory is serialized as an empty string
if data == "":
return {}
- mutable = self.is_mutable()
+ writeable = not self.is_readonly()
children = {}
while len(data) > 0:
entry, data = split_netstring(data, 1, True)
name, rocap, rwcapdata, metadata_s = split_netstring(entry, 4)
- if mutable:
+ if writeable:
rwcap = self._decrypt_rwcapdata(rwcapdata)
child = self._create_node(rwcap)
else:
child, metadata = children[name]
assert (IFileNode.providedBy(child)
or IMutableFileNode.providedBy(child)
- or IDirectoryNode.providedBy(child))
+ or IDirectoryNode.providedBy(child)), children
assert isinstance(metadata, dict)
- rwcap = child.get_uri() # might be RO if the child is not mutable
- rocap = child.get_readonly()
+ rwcap = child.get_uri() # might be RO if the child is not writeable
+ rocap = child.get_readonly_uri()
entry = "".join([netstring(name),
netstring(rocap),
netstring(self._encrypt_rwcap(rwcap)),
def get_uri(self):
return self._uri.to_string()
- def get_readonly(self):
- return self._uri.get_readonly().to_string()
-
- def get_immutable_uri(self):
+ def get_readonly_uri(self):
return self._uri.get_readonly().to_string()
def get_verifier(self):
def list(self):
"""I return a Deferred that fires with a dictionary mapping child
- name to an IFileNode or IDirectoryNode."""
+ name to a tuple of (IFileNode or IDirectoryNode, metadata)."""
return self._read()
def has_child(self, name):
d.addCallback(lambda children: children.has_key(name))
return d
+ def _get(self, children, name):
+ child = children.get(name)
+ if child is None:
+ raise KeyError(name)
+ return child[0]
+
def get(self, name):
- """I return a Deferred that fires with a specific named child node,
- either an IFileNode or an IDirectoryNode."""
+ """I return a Deferred that fires with the named child node,
+ which is either an IFileNode or an IDirectoryNode."""
d = self._read()
- d.addCallback(lambda children: children[name][0])
+ d.addCallback(self._get, name)
return d
def get_metadata_for(self, name):
d.addCallback(_got)
return d
- def set_uri(self, name, child_uri, metadata={}):
+ def set_uri(self, name, child_uri, metadata={}, wait_for_numpeers=None):
"""I add a child (by URI) at the specific name. I return a Deferred
- that fires when the operation finishes. I will replace any existing
- child of the same name.
+ that fires with the child node when the operation finishes. I will
+ replace any existing child of the same name.
The child_uri could be for a file, or for a directory (either
read-write or read-only, using a URI that came from get_uri() ).
If this directory node is read-only, the Deferred will errback with a
NotMutableError."""
- return self.set_node(name, self._create_node(child_uri), metadata)
+ return self.set_node(name, self._create_node(child_uri), metadata, wait_for_numpeers=wait_for_numpeers)
- def set_node(self, name, child, metadata={}):
+ def set_node(self, name, child, metadata={}, wait_for_numpeers=None):
"""I add a child at the specific name. I return a Deferred that fires
when the operation finishes. This Deferred will fire with the child
node that was just added. I will replace any existing child of the
def _add(children):
children[name] = (child, metadata)
new_contents = self._pack_contents(children)
- return self._node.replace(new_contents)
+ return self._node.replace(new_contents, wait_for_numpeers=wait_for_numpeers)
d.addCallback(_add)
- d.addCallback(lambda res: None)
+ d.addCallback(lambda res: child)
return d
- def add_file(self, name, uploadable):
+ def add_file(self, name, uploadable, wait_for_numpeers=None):
"""I upload a file (using the given IUploadable), then attach the
resulting FileNode to the directory at the given name. I return a
Deferred that fires (with the IFileNode of the uploaded file) when
the operation completes."""
if self.is_readonly():
return defer.fail(NotMutableError())
- d = self._client.upload(uploadable)
+ d = self._client.upload(uploadable, wait_for_numpeers=wait_for_numpeers)
d.addCallback(self._client.create_node_from_uri)
- d.addCallback(lambda node: self.set_node(name, node))
+ d.addCallback(lambda node: self.set_node(name, node, wait_for_numpeers=wait_for_numpeers))
return d
def delete(self, name):
d.addCallback(_delete)
return d
- def create_empty_directory(self, name):
+ def create_empty_directory(self, name, wait_for_numpeers=None):
"""I create and attach an empty directory at the given name. I return
a Deferred that fires (with the new directory node) when the
operation finishes."""
if self.is_readonly():
return defer.fail(NotMutableError())
- d = self._client.create_empty_dirnode()
+ d = self._client.create_empty_dirnode(wait_for_numpeers=wait_for_numpeers)
def _created(child):
- d = self.set_node(name, child)
+ d = self.set_node(name, child, wait_for_numpeers=wait_for_numpeers)
d.addCallback(lambda res: child)
return d
d.addCallback(_created)
return d
def move_child_to(self, current_child_name, new_parent,
- new_child_name=None):
+ new_child_name=None, wait_for_numpeers=None):
"""I take one of my children and move them to a new parent. The child
is referenced by name. On the new parent, the child will live under
'new_child_name', which defaults to 'current_child_name'. I return a
if new_child_name is None:
new_child_name = current_child_name
d = self.get(current_child_name)
- d.addCallback(lambda child: new_parent.set_node(new_child_name, child))
+ def sn(child):
+ return new_parent.set_node(new_child_name, child,
+ wait_for_numpeers=wait_for_numpeers)
+ d.addCallback(sn)
d.addCallback(lambda child: self.delete(current_child_name))
return d
--- /dev/null
+
+from zope.interface import implements
+from allmydata.interfaces import IFileNode, IFileURI
+
+class FileNode:
+ implements(IFileNode)
+
+ def __init__(self, uri, client):
+ u = IFileURI(uri)
+ self.uri = u.to_string()
+ self._client = client
+
+ def get_uri(self):
+ return self.uri
+
+ def is_readonly(self):
+ return True
+
+ def get_readonly_uri(self):
+ return self.uri
+
+ def get_size(self):
+ return IFileURI(self.uri).get_size()
+
+ def __hash__(self):
+ return hash((self.__class__, self.uri))
+ def __cmp__(self, them):
+ if cmp(type(self), type(them)):
+ return cmp(type(self), type(them))
+ if cmp(self.__class__, them.__class__):
+ return cmp(self.__class__, them.__class__)
+ return cmp(self.uri, them.uri)
+
+ def get_verifier(self):
+ return IFileURI(self.uri).get_verifier()
+
+ def check(self):
+ verifier = self.get_verifier()
+ return self._client.getServiceNamed("checker").check(verifier)
+
+ def download(self, target):
+ downloader = self._client.getServiceNamed("downloader")
+ return downloader.download(self.uri, target)
+
+ def download_to_data(self):
+ downloader = self._client.getServiceNamed("downloader")
+ return downloader.download_to_data(self.uri)
+
# hm, we need a solution for forward references in schemas
from foolscap.schema import Any
-RIMutableDirectoryNode_ = Any() # TODO: how can we avoid this?
FileNode_ = Any() # TODO: foolscap needs constraints on copyables
DirectoryNode_ = Any() # TODO: same
AnyNode_ = ChoiceOf(FileNode_, DirectoryNode_)
EncryptedThing = str
-class RIVirtualDriveServer(RemoteInterface):
- def get_public_root_uri():
- """Obtain the URI for this server's global publically-writable root
- directory. This returns a read-write directory URI.
-
- If this vdrive server does not offer a public root, this will
- raise an exception."""
- return DirnodeURI
-
- def create_directory(index=Hash, write_enabler=Hash):
- """Create a new (empty) directory, unattached to anything else.
-
- This returns the same index that was passed in.
- """
- return Hash
-
- def get(index=Hash, key=Hash):
- """Retrieve a named child of the given directory. 'index' specifies
- which directory is being accessed, and is generally the hash of the
- read key. 'key' is the hash of the read key and the child name.
-
- This operation returns a pair of encrypted strings. The first string
- is meant to be decrypted by the Write Key and provides read-write
- access to the child. If this directory holds read-only access to the
- child, this first string will be an empty string. The second string
- is meant to be decrypted by the Read Key and provides read-only
- access to the child.
-
- When the child is a read-write directory, the encrypted URI:DIR-RO
- will be in the read slot, and the encrypted URI:DIR will be in the
- write slot. When the child is a read-only directory, the encrypted
- URI:DIR-RO will be in the read slot and the write slot will be empty.
- When the child is a CHK file, the encrypted URI:CHK will be in the
- read slot, and the write slot will be empty.
-
- This might raise IndexError if there is no child by the desired name.
- """
- return (EncryptedThing, EncryptedThing)
-
- def list(index=Hash):
- """List the contents of a directory.
-
- This returns a list of (NAME, WRITE, READ) tuples. Each value is an
- encrypted string (although the WRITE value may sometimes be an empty
- string).
-
- NAME: the child name, encrypted with the Read Key
- WRITE: the child write URI, encrypted with the Write Key, or an
- empty string if this child is read-only
- READ: the child read URI, encrypted with the Read Key
- """
- return ListOf((EncryptedThing, EncryptedThing, EncryptedThing),
- maxLength=1000,
- )
-
- def set(index=Hash, write_enabler=Hash, key=Hash,
- name=EncryptedThing, write=EncryptedThing, read=EncryptedThing):
- """Set a child object. I will replace any existing child of the same
- name.
- """
-
- def delete(index=Hash, write_enabler=Hash, key=Hash):
- """Delete a specific child.
-
- This uses the hashed key to locate a specific child, and deletes it.
- """
-
-
class IURI(Interface):
def init_from_string(uri):
"""Accept a string (as created by my to_string() method) and populate
pass
class INewDirectoryURI(Interface):
pass
+class IReadonlyNewDirectoryURI(Interface):
+ pass
-class IFileNode(Interface):
- def download(target):
- """Download the file's contents to a given IDownloadTarget"""
- def download_to_data():
- """Download the file's contents. Return a Deferred that fires
- with those contents."""
-
+class IFilesystemNode(Interface):
def get_uri():
"""Return the URI that can be used by others to get access to this
- file.
+ file or directory.
"""
- def get_size():
- """Return the length (in bytes) of the data this node represents."""
def get_verifier():
"""Return an IVerifierURI instance that represents the
def check():
"""Perform a file check. See IChecker.check for details."""
-class IMutableFileNode(IFileNode):
+ def is_mutable():
+ """Return True if this file or directory is mutable, False if it is read-only.
+ """
+
+class IMutableFilesystemNode(IFilesystemNode):
+ def get_uri():
+ """
+ Return the URI that can be used by others to get access to this
+ node. If this node is read-only, the URI will only offer read-only
+ access. If this node is read-write, the URI will offer read-write
+ access.
+
+ If you have read-write access to a node and wish to share merely
+ read-only access with others, use get_readonly_uri().
+ """
+
+ def get_readonly_uri():
+ """Return the directory URI that can be used by others to get
+ read-only access to this directory node. The result is a read-only
+ URI, regardless of whether this dirnode is read-only or read-write.
+
+ If you have merely read-only access to this dirnode,
+ get_readonly_uri() will return the same thing as get_uri().
+ """
+
+class IFileNode(IFilesystemNode):
+ def download(target):
+ """Download the file's contents to a given IDownloadTarget"""
+
+ def download_to_data():
+ """Download the file's contents. Return a Deferred that fires
+ with those contents."""
+
+ def get_size():
+ """Return the length (in bytes) of the data this node represents."""
+
+class IMutableFileNode(IFileNode, IMutableFilesystemNode):
def download_to_data():
"""Download the file's contents. Return a Deferred that fires with
those contents. If there are multiple retrievable versions in the
reconstruct, and will silently ignore the others. In the future, a
more advanced API will signal and provide access to the multiple
heads."""
- def replace(newdata):
+
+ def replace(newdata, wait_for_numpeers=None):
"""Replace the old contents with the new data. Returns a Deferred
that fires (with None) when the operation is complete.
the replace() operation.
"""
- def get_uri():
- pass
- def get_verifier():
- pass
- def check():
- pass
-
def get_writekey():
"""Return this filenode's writekey, or None if the node does not have
write-capability. This may be used to assist with data structures
writer-visible data using this writekey.
"""
-class IDirectoryNode(Interface):
- def is_mutable():
- """Return True if this directory is mutable, False if it is read-only.
- """
-
+class IDirectoryNode(IMutableFilesystemNode):
def get_uri():
- """Return the directory URI that can be used by others to get access
- to this directory node. If this node is read-only, the URI will only
- offer read-only access. If this node is read-write, the URI will
- offer read-write acess.
-
- If you have read-write access to a directory and wish to share merely
- read-only access with others, use get_immutable_uri().
-
+ """
The dirnode ('1') URI returned by this method can be used in
set_uri() on a different directory ('2') to 'mount' a reference to
this directory ('1') under the other ('2'). This URI is just a
protocol.
"""
- def get_immutable_uri():
- """Return the directory URI that can be used by others to get
- read-only access to this directory node. The result is a read-only
- URI, regardless of whether this dirnode is read-only or read-write.
-
- If you have merely read-only access to this dirnode,
- get_immutable_uri() will return the same thing as get_uri().
+ def get_readonly_uri():
"""
-
- def get_verifier():
- """Return an IVerifierURI instance that represents the
- 'verifiy/refresh capability' for this node. The holder of this
- capability will be able to renew the lease for this node, protecting
- it from garbage-collection. They will also be able to ask a server if
- it holds a share for the file or directory.
+ The dirnode ('1') URI returned by this method can be used in
+ set_uri() on a different directory ('2') to 'mount' a reference to
+ this directory ('1') under the other ('2'). This URI is just a
+ string, so it can be passed around through email or other out-of-band
+ protocol.
"""
- def check():
- """Perform a file check. See IChecker.check for details."""
-
def list():
"""I return a Deferred that fires with a dictionary mapping child
name to an IFileNode or IDirectoryNode."""
closed."""
class IUploader(Interface):
- def upload(uploadable):
+ def upload(uploadable, wait_for_numpeers=None):
"""Upload the file. 'uploadable' must impement IUploadable. This
returns a Deferred which fires with the URI of the file."""
"""
-class IVirtualDrive(Interface):
- """I am a service that may be available to a client.
-
- Within any client program, this service can be retrieved by using
- client.getService('vdrive').
- """
-
- def have_public_root():
- """Return a Boolean, True if get_public_root() will work."""
- def get_public_root():
- """Get the public read-write directory root.
-
- This returns a Deferred that fires with an IDirectoryNode instance
- corresponding to the global shared root directory."""
-
-
- def have_private_root():
- """Return a Boolean, True if get_public_root() will work."""
- def get_private_root():
- """Get the private directory root.
-
- This returns a Deferred that fires with an IDirectoryNode instance
- corresponding to this client's private root directory."""
-
- def get_node_at_path(path):
- """Transform a path into an IDirectoryNode or IFileNode.
-
- The path can either be a single string or a list of path-name
- elements. The former is generated from the latter by using
- .join('/'). If the first element of this list is '~', the rest will
- be interpreted relative to the local user's private root directory.
- Otherwse it will be interpreted relative to the global public root
- directory. As a result, the following three values of 'path' are
- equivalent::
-
- '/dirname/foo.txt'
- 'dirname/foo.txt'
- ['dirname', 'foo.txt']
-
- This method returns a Deferred that fires with the node in question,
- or errbacks with an IndexError if the target node could not be found.
- """
-
- def get_node(uri):
- """Transform a URI (or IURI) into an IDirectoryNode or IFileNode.
-
- This returns a Deferred that will fire with an instance that provides
- either IDirectoryNode or IFileNode, as appropriate."""
-
- def create_directory():
- """Return a new IDirectoryNode that is empty and not linked by
- anything."""
-
-
class NotCapableError(Exception):
"""You have tried to write to a read-only node."""
class BadWriteEnablerError(Exception):
pass
-class NotMutableError(Exception):
- pass
-
-
class RIControlClient(RemoteInterface):
def wait_for_client_connections(num_clients=int):
-from base64 import b32encode, b32decode
-
import re
+from base64 import b32encode, b32decode
from zope.interface import implements
from twisted.application import service
from twisted.python import log
from foolscap import Referenceable
+from allmydata import node
from allmydata.interfaces import RIIntroducer, RIIntroducerClient
from allmydata.util import observer
-class Introducer(service.MultiService, Referenceable):
+class IntroducerNode(node.Node):
+ PORTNUMFILE = "introducer.port"
+ NODETYPE = "introducer"
+ ENCODING_PARAMETERS_FILE = "encoding_parameters"
+ DEFAULT_K, DEFAULT_DESIRED, DEFAULT_N = 3, 7, 10
+
+ def tub_ready(self):
+ k, desired, n = self.DEFAULT_K, self.DEFAULT_DESIRED, self.DEFAULT_N
+ data = self.get_config("encoding_parameters")
+ if data is not None:
+ k,desired,n = data.split()
+ k = int(k); desired = int(desired); n = int(n)
+ introducerservice = IntroducerService(self.basedir, (k, desired, n))
+ self.add_service(introducerservice)
+ self.introducer_url = self.tub.registerReference(introducerservice, "introducer")
+ self.log(" introducer is at %s" % self.introducer_url)
+ self.write_config("introducer.furl", self.introducer_url + "\n")
+
+class IntroducerService(service.MultiService, Referenceable):
implements(RIIntroducer)
name = "introducer"
- def __init__(self):
+ def __init__(self, basedir=".", encoding_parameters=None):
service.MultiService.__init__(self)
+ self.introducer_url = None
self.nodes = set()
self.furls = set()
- self._encoding_parameters = None
-
- def set_encoding_parameters(self, parameters):
- self._encoding_parameters = parameters
+ self._encoding_parameters = encoding_parameters
def remote_hello(self, node, furl):
log.msg("introducer: new contact at %s, node is %s" % (furl, node))
othernode.callRemote("new_peers", set([furl]))
self.nodes.add(node)
-
class IntroducerClient(service.Service, Referenceable):
implements(RIIntroducerClient)
self.connection_observers = observer.ObserverList()
self.encoding_parameters = None
+ # The N'th element of _observers_of_enough_peers is None if nobody has
+ # asked to be informed when N peers become connected, it is a
+ # OneShotObserverList if someone has asked to be informed, and that
+ # list is fired when N peers next become connected (or immediately if
+ # N peers are already connected when someone asks), and the N'th
+ # element is replaced by None when the number of connected peers falls
+ # below N. _observers_of_enough_peers is always just long enough to
+ # hold the highest-numbered N that anyone is interested in (i.e.,
+ # there are never trailing Nones in _observers_of_enough_peers).
+ self._observers_of_enough_peers = []
+
def startService(self):
service.Service.startService(self)
self.introducer_reconnector = self.tub.connectTo(self.introducer_furl,
self.log("connected to %s" % b32encode(nodeid).lower()[:8])
self.connection_observers.notify(nodeid, rref)
self.connections[nodeid] = rref
+ if len(self._observers_of_enough_peers) > len(self.connections):
+ osol = self._observers_of_enough_peers[len(self.connections)]
+ if osol:
+ osol.fire(None)
def _lost():
# TODO: notifyOnDisconnect uses eventually(), but connects do
# not. Could this cause a problem?
del self.connections[nodeid]
+ if len(self._observers_of_enough_peers) > len(self.connections):
+ self._observers_of_enough_peers[len(self.connections)] = None
+ while self._observers_of_enough_peers and (not self._observers_of_enough_peers[-1]):
+ self._observers_of_enough_peers.pop()
+ for numpeers in self._observers_of_enough_peers:
+ if len(self.connections) == (numpeers-1):
+ # We know that this observer list must have been
+ # fired, since we had enough peers before this one was
+ # lost.
+ del self._observers_of_enough_peers[numpeers]
+
rref.notifyOnDisconnect(_lost)
self.log("connecting to %s" % b32encode(nodeid).lower()[:8])
self.reconnectors[furl] = self.tub.connectTo(furl, _got_peer)
def get_all_peers(self):
return self.connections.iteritems()
+
+ def when_enough_peers(self, numpeers):
+ """
+ I return a deferred that fires the next time that at least numpeers
+ are connected, or fires immediately if numpeers are currently
+ available.
+ """
+ self._observers_of_enough_peers.extend([None]*(numpeers+1-len(self._observers_of_enough_peers)))
+ if not self._observers_of_enough_peers[numpeers]:
+ self._observers_of_enough_peers[numpeers] = observer.OneShotObserverList()
+ if len(self.connections) >= numpeers:
+ self._observers_of_enough_peers[numpeers].fire(self)
+ return self._observers_of_enough_peers[numpeers].when_fired()
+++ /dev/null
-
-import os.path
-from allmydata import node
-from allmydata.dirnode import VirtualDriveServer
-from allmydata.introducer import Introducer
-
-
-class IntroducerAndVdrive(node.Node):
- PORTNUMFILE = "introducer.port"
- NODETYPE = "introducer"
- VDRIVEDIR = "vdrive"
- ENCODING_PARAMETERS_FILE = "encoding_parameters"
- DEFAULT_K, DEFAULT_DESIRED, DEFAULT_N = 3, 7, 10
-
- def __init__(self, basedir="."):
- node.Node.__init__(self, basedir)
- self.urls = {}
- self.read_encoding_parameters()
-
- def tub_ready(self):
- i = Introducer()
- r = self.add_service(i)
- self.urls["introducer"] = self.tub.registerReference(r, "introducer")
- self.log(" introducer is at %s" % self.urls["introducer"])
- self.write_config("introducer.furl", self.urls["introducer"] + "\n")
-
- vdrive_dir = os.path.join(self.basedir, self.VDRIVEDIR)
- vds = self.add_service(VirtualDriveServer(vdrive_dir))
- vds_furl = self.tub.registerReference(vds, "vdrive")
- vds.set_furl(vds_furl)
- self.urls["vdrive"] = vds_furl
- self.log(" vdrive is at %s" % self.urls["vdrive"])
- self.write_config("vdrive.furl", self.urls["vdrive"] + "\n")
-
- encoding_parameters = self.read_encoding_parameters()
- i.set_encoding_parameters(encoding_parameters)
-
- def read_encoding_parameters(self):
- k, desired, n = self.DEFAULT_K, self.DEFAULT_DESIRED, self.DEFAULT_N
- data = self.get_config("encoding_parameters")
- if data is not None:
- k,desired,n = data.split()
- k = int(k); desired = int(desired); n = int(n)
- return k, desired, n
-
from pycryptopp.publickey import rsa
+class NotMutableError(Exception):
+ pass
+
class NeedMoreDataError(Exception):
def __init__(self, needed_bytes, encprivkey_offset, encprivkey_length):
Exception.__init__(self)
def log_err(self, f):
log.err(f)
- def publish(self, newdata):
- """Publish the filenode's current contents. Returns a Deferred that
+ def publish(self, newdata, wait_for_numpeers=None):
+ """Publish the filenode's current contents. Returns a Deferred that
fires (with None) when the publish has done as much work as it's ever
going to do, or errbacks with ConsistencyError if it detects a
- simultaneous write."""
+ simultaneous write.
+
+ It will wait until at least wait_for_numpeers peers are connected
+ before it starts uploading
+
+ If wait_for_numpeers is None, then wait_for_numpeers is set to the
+ number of shares total (M).
+ """
+
+ self.log("starting publish")
+
+ if wait_for_numpeers is None:
+ # TODO: perhaps the default should be something like:
+ # wait_for_numpeers = self._node.get_total_shares()
+ wait_for_numpeers = 1
+ d = self._node._client.introducer_client.when_enough_peers(wait_for_numpeers)
+ d.addCallback(lambda dummy: self._after_enough_peers(newdata))
+ return d
+
+ def _after_enough_peers(self, newdata):
# 1: generate shares (SDMF: files are small, so we can do it in RAM)
# 2: perform peer selection, get candidate servers
# 2a: send queries to n+epsilon servers, to determine current shares
# 4a: may need to run recovery algorithm
# 5: when enough responses are back, we're done
- self.log("starting publish, data is %r" % (newdata,))
+ self.log("got enough peers")
self._storage_index = self._node.get_storage_index()
self._writekey = self._node.get_writekey()
def _got_query_results(self, datavs, peerid, permutedid,
reachable_peers, current_share_peers):
- self.log("_got_query_results")
-
assert isinstance(datavs, dict)
reachable_peers[peerid] = permutedid
for shnum, datav in datavs.items():
# one-per-peer in the normal permuted order.
while shares_needing_homes:
if not reachable_peers:
- raise NotEnoughPeersError("ran out of peers during upload")
+ prefix = idlib.b2a(self._node.get_storage_index())[:6]
+ raise NotEnoughPeersError("ran out of peers during upload of (%s); shares_needing_homes: %s, reachable_peers: %s" % (prefix, shares_needing_homes, reachable_peers,))
shnum = shares_needing_homes.pop(0)
possible_homes = reachable_peers.keys()
possible_homes.sort(lambda a,b:
key = hashutil.ssk_readkey_data_hash(IV, readkey)
enc = AES.new(key=key, mode=AES.MODE_CTR, counterstart="\x00"*16)
crypttext = enc.encrypt(newdata)
+ assert len(crypttext) == len(newdata)
# now apply FEC
self.MAX_SEGMENT_SIZE = 1024*1024
if surprised:
self._surprised = True
- def log_dispatch_map(self, dispatch_map):
+ def _log_dispatch_map(self, dispatch_map):
for shnum, places in dispatch_map.items():
sent_to = [(idlib.shortnodeid_b2a(peerid),
seqnum,
def _maybe_recover(self, (surprised, dispatch_map)):
self.log("_maybe_recover, surprised=%s, dispatch_map:" % surprised)
- self.log_dispatch_map(dispatch_map)
+ self._log_dispatch_map(dispatch_map)
if not surprised:
self.log(" no recovery needed")
return
self._current_roothash = None # ditto
self._current_seqnum = None # ditto
+ def __repr__(self):
+ return "<%s %x %s %s>" % (self.__class__.__name__, id(self), self.is_readonly() and 'RO' or 'RW', hasattr(self, '_uri') and self._uri.abbrev())
+
def init_from_uri(self, myuri):
# we have the URI, but we have not yet retrieved the public
# verification key, nor things like 'k' or 'N'. If and when someone
self._encprivkey = None
return self
- def create(self, initial_contents):
+ def create(self, initial_contents, wait_for_numpeers=None):
"""Call this when the filenode is first created. This will generate
- the keys, generate the initial shares, allocate shares, and upload
- the initial contents. Returns a Deferred that fires (with the
- MutableFileNode instance you should use) when it completes.
+ the keys, generate the initial shares, wait until at least numpeers
+ are connected, allocate shares, and upload the initial
+ contents. Returns a Deferred that fires (with the MutableFileNode
+ instance you should use) when it completes.
"""
self._required_shares = 3
self._total_shares = 10
# nobody knows about us yet"
self._current_seqnum = 0
self._current_roothash = "\x00"*32
- return self._publish(initial_contents)
+ return self._publish(initial_contents, wait_for_numpeers=wait_for_numpeers)
d.addCallback(_generated)
return d
verifier = signer.get_verifying_key()
return verifier, signer
- def _publish(self, initial_contents):
+ def _publish(self, initial_contents, wait_for_numpeers):
p = self.publish_class(self)
- d = p.publish(initial_contents)
+ d = p.publish(initial_contents, wait_for_numpeers=wait_for_numpeers)
d.addCallback(lambda res: self)
return d
ro.init_from_uri(self._uri.get_readonly())
return ro
+ def get_readonly_uri(self):
+ return self._uri.get_readonly().to_string()
+
def is_mutable(self):
return self._uri.is_mutable()
def is_readonly(self):
- # but maybe not you
return self._uri.is_readonly()
def __hash__(self):
r = Retrieve(self)
return r.retrieve()
- def replace(self, newdata):
+ def replace(self, newdata, wait_for_numpeers=None):
r = Retrieve(self)
d = r.retrieve()
- d.addCallback(lambda res: self._publish(newdata))
+ d.addCallback(lambda res: self._publish(newdata, wait_for_numpeers=wait_for_numpeers))
return d
from allmydata.logpublisher import LogPublisher
# Just to get their versions:
-import allmydata
-import zfec
-import foolscap
+import allmydata, foolscap, pycryptopp, zfec
# group 1 will be addr (dotted quad string), group 3 if any will be portnum (string)
ADDR_RE=re.compile("^([1-9][0-9]*\.[1-9][0-9]*\.[1-9][0-9]*\.[1-9][0-9]*)(:([1-9][0-9]*))?$")
return d.isoformat(" ") + ".000Z"
class Node(service.MultiService):
- # this implements common functionality of both Client nodes, Introducer
- # nodes, and Vdrive nodes
+ # this implements common functionality of both Client nodes and Introducer
+ # nodes.
NODETYPE = "unknown NODETYPE"
PORTNUMFILE = None
CERTFILE = "node.pem"
'foolscap': foolscap.__version__,
'twisted': twisted.__version__,
'zfec': zfec.__version__,
+ 'pycryptopp': pycryptopp.__version__,
}
def startService(self):
# Note: this class can be started and stopped at most once.
self.log("Node.startService")
+ # Delay until the reactor is running.
eventual.eventually(self._startService)
def _startService(self):
"Look here to find out which Tahoe node should be used for all "
"operations. The directory should either contain a full Tahoe node, "
"or a file named node.url which points to some other Tahoe node. "
- "It should also contain a file named my_vdrive.uri which contains "
- "the root dirnode URI that should be used, and a file named "
- "global_root.uri which contains the public global root dirnode URI."
+ "It should also contain a file named my_private_dir.uri which contains "
+ "the root dirnode URI that should be used."
],
["node-url", "u", None,
"URL of the tahoe node to use, a URL like \"http://127.0.0.1:8123\". "
"This overrides the URL found in the --node-directory ."],
["root-uri", "r", "private",
- "Which dirnode URI should be used as a root directory. The string "
- "'public' is special, and means we should use the public global root "
- "as found in the global_root.uri file in the --node-directory . The "
- "string 'private' is also special, and means we should use the "
- "private vdrive as found in the my_vdrive.uri file in the "
+ "Which dirnode URI should be used as a root directory. The "
+ "string 'private' is also, and means we should use the "
+ "private vdrive as found in the my_private_dir.uri file in the "
"--node-directory ."],
]
# also compute self['root-uri']
if self['root-uri'] == "private":
- uri_file = os.path.join(self['node-directory'], "my_vdrive.uri")
- self['root-uri'] = open(uri_file, "r").read().strip()
- elif self['root-uri'] == "public":
- uri_file = os.path.join(self['node-directory'], "global_root.uri")
+ uri_file = os.path.join(self['node-directory'], "my_private_dir.uri")
self['root-uri'] = open(uri_file, "r").read().strip()
else:
from allmydata import uri
introducer_tac = """
# -*- python -*-
-from allmydata import introducer_and_vdrive
+from allmydata import introducer
from twisted.application import service
-c = introducer_and_vdrive.IntroducerAndVdrive()
+c = introducer.IntroducerNode()
application = service.Application("allmydata_introducer")
c.setServiceParent(application)
f = open(os.path.join(basedir, "webport"), "w")
f.write(config['webport'] + "\n")
f.close()
+ # Create an empty my_private_dir.uri file, indicating that the node
+ # should fill it with the URI after creating the directory.
+ open(os.path.join(basedir, "my_private_dir.uri"), "w")
print >>out, "client created in %s" % basedir
- print >>out, " please copy introducer.furl and vdrive.furl into the directory"
+ print >>out, " please copy introducer.furl into the directory"
def create_introducer(basedir, config, out=sys.stdout, err=sys.stderr):
if os.path.exists(basedir):
# do not import any allmydata modules at this level. Do that from inside
# individual functions instead.
-import os, sys, struct, time
+import sys, struct, time
from twisted.python import usage
-from allmydata.scripts.common import BasedirMixin
class DumpOptions(usage.Options):
optParameters = [
if not self['filename']:
raise usage.UsageError("<filename> parameter is required")
-class DumpRootDirnodeOptions(BasedirMixin, usage.Options):
- optParameters = [
- ["basedir", "C", None, "the vdrive-server's base directory"],
- ]
-
-class DumpDirnodeOptions(BasedirMixin, usage.Options):
- optParameters = [
- ["uri", "u", None, "the URI of the dirnode to dump."],
- ["basedir", "C", None, "which directory to create the introducer in"],
- ]
- optFlags = [
- ["verbose", "v", "be extra noisy (show encrypted data)"],
- ]
- def parseArgs(self, *args):
- if len(args) == 1:
- self['uri'] = args[-1]
- args = args[:-1]
- BasedirMixin.parseArgs(self, *args)
-
- def postOptions(self):
- BasedirMixin.postOptions(self)
- if not self['uri']:
- raise usage.UsageError("<uri> parameter is required")
-
def dump_share(config, out=sys.stdout, err=sys.stderr):
from allmydata import uri, storage
print >>out
-def dump_root_dirnode(config, out=sys.stdout, err=sys.stderr):
- from allmydata import uri
-
- basedir = config['basedirs'][0]
- root_dirnode_file = os.path.join(basedir, "vdrive", "root")
- try:
- f = open(root_dirnode_file, "rb")
- key = f.read()
- rooturi = uri.DirnodeURI("fakeFURL", key)
- print >>out, rooturi.to_string()
- return 0
- except EnvironmentError:
- print >>out, "unable to read root dirnode file from %s" % \
- root_dirnode_file
- return 1
-
-def dump_directory_node(config, out=sys.stdout, err=sys.stderr):
- from allmydata import dirnode
- from allmydata.util import hashutil, idlib
- from allmydata.interfaces import IDirnodeURI
- basedir = config['basedirs'][0]
- dir_uri = IDirnodeURI(config['uri'])
- verbose = config['verbose']
-
- if dir_uri.is_readonly():
- wk, we, rk, index = \
- hashutil.generate_dirnode_keys_from_readkey(dir_uri.readkey)
- else:
- wk, we, rk, index = \
- hashutil.generate_dirnode_keys_from_writekey(dir_uri.writekey)
-
- filename = os.path.join(basedir, "vdrive", idlib.b2a(index))
-
- print >>out
- print >>out, "dirnode uri: %s" % dir_uri.to_string()
- print >>out, "filename : %s" % filename
- print >>out, "index : %s" % idlib.b2a(index)
- if wk:
- print >>out, "writekey : %s" % idlib.b2a(wk)
- print >>out, "write_enabler: %s" % idlib.b2a(we)
- else:
- print >>out, "writekey : None"
- print >>out, "write_enabler: None"
- print >>out, "readkey : %s" % idlib.b2a(rk)
-
- print >>out
-
- vds = dirnode.VirtualDriveServer(os.path.join(basedir, "vdrive"), False)
- data = vds._read_from_file(index)
- if we:
- if we != data[0]:
- print >>out, "ERROR: write_enabler does not match"
-
- for (H_key, E_key, E_write, E_read) in data[1]:
- if verbose:
- print >>out, " H_key %s" % idlib.b2a(H_key)
- print >>out, " E_key %s" % idlib.b2a(E_key)
- print >>out, " E_write %s" % idlib.b2a(E_write)
- print >>out, " E_read %s" % idlib.b2a(E_read)
- key = dirnode.decrypt(rk, E_key)
- print >>out, " key %s" % key
- if hashutil.dir_name_hash(rk, key) != H_key:
- print >>out, " ERROR: H_key does not match"
- if wk and E_write:
- if len(E_write) < 14:
- print >>out, " ERROR: write data is short:", idlib.b2a(E_write)
- write = dirnode.decrypt(wk, E_write)
- print >>out, " write: %s" % write
- read = dirnode.decrypt(rk, E_read)
- print >>out, " read: %s" % read
- print >>out
-
- return 0
-
-
subCommands = [
["dump-share", None, DumpOptions,
"Unpack and display the contents of a share (uri_extension and leases)."],
- ["dump-root-dirnode", None, DumpRootDirnodeOptions,
- "Compute most of the URI for the vdrive server's root dirnode."],
- ["dump-dirnode", None, DumpDirnodeOptions,
- "Unpack and display the contents of a vdrive DirectoryNode."],
]
dispatch = {
"dump-share": dump_share,
- "dump-root-dirnode": dump_root_dirnode,
- "dump-dirnode": dump_directory_node,
}
from foolscap import Referenceable
from twisted.application import service
from twisted.internet import defer
+from twisted.python import log
from zope.interface import implements
from allmydata.interfaces import RIStorageServer, RIBucketWriter, \
from twisted.internet import defer, reactor, protocol, error
from twisted.application import service, internet
from twisted.web import client as tw_client
-from allmydata import client, introducer_and_vdrive
+from allmydata import client, introducer
from allmydata.scripts import create_node
from allmydata.util import testutil, fileutil
import foolscap
#print "STARTING"
self.stats = {}
self.statsfile = open(os.path.join(self.basedir, "stats.out"), "a")
- d = self.make_introducer_and_vdrive()
+ d = self.make_introducer()
def _more(res):
return self.start_client()
d.addCallback(_more)
s.setServiceParent(self.sparent)
return s
- def make_introducer_and_vdrive(self):
- iv_basedir = os.path.join(self.testdir, "introducer_and_vdrive")
+ def make_introducer(self):
+ iv_basedir = os.path.join(self.testdir, "introducer")
os.mkdir(iv_basedir)
- iv = introducer_and_vdrive.IntroducerAndVdrive(basedir=iv_basedir)
- self.introducer_and_vdrive = self.add_service(iv)
- d = self.introducer_and_vdrive.when_tub_ready()
+ iv = introducer.IntroducerNode(basedir=iv_basedir)
+ self.introducer = self.add_service(iv)
+ d = self.introducer.when_tub_ready()
def _introducer_ready(res):
- q = self.introducer_and_vdrive
- self.introducer_furl = q.urls["introducer"]
- self.vdrive_furl = q.urls["vdrive"]
+ q = self.introducer
+ self.introducer_furl = q.introducer_url
d.addCallback(_introducer_ready)
return d
f = open(os.path.join(nodedir, "introducer.furl"), "w")
f.write(self.introducer_furl)
f.close()
- f = open(os.path.join(nodedir, "vdrive.furl"), "w")
- f.write(self.vdrive_furl)
- f.close()
# the only tests for which we want the internal nodes to actually
# retain shares are the ones where somebody's going to download
# them.
c = self.add_service(client.Client(basedir=nodedir))
self.nodes.append(c)
# the peers will start running, eventually they will connect to each
- # other and the introducer_and_vdrive
+ # other and the introducer
def touch_keepalive(self):
if os.path.exists(self.keepalive_file):
f = open(os.path.join(clientdir, "introducer.furl"), "w")
f.write(self.introducer_furl + "\n")
f.close()
- f = open(os.path.join(clientdir, "vdrive.furl"), "w")
- f.write(self.vdrive_furl + "\n")
- f.close()
f = open(os.path.join(clientdir, "webport"), "w")
# TODO: ideally we would set webport=0 and then ask the node what
# port it picked. But at the moment it is not convenient to do this,
d.addCallback(_done)
elif self.mode == "upload-POST":
data = "a" * size
- url = "/vdrive/global"
+ url = "/vdrive/private"
d = self.POST(url, t="upload", file=("%d.data" % size, data))
elif self.mode in ("receive",
"download", "download-GET", "download-GET-slow"):
fileutil.make_dirs("cli/test_options")
open("cli/test_options/node.url","w").write("http://localhost:8080/\n")
private_uri = uri.DirnodeURI("furl", "key").to_string()
- public_uri = uri.DirnodeURI("furl", "publickey").to_string()
- open("cli/test_options/my_vdrive.uri", "w").write(private_uri + "\n")
- open("cli/test_options/global_root.uri", "w").write(public_uri + "\n")
+ open("cli/test_options/my_private_dir.uri", "w").write(private_uri + "\n")
o = cli.ListOptions()
o.parseOptions(["--node-directory", "cli/test_options"])
self.failUnlessEqual(o['node-url'], "http://localhost:8080/")
self.failUnlessEqual(o['vdrive_pathname'], "")
o = cli.ListOptions()
- o.parseOptions(["--node-directory", "cli/test_options",
- "--root-uri", "public"])
+ o.parseOptions(["--node-directory", "cli/test_options"])
self.failUnlessEqual(o['node-url'], "http://localhost:8080/")
- self.failUnlessEqual(o['root-uri'], public_uri)
self.failUnlessEqual(o['vdrive_pathname'], "")
o = cli.ListOptions()
+++ /dev/null
-
-from twisted.trial import unittest
-from cStringIO import StringIO
-from foolscap import eventual
-from twisted.internet import defer
-from twisted.python import failure
-from allmydata import uri, dirnode
-from allmydata.util import hashutil
-from allmydata.interfaces import IDirectoryNode, IDirnodeURI, IURI, IFileURI
-from allmydata.scripts import runner
-from allmydata.dirnode import VirtualDriveServer, \
- BadWriteEnablerError, NoPublicRootError
-
-# test the host-side code
-
-class DirectoryNode(unittest.TestCase):
- def test_vdrive_server(self):
- basedir = "dirnode_host/DirectoryNode/test_vdrive_server"
- vds = VirtualDriveServer(basedir)
- vds.set_furl("myFURL")
-
- root_uri = vds.get_public_root_uri()
- u = IDirnodeURI(root_uri)
- self.failIf(u.is_readonly())
- self.failUnlessEqual(u.furl, "myFURL")
- self.failUnlessEqual(len(u.writekey), hashutil.KEYLEN)
-
- wk, we, rk, index = \
- hashutil.generate_dirnode_keys_from_writekey(u.writekey)
- empty_list = vds.list(index)
- self.failUnlessEqual(empty_list, [])
-
- vds.set(index, we, "key1", "name1", "write1", "read1")
- vds.set(index, we, "key2", "name2", "write2", "read2")
- # we should be able to replace entries without complaint
- vds.set(index, we, "key2", "name2", "", "read2")
-
- self.failUnlessRaises(BadWriteEnablerError,
- vds.set,
- index, "not the write enabler",
- "key2", "name2", "write2", "read2")
-
- self.failUnlessEqual(vds.get(index, "key1"),
- ("write1", "read1"))
- self.failUnlessEqual(vds.get(index, "key2"),
- ("", "read2"))
- self.failUnlessRaises(KeyError,
- vds.get, index, "key3")
-
- self.failUnlessEqual(sorted(vds.list(index)),
- [ ("name1", "write1", "read1"),
- ("name2", "", "read2"),
- ])
-
- self.failUnlessRaises(BadWriteEnablerError,
- vds.delete,
- index, "not the write enabler", "name1")
- self.failUnlessEqual(sorted(vds.list(index)),
- [ ("name1", "write1", "read1"),
- ("name2", "", "read2"),
- ])
- self.failUnlessRaises(KeyError,
- vds.delete,
- index, we, "key3")
-
- vds.delete(index, we, "key1")
- self.failUnlessEqual(sorted(vds.list(index)),
- [ ("name2", "", "read2"),
- ])
- self.failUnlessRaises(KeyError,
- vds.get, index, "key1")
- self.failUnlessEqual(vds.get(index, "key2"),
- ("", "read2"))
-
-
- vds2 = VirtualDriveServer(basedir)
- vds2.set_furl("myFURL")
- root_uri2 = vds.get_public_root_uri()
- u2 = IDirnodeURI(root_uri2)
- self.failIf(u2.is_readonly())
- (wk2, we2, rk2, index2) = \
- hashutil.generate_dirnode_keys_from_writekey(u2.writekey)
- self.failUnlessEqual(sorted(vds2.list(index2)),
- [ ("name2", "", "read2"),
- ])
-
- def test_no_root(self):
- basedir = "dirnode_host/DirectoryNode/test_no_root"
- vds = VirtualDriveServer(basedir, offer_public_root=False)
- vds.set_furl("myFURL")
-
- self.failUnlessRaises(NoPublicRootError,
- vds.get_public_root_uri)
-
-
-# and the client-side too
-
-class LocalReference:
- def __init__(self, target):
- self.target = target
- def callRemote(self, methname, *args, **kwargs):
- def _call(ignored):
- meth = getattr(self.target, methname)
- return meth(*args, **kwargs)
- d = eventual.fireEventually(None)
- d.addCallback(_call)
- return d
-
-class MyTub:
- def __init__(self, vds, myfurl):
- self.vds = vds
- self.myfurl = myfurl
- def getReference(self, furl):
- assert furl == self.myfurl
- return eventual.fireEventually(LocalReference(self.vds))
-
-class MyClient:
- def __init__(self, vds, myfurl):
- self.tub = MyTub(vds, myfurl)
-
- def create_node_from_uri(self, u):
- u = IURI(u)
- assert IFileURI.providedBy(u)
- return dirnode.FileNode(u, self)
-
-class Test(unittest.TestCase):
- def test_create_directory(self):
- basedir = "vdrive/test_create_directory/vdrive"
- vds = dirnode.VirtualDriveServer(basedir)
- vds.set_furl("myFURL")
- self.client = client = MyClient(vds, "myFURL")
- d = dirnode.create_directory(client, "myFURL")
- def _created(node):
- self.failUnless(IDirectoryNode.providedBy(node))
- self.failUnless(node.is_mutable())
- d.addCallback(_created)
- return d
-
- def test_one(self):
- self.basedir = basedir = "vdrive/test_one/vdrive"
- vds = dirnode.VirtualDriveServer(basedir)
- vds.set_furl("myFURL")
- root_uri = vds.get_public_root_uri()
-
- self.client = client = MyClient(vds, "myFURL")
- d1 = dirnode.create_directory_node(client, root_uri)
- d2 = dirnode.create_directory_node(client, root_uri)
- d = defer.gatherResults( [d1,d2] )
- d.addCallback(self._test_one_1)
- return d
-
- def _test_one_1(self, (rootnode1, rootnode2) ):
- self.failUnlessEqual(rootnode1, rootnode2)
- self.failIfEqual(rootnode1, "not")
-
- self.rootnode = rootnode = rootnode1
- self.failUnless(rootnode.is_mutable())
- self.readonly_uri = rootnode.get_immutable_uri()
- d = dirnode.create_directory_node(self.client, self.readonly_uri)
- d.addCallback(self._test_one_2)
- return d
-
- def _test_one_2(self, ro_rootnode):
- self.ro_rootnode = ro_rootnode
- self.failIf(ro_rootnode.is_mutable())
- self.failUnlessEqual(ro_rootnode.get_immutable_uri(),
- self.readonly_uri)
-
- rootnode = self.rootnode
-
- ignored = rootnode.dump()
-
- # root/
- d = rootnode.list()
- def _listed(res):
- self.failUnlessEqual(res, {})
- d.addCallback(_listed)
-
- file1 = uri.CHKFileURI(key="k"*15+"1",
- uri_extension_hash="e"*32,
- needed_shares=25,
- total_shares=100,
- size=12345).to_string()
- file2 = uri.CHKFileURI(key="k"*15+"2",
- uri_extension_hash="e"*32,
- needed_shares=25,
- total_shares=100,
- size=12345).to_string()
- file2_node = dirnode.FileNode(file2, None)
- d.addCallback(lambda res: rootnode.set_uri("foo", file1))
- # root/
- # root/foo =file1
-
- d.addCallback(lambda res: rootnode.list())
- def _listed2(res):
- self.failUnlessEqual(res.keys(), ["foo"])
- file1_node = res["foo"]
- self.file1_node = file1_node
- self.failUnless(isinstance(file1_node, dirnode.FileNode))
- self.failUnlessEqual(file1_node.uri, file1)
- d.addCallback(_listed2)
-
- d.addCallback(lambda res: rootnode.get("foo"))
- def _got_foo(res):
- self.failUnless(isinstance(res, dirnode.FileNode))
- self.failUnlessEqual(res.uri, file1)
- d.addCallback(_got_foo)
-
- d.addCallback(lambda res: rootnode.get("missing"))
- # this should raise an exception
- d.addBoth(self.shouldFail, KeyError, "get('missing')",
- "unable to find child named 'missing'")
-
- d.addCallback(lambda res: rootnode.create_empty_directory("bar"))
- # root/
- # root/foo =file1
- # root/bar/
-
- d.addCallback(lambda res: rootnode.list())
- d.addCallback(self.failUnlessKeysMatch, ["foo", "bar"])
- def _listed3(res):
- self.failIfEqual(res["foo"], res["bar"])
- self.failIfEqual(res["bar"], res["foo"])
- self.failIfEqual(res["foo"], "not")
- self.failIfEqual(res["bar"], self.rootnode)
- self.failUnlessEqual(res["foo"], res["foo"])
- # make sure the objects can be used as dict keys
- testdict = {res["foo"]: 1, res["bar"]: 2}
- bar_node = res["bar"]
- self.failUnless(isinstance(bar_node, dirnode.MutableDirectoryNode))
- self.bar_node = bar_node
- bar_ro_uri = bar_node.get_immutable_uri()
- return rootnode.set_uri("bar-ro", bar_ro_uri)
- d.addCallback(_listed3)
- # root/
- # root/foo =file1
- # root/bar/
- # root/bar-ro/ (read-only)
-
- d.addCallback(lambda res: rootnode.list())
- d.addCallback(self.failUnlessKeysMatch, ["foo", "bar", "bar-ro"])
- def _listed4(res):
- self.failIf(res["bar-ro"].is_mutable())
- self.bar_node_readonly = res["bar-ro"]
-
- # add another file to bar/
- bar = res["bar"]
- return bar.set_node("file2", file2_node)
- d.addCallback(_listed4)
- d.addCallback(self.failUnlessIdentical, file2_node)
- # and a directory
- d.addCallback(lambda res: self.bar_node.create_empty_directory("baz"))
- def _added_baz(baz_node):
- self.failUnless(IDirectoryNode.providedBy(baz_node))
- self.baz_node = baz_node
- d.addCallback(_added_baz)
- # root/
- # root/foo =file1
- # root/bar/
- # root/bar/file2 =file2
- # root/bar/baz/
- # root/bar-ro/ (read-only)
- # root/bar-ro/file2 =file2
- # root/bar-ro/baz/
-
- d.addCallback(lambda res: self.bar_node.list())
- d.addCallback(self.failUnlessKeysMatch, ["file2", "baz"])
- d.addCallback(lambda res:
- self.failUnless(res["baz"].is_mutable()))
-
- d.addCallback(lambda res: self.bar_node_readonly.list())
- d.addCallback(self.failUnlessKeysMatch, ["file2", "baz"])
- d.addCallback(lambda res:
- self.failIf(res["baz"].is_mutable()))
-
- d.addCallback(lambda res: rootnode.get_child_at_path("bar/file2"))
- def _got_file2(res):
- self.failUnless(isinstance(res, dirnode.FileNode))
- self.failUnlessEqual(res.uri, file2)
- d.addCallback(_got_file2)
-
- d.addCallback(lambda res: rootnode.get_child_at_path(["bar", "file2"]))
- d.addCallback(_got_file2)
-
- d.addCallback(lambda res: self.bar_node.get_child_at_path(["file2"]))
- d.addCallback(_got_file2)
-
- d.addCallback(lambda res: self.bar_node.get_child_at_path([]))
- d.addCallback(lambda res: self.failUnlessIdentical(res, self.bar_node))
-
- # test the manifest
- d.addCallback(lambda res: self.rootnode.build_manifest())
- def _check_manifest(manifest):
- manifest = sorted(list(manifest))
- self.failUnlessEqual(len(manifest), 5)
- expected = [self.rootnode.get_verifier().to_string(),
- self.bar_node.get_verifier().to_string(),
- self.file1_node.get_verifier().to_string(),
- file2_node.get_verifier().to_string(),
- self.baz_node.get_verifier().to_string(),
- ]
- expected.sort()
- self.failUnlessEqual(manifest, expected)
- d.addCallback(_check_manifest)
-
- # try to add a file to bar-ro, should get exception
- d.addCallback(lambda res:
- self.bar_node_readonly.set_uri("file3", file2))
- d.addBoth(self.shouldFail, dirnode.NotMutableError,
- "bar-ro.set('file3')")
-
- # try to delete a file from bar-ro, should get exception
- d.addCallback(lambda res: self.bar_node_readonly.delete("file2"))
- d.addBoth(self.shouldFail, dirnode.NotMutableError,
- "bar-ro.delete('file2')")
-
- # try to mkdir in bar-ro, should get exception
- d.addCallback(lambda res:
- self.bar_node_readonly.create_empty_directory("boffo"))
- d.addBoth(self.shouldFail, dirnode.NotMutableError,
- "bar-ro.mkdir('boffo')")
-
- d.addCallback(lambda res: rootnode.delete("foo"))
- # root/
- # root/bar/
- # root/bar/file2 =file2
- # root/bar/baz/
- # root/bar-ro/ (read-only)
- # root/bar-ro/file2 =file2
- # root/bar-ro/baz/
-
- d.addCallback(lambda res: rootnode.list())
- d.addCallback(self.failUnlessKeysMatch, ["bar", "bar-ro"])
-
- d.addCallback(lambda res:
- self.bar_node.move_child_to("file2",
- self.rootnode, "file4"))
- # root/
- # root/file4 = file2
- # root/bar/
- # root/bar/baz/
- # root/bar-ro/ (read-only)
- # root/bar-ro/baz/
-
- d.addCallback(lambda res: rootnode.list())
- d.addCallback(self.failUnlessKeysMatch, ["bar", "bar-ro", "file4"])
- d.addCallback(lambda res:self.bar_node.list())
- d.addCallback(self.failUnlessKeysMatch, ["baz"])
- d.addCallback(lambda res:self.bar_node_readonly.list())
- d.addCallback(self.failUnlessKeysMatch, ["baz"])
-
-
- d.addCallback(lambda res:
- rootnode.move_child_to("file4",
- self.bar_node_readonly, "boffo"))
- d.addBoth(self.shouldFail, dirnode.NotMutableError,
- "mv root/file4 root/bar-ro/boffo")
-
- d.addCallback(lambda res: rootnode.list())
- d.addCallback(self.failUnlessKeysMatch, ["bar", "bar-ro", "file4"])
- d.addCallback(lambda res:self.bar_node.list())
- d.addCallback(self.failUnlessKeysMatch, ["baz"])
- d.addCallback(lambda res:self.bar_node_readonly.list())
- d.addCallback(self.failUnlessKeysMatch, ["baz"])
-
-
- d.addCallback(lambda res:
- rootnode.move_child_to("file4", self.bar_node))
-
- d.addCallback(lambda res: rootnode.list())
- d.addCallback(self.failUnlessKeysMatch, ["bar", "bar-ro"])
- d.addCallback(lambda res:self.bar_node.list())
- d.addCallback(self.failUnlessKeysMatch, ["baz", "file4"])
- d.addCallback(lambda res:self.bar_node_readonly.list())
- d.addCallback(self.failUnlessKeysMatch, ["baz", "file4"])
- # root/
- # root/bar/
- # root/bar/file4 = file2
- # root/bar/baz/
- # root/bar-ro/ (read-only)
- # root/bar-ro/file4 = file2
- # root/bar-ro/baz/
-
- # test has_child
- d.addCallback(lambda res: rootnode.has_child("bar"))
- d.addCallback(self.failUnlessEqual, True)
- d.addCallback(lambda res: rootnode.has_child("missing"))
- d.addCallback(self.failUnlessEqual, False)
-
- # test the manifest
- d.addCallback(lambda res: self.rootnode.build_manifest())
- def _check_manifest2(manifest):
- manifest = sorted(list(manifest))
- self.failUnlessEqual(len(manifest), 4)
- expected = [self.rootnode.get_verifier().to_string(),
- self.bar_node.get_verifier().to_string(),
- file2_node.get_verifier().to_string(),
- self.baz_node.get_verifier().to_string(),
- ]
- expected.sort()
- self.failUnlessEqual(manifest, expected)
- d.addCallback(_check_manifest2)
-
- d.addCallback(self._test_one_3)
- return d
-
- def _test_one_3(self, res):
- # now test some of the diag tools with the data we've created
- out,err = StringIO(), StringIO()
- rc = runner.runner(["dump-root-dirnode", "vdrive/test_one"],
- stdout=out, stderr=err)
- output = out.getvalue()
- self.failUnless(output.startswith("URI:DIR:fakeFURL:"))
- self.failUnlessEqual(rc, 0)
-
- out,err = StringIO(), StringIO()
- rc = runner.runner(["dump-dirnode",
- "--basedir", "vdrive/test_one",
- "--verbose",
- self.bar_node.get_uri()],
- stdout=out, stderr=err)
- output = out.getvalue()
- #print output
- self.failUnlessEqual(rc, 0)
- self.failUnless("dirnode uri: URI:DIR:myFURL" in output)
- self.failUnless("write_enabler" in output)
- self.failIf("write_enabler: None" in output)
- self.failUnless("key baz\n" in output)
- self.failUnless(" write: URI:DIR:myFURL:" in output)
- self.failUnless(" read: URI:DIR-RO:myFURL:" in output)
- self.failUnless("key file4\n" in output)
- self.failUnless("H_key " in output)
-
- out,err = StringIO(), StringIO()
- rc = runner.runner(["dump-dirnode",
- "--basedir", "vdrive/test_one",
- # non-verbose
- "--uri", self.bar_node.get_uri()],
- stdout=out, stderr=err)
- output = out.getvalue()
- #print output
- self.failUnlessEqual(rc, 0)
- self.failUnless("dirnode uri: URI:DIR:myFURL" in output)
- self.failUnless("write_enabler" in output)
- self.failIf("write_enabler: None" in output)
- self.failUnless("key baz\n" in output)
- self.failUnless(" write: URI:DIR:myFURL:" in output)
- self.failUnless(" read: URI:DIR-RO:myFURL:" in output)
- self.failUnless("key file4\n" in output)
- self.failIf("H_key " in output)
-
- out,err = StringIO(), StringIO()
- rc = runner.runner(["dump-dirnode",
- "--basedir", "vdrive/test_one",
- "--verbose",
- self.bar_node_readonly.get_uri()],
- stdout=out, stderr=err)
- output = out.getvalue()
- #print output
- self.failUnlessEqual(rc, 0)
- self.failUnless("dirnode uri: URI:DIR-RO:myFURL" in output)
- self.failUnless("write_enabler: None" in output)
- self.failUnless("key baz\n" in output)
- self.failIf(" write: URI:DIR:myFURL:" in output)
- self.failUnless(" read: URI:DIR-RO:myFURL:" in output)
- self.failUnless("key file4\n" in output)
-
- def shouldFail(self, res, expected_failure, which, substring=None):
- if isinstance(res, failure.Failure):
- res.trap(expected_failure)
- if substring:
- self.failUnless(substring in str(res),
- "substring '%s' not in '%s'"
- % (substring, str(res)))
- else:
- self.fail("%s was supposed to raise %s, not get '%s'" %
- (which, expected_failure, res))
-
- def failUnlessKeysMatch(self, res, expected_keys):
- self.failUnlessEqual(sorted(res.keys()),
- sorted(expected_keys))
- return res
-
-def flip_bit(data, offset):
- if offset < 0:
- offset = len(data) + offset
- return data[:offset] + chr(ord(data[offset]) ^ 0x01) + data[offset+1:]
-
-class Encryption(unittest.TestCase):
- def test_loopback(self):
- key = "k" * 16
- data = "This is some plaintext data."
- crypttext = dirnode.encrypt(key, data)
- plaintext = dirnode.decrypt(key, crypttext)
- self.failUnlessEqual(data, plaintext)
-
- def test_hmac(self):
- key = "j" * 16
- data = "This is some more plaintext data."
- crypttext = dirnode.encrypt(key, data)
- # flip a bit in the IV
- self.failUnlessRaises(dirnode.IntegrityCheckError,
- dirnode.decrypt,
- key, flip_bit(crypttext, 0))
- # flip a bit in the crypttext
- self.failUnlessRaises(dirnode.IntegrityCheckError,
- dirnode.decrypt,
- key, flip_bit(crypttext, 16))
- # flip a bit in the HMAC
- self.failUnlessRaises(dirnode.IntegrityCheckError,
- dirnode.decrypt,
- key, flip_bit(crypttext, -1))
- plaintext = dirnode.decrypt(key, crypttext)
- self.failUnlessEqual(data, plaintext)
-
from base64 import b32encode
+import os
+
from twisted.trial import unittest
from twisted.internet import defer, reactor
from twisted.python import log
from foolscap import Tub, Referenceable
-from foolscap.eventual import flushEventualQueue
+from foolscap.eventual import fireEventually, flushEventualQueue
from twisted.application import service
-from allmydata.introducer import IntroducerClient, Introducer
+from allmydata.introducer import IntroducerClient, IntroducerService, IntroducerNode
from allmydata.util import testutil
class MyNode(Referenceable):
def log(self, msg):
log.msg(msg)
+class TestIntroducerNode(testutil.SignalMixin, unittest.TestCase):
+ def test_loadable(self):
+ basedir = "introducer.IntroducerNode.test_loadable"
+ os.mkdir(basedir)
+ q = IntroducerNode(basedir)
+ d = fireEventually(None)
+ d.addCallback(lambda res: q.startService())
+ d.addCallback(lambda res: q.when_tub_ready())
+ def _check_parameters(res):
+ i = q.getServiceNamed("introducer")
+ self.failUnlessEqual(i._encoding_parameters, (3, 7, 10))
+ d.addCallback(_check_parameters)
+ d.addCallback(lambda res: q.stopService())
+ d.addCallback(flushEventualQueue)
+ return d
+
+ def test_set_parameters(self):
+ basedir = "introducer.IntroducerNode.test_set_parameters"
+ os.mkdir(basedir)
+ f = open(os.path.join(basedir, "encoding_parameters"), "w")
+ f.write("25 75 100")
+ f.close()
+ q = IntroducerNode(basedir)
+ d = fireEventually(None)
+ d.addCallback(lambda res: q.startService())
+ d.addCallback(lambda res: q.when_tub_ready())
+ def _check_parameters(res):
+ i = q.getServiceNamed("introducer")
+ self.failUnlessEqual(i._encoding_parameters, (25, 75, 100))
+ d.addCallback(_check_parameters)
+ d.addCallback(lambda res: q.stopService())
+ d.addCallback(flushEventualQueue)
+ return d
+
class TestIntroducer(unittest.TestCase, testutil.PollMixin):
def setUp(self):
self.parent = LoggingMultiService()
ic.notify_on_new_connection(_ignore)
def test_listen(self):
- i = Introducer()
+ i = IntroducerService()
i.setServiceParent(self.parent)
def test_system(self):
portnum = l.getPortnum()
tub.setLocation("localhost:%d" % portnum)
- i = Introducer()
+ i = IntroducerService()
i.setServiceParent(self.parent)
iurl = tub.registerReference(i)
NUMCLIENTS = 5
portnum = l.getPortnum()
tub.setLocation("localhost:%d" % portnum)
- i = Introducer()
+ i = IntroducerService()
i.setServiceParent(self.parent)
iurl = tub.registerReference(i)
portnum = l.getPortnum()
tub.setLocation("localhost:%d" % portnum)
- i = Introducer()
+ i = IntroducerService()
i.setServiceParent(self.parent)
iurl = tub.registerReference(i)
+++ /dev/null
-
-import os
-from twisted.trial import unittest
-from foolscap.eventual import fireEventually, flushEventualQueue
-
-from allmydata import introducer_and_vdrive
-from allmydata.util import testutil
-
-class Basic(testutil.SignalMixin, unittest.TestCase):
- def test_loadable(self):
- basedir = "introducer_and_vdrive.Basic.test_loadable"
- os.mkdir(basedir)
- q = introducer_and_vdrive.IntroducerAndVdrive(basedir)
- d = fireEventually(None)
- d.addCallback(lambda res: q.startService())
- d.addCallback(lambda res: q.when_tub_ready())
- def _check_parameters(res):
- i = q.getServiceNamed("introducer")
- self.failUnlessEqual(i._encoding_parameters, (3, 7, 10))
- d.addCallback(_check_parameters)
- d.addCallback(lambda res: q.stopService())
- d.addCallback(flushEventualQueue)
- return d
-
- def test_set_parameters(self):
- basedir = "introducer_and_vdrive.Basic.test_set_parameters"
- os.mkdir(basedir)
- f = open(os.path.join(basedir, "encoding_parameters"), "w")
- f.write("25 75 100")
- f.close()
- q = introducer_and_vdrive.IntroducerAndVdrive(basedir)
- d = fireEventually(None)
- d.addCallback(lambda res: q.startService())
- d.addCallback(lambda res: q.when_tub_ready())
- def _check_parameters(res):
- i = q.getServiceNamed("introducer")
- self.failUnlessEqual(i._encoding_parameters, (25, 75, 100))
- d.addCallback(_check_parameters)
- d.addCallback(lambda res: q.stopService())
- d.addCallback(flushEventualQueue)
- return d
-
class FakeFilenode(mutable.MutableFileNode):
counter = itertools.count(1)
all_contents = {}
-
+ all_rw_friends = {}
+
+ def create(self, initial_contents, wait_for_numpeers=None):
+ d = mutable.MutableFileNode.create(self, initial_contents, wait_for_numpeers=None)
+ def _then(res):
+ self.all_contents[self.get_uri()] = initial_contents
+ return res
+ d.addCallback(_then)
+ return d
+ def init_from_uri(self, myuri):
+ mutable.MutableFileNode.init_from_uri(self, myuri)
+ return self
+ def replace(self, newdata, wait_for_numpeers=None):
+ self.all_contents[self.get_uri()] = initial_contents
+ return defer.succeed(self)
def _generate_pubprivkeys(self):
count = self.counter.next()
return FakePubKey(count), FakePrivKey(count)
- def _publish(self, initial_contents):
- self.all_contents[self._uri] = initial_contents
+ def _publish(self, initial_contents, wait_for_numpeers):
+ self.all_contents[self.get_uri()] = initial_contents
return defer.succeed(self)
def download_to_data(self):
- return defer.succeed(self.all_contents[self._uri])
- def replace(self, newdata):
- self.all_contents[self._uri] = newdata
+ if self.is_readonly():
+ assert self.all_rw_friends.has_key(self.get_uri()), (self.get_uri(), id(self.all_rw_friends))
+ return defer.succeed(self.all_contents[self.all_rw_friends[self.get_uri()]])
+ else:
+ return defer.succeed(self.all_contents[self.get_uri()])
+ def replace(self, newdata, wait_for_numpeers=None):
+ self.all_contents[self.get_uri()] = newdata
return defer.succeed(None)
- def is_readonly(self):
- return False
- def get_readonly(self):
- return "fake readonly"
class FakePublish(mutable.Publish):
def _do_query(self, conn, peerid, peer_storage_servers, storage_index):
class FakeNewDirectoryNode(dirnode2.NewDirectoryNode):
filenode_class = FakeFilenode
-class MyClient:
+class FakeIntroducerClient:
+ def when_enough_peers(self, numpeers):
+ return defer.succeed(None)
+
+class FakeClient:
def __init__(self, num_peers=10):
self._num_peers = num_peers
self._peerids = [tagged_hash("peerid", "%d" % i)[:20]
for i in range(self._num_peers)]
+ self.introducer_client = FakeIntroducerClient()
def log(self, msg):
log.msg(msg)
def get_cancel_secret(self):
return "I hereby permit you to cancel my leases"
- def create_empty_dirnode(self):
+ def create_empty_dirnode(self, wait_for_numpeers):
n = FakeNewDirectoryNode(self)
- d = n.create()
+ d = n.create(wait_for_numpeers=wait_for_numpeers)
d.addCallback(lambda res: n)
return d
def create_dirnode_from_uri(self, u):
return FakeNewDirectoryNode(self).init_from_uri(u)
- def create_mutable_file(self, contents=""):
+ def create_mutable_file(self, contents="", wait_for_numpeers=None):
n = FakeFilenode(self)
- d = n.create(contents)
+ d = n.create(contents, wait_for_numpeers=wait_for_numpeers)
d.addCallback(lambda res: n)
return d
def create_node_from_uri(self, u):
- # this returns synchronously. As a result, it cannot be used to
- # create old-style dirnodes, since those contain a RemoteReference.
- # This means that new-style dirnodes cannot contain old-style
- # dirnodes as children.
u = IURI(u)
if INewDirectoryURI.providedBy(u):
return self.create_dirnode_from_uri(u)
- if IDirnodeURI.providedBy(u):
- raise RuntimeError("not possible, sorry")
- #if IFileURI.providedBy(u):
- # # CHK
- # return FileNode(u, self)
assert IMutableFileURI.providedBy(u)
- return FakeFilenode(self).init_from_uri(u)
+ res = FakeFilenode(self).init_from_uri(u)
+ return res
def get_permuted_peers(self, key, include_myself=True):
"""
class Filenode(unittest.TestCase):
def setUp(self):
- self.client = MyClient()
+ self.client = FakeClient()
def test_create(self):
- d = self.client.create_mutable_file()
+ d = self.client.create_mutable_file(wait_for_numpeers=1)
def _created(n):
d = n.replace("contents 1")
d.addCallback(lambda res: self.failUnlessIdentical(res, None))
class Publish(unittest.TestCase):
def test_encrypt(self):
- c = MyClient()
+ c = FakeClient()
fn = FakeFilenode(c)
# .create usually returns a Deferred, but we happen to know it's
# synchronous
CONTENTS = "some initial contents"
- fn.create(CONTENTS)
+ fn.create(CONTENTS, wait_for_numpeers=1)
p = mutable.Publish(fn)
target_info = None
d = defer.maybeDeferred(p._encrypt_and_encode, target_info,
return d
def test_generate(self):
- c = MyClient()
+ c = FakeClient()
fn = FakeFilenode(c)
# .create usually returns a Deferred, but we happen to know it's
# synchronous
CONTENTS = "some initial contents"
- fn.create(CONTENTS)
+ fn.create(CONTENTS, wait_for_numpeers=1)
p = mutable.Publish(fn)
r = mutable.Retrieve(fn)
# make some fake shares
return d
def setup_for_sharemap(self, num_peers):
- c = MyClient(num_peers)
+ c = FakeClient(num_peers)
fn = FakeFilenode(c)
# .create usually returns a Deferred, but we happen to know it's
# synchronous
return d
def setup_for_publish(self, num_peers):
- c = MyClient(num_peers)
+ c = FakeClient(num_peers)
fn = FakeFilenode(c)
# .create usually returns a Deferred, but we happen to know it's
# synchronous
class Dirnode(unittest.TestCase):
def setUp(self):
- self.client = MyClient()
+ self.client = FakeClient()
def test_create(self):
self.expected_manifest = []
- d = self.client.create_empty_dirnode()
- def _check(n):
+ d = self.client.create_empty_dirnode(wait_for_numpeers=1)
+ def _then(n):
self.failUnless(n.is_mutable())
u = n.get_uri()
self.failUnless(u)
self.failUnless(u.startswith("URI:DIR2:"), u)
- u_ro = n.get_immutable_uri()
+ u_ro = n.get_readonly_uri()
self.failUnless(u_ro.startswith("URI:DIR2-RO:"), u_ro)
u_v = n.get_verifier()
self.failUnless(u_v.startswith("URI:DIR2-Verifier:"), u_v)
ffu_v = fake_file_uri.get_verifier().to_string()
self.expected_manifest.append(ffu_v)
d.addCallback(lambda res: n.set_uri("child", fake_file_uri))
- d.addCallback(lambda res: self.failUnlessEqual(res, None))
- d.addCallback(lambda res: n.create_empty_directory("subdir"))
+ d.addCallback(lambda res: n.create_empty_directory("subdir", wait_for_numpeers=1))
def _created(subdir):
self.failUnless(isinstance(subdir, FakeNewDirectoryNode))
self.subdir = subdir
d.addCallback(_check_manifest)
def _add_subsubdir(res):
- return self.subdir.create_empty_directory("subsubdir")
+ return self.subdir.create_empty_directory("subsubdir", wait_for_numpeers=1)
d.addCallback(_add_subsubdir)
d.addCallback(lambda res: n.get_child_at_path("subdir/subsubdir"))
d.addCallback(lambda subsubdir:
return d
- d.addCallback(_check)
+ d.addCallback(_then)
return d
[],
run_by_human=False)
-class Diagnostics(unittest.TestCase):
- def test_dump_root_dirnode_failure(self):
- s = StringIO()
- config = {'basedirs': ["missing_basedir"]}
- rc = debug.dump_root_dirnode(config, s)
- output = s.getvalue()
- self.failUnless("unable to read root dirnode file from" in output)
- self.failIfEqual(rc, 0)
-
class RunNode(unittest.TestCase, testutil.PollMixin):
def workdir(self, name):
basedir = os.path.join("test_runner", "RunNode", name)
from twisted.internet import threads # CLI tests use deferToThread
from twisted.application import service
from allmydata import client, uri, download, upload
-from allmydata.introducer_and_vdrive import IntroducerAndVdrive
-from allmydata.util import fileutil, testutil, deferredutil, idlib
+from allmydata.introducer import IntroducerNode
+from allmydata.util import deferredutil, fileutil, idlib, mathutil, testutil
from allmydata.scripts import runner
from allmydata.interfaces import IDirectoryNode, IFileNode, IFileURI
-from allmydata.dirnode import NotMutableError
+from allmydata.mutable import NotMutableError
from foolscap.eventual import flushEventualQueue
from twisted.python import log
from twisted.python.failure import Failure
s.setServiceParent(self.sparent)
return s
- def set_up_nodes(self, NUMCLIENTS=5):
+ def set_up_nodes(self, NUMCLIENTS=5, createprivdir=False):
self.numclients = NUMCLIENTS
- iv_dir = self.getdir("introducer_and_vdrive")
+ self.createprivdir = createprivdir
+ iv_dir = self.getdir("introducer")
if not os.path.isdir(iv_dir):
fileutil.make_dirs(iv_dir)
- iv = IntroducerAndVdrive(basedir=iv_dir)
- self.introducer_and_vdrive = self.add_service(iv)
- d = self.introducer_and_vdrive.when_tub_ready()
+ iv = IntroducerNode(basedir=iv_dir)
+ self.introducer = self.add_service(iv)
+ d = self.introducer.when_tub_ready()
d.addCallback(self._set_up_nodes_2)
return d
def _set_up_nodes_2(self, res):
- q = self.introducer_and_vdrive
- self.introducer_furl = q.urls["introducer"]
- self.vdrive_furl = q.urls["vdrive"]
+ q = self.introducer
+ self.introducer_furl = q.introducer_url
self.clients = []
for i in range(self.numclients):
basedir = self.getdir("client%d" % i)
- if not os.path.isdir(basedir):
- fileutil.make_dirs(basedir)
+ fileutil.make_dirs(basedir)
if i == 0:
open(os.path.join(basedir, "webport"), "w").write("tcp:0:interface=127.0.0.1")
+ if self.createprivdir:
+ open(os.path.join(basedir, "my_private_dir.uri"), "w")
open(os.path.join(basedir, "introducer.furl"), "w").write(self.introducer_furl)
- open(os.path.join(basedir, "vdrive.furl"), "w").write(self.vdrive_furl)
c = self.add_service(client.Client(basedir=basedir))
self.clients.append(c)
log.msg("STARTING")
if not os.path.isdir(basedir):
fileutil.make_dirs(basedir)
open(os.path.join(basedir, "introducer.furl"), "w").write(self.introducer_furl)
- open(os.path.join(basedir, "vdrive.furl"), "w").write(self.vdrive_furl)
c = client.Client(basedir=basedir)
self.clients.append(c)
self.basedir = "system/SystemTest/test_connections"
d = self.set_up_nodes()
self.extra_node = None
- d.addCallback(lambda res: self.add_extra_node(5))
+ d.addCallback(lambda res: self.add_extra_node(self.numclients))
def _check(extra_node):
self.extra_node = extra_node
for c in self.clients:
all_peerids = list(c.get_all_peerids())
- self.failUnlessEqual(len(all_peerids), 6)
+ self.failUnlessEqual(len(all_peerids), self.numclients+1)
permuted_peers = list(c.get_permuted_peers("a", True))
- self.failUnlessEqual(len(permuted_peers), 6)
+ self.failUnlessEqual(len(permuted_peers), self.numclients+1)
permuted_other_peers = list(c.get_permuted_peers("a", False))
- self.failUnlessEqual(len(permuted_other_peers), 5)
+ self.failUnlessEqual(len(permuted_other_peers), self.numclients)
d.addCallback(_check)
def _shutdown_extra_node(res):
def _check_connections(res):
for c in self.clients:
all_peerids = list(c.get_all_peerids())
- self.failUnlessEqual(len(all_peerids), 5)
+ self.failUnlessEqual(len(all_peerids), self.numclients)
permuted_peers = list(c.get_permuted_peers("a", True))
- self.failUnlessEqual(len(permuted_peers), 5)
+ self.failUnlessEqual(len(permuted_peers), self.numclients)
permuted_other_peers = list(c.get_permuted_peers("a", False))
- self.failUnlessEqual(len(permuted_other_peers), 4)
+ self.failUnlessEqual(len(permuted_other_peers), self.numclients-1)
d.addCallback(_check_connections)
def _do_upload(res):
log.msg("UPLOADING")
NEWERDATA = "this is getting old"
d = self.set_up_nodes()
-
def _create_mutable(res):
c = self.clients[0]
log.msg("starting create_mutable_file")
- d1 = c.create_mutable_file(DATA)
+ d1 = c.create_mutable_file(DATA, wait_for_numpeers=self.numclients)
def _done(res):
log.msg("DONE: %s" % (res,))
self._mutable_node_1 = res
m = re.search(r'^ container_size: (\d+)$', output, re.M)
self.failUnless(m)
container_size = int(m.group(1))
- self.failUnless(2044 <= container_size <= 2049, container_size)
+ self.failUnless(2037 <= container_size <= 2049, container_size)
m = re.search(r'^ data_length: (\d+)$', output, re.M)
self.failUnless(m)
data_length = int(m.group(1))
- self.failUnless(2044 <= data_length <= 2049, data_length)
+ self.failUnless(2037 <= data_length <= 2049, data_length)
self.failUnless(" secrets are for nodeid: %s\n" % peerid
in output)
self.failUnless(" SDMF contents:\n" in output)
self.failUnless(" seqnum: 1\n" in output)
self.failUnless(" required_shares: 3\n" in output)
self.failUnless(" total_shares: 10\n" in output)
- self.failUnless(" segsize: 27\n" in output)
+ self.failUnless(" segsize: 27\n" in output, (output, filename))
self.failUnless(" datalen: 25\n" in output)
# the exact share_hash_chain nodes depends upon the sharenum,
# and is more of a hassle to compute than I want to deal with
self.failUnlessEqual(res, DATA)
# replace the data
log.msg("starting replace1")
- d1 = newnode.replace(NEWDATA)
+ d1 = newnode.replace(NEWDATA, wait_for_numpeers=self.numclients)
d1.addCallback(lambda res: newnode.download_to_data())
return d1
d.addCallback(_check_download_3)
newnode1 = self.clients[2].create_node_from_uri(uri)
newnode2 = self.clients[3].create_node_from_uri(uri)
log.msg("starting replace2")
- d1 = newnode1.replace(NEWERDATA)
+ d1 = newnode1.replace(NEWERDATA, wait_for_numpeers=self.numclients)
d1.addCallback(lambda res: newnode2.download_to_data())
return d1
d.addCallback(_check_download_4)
def _check_download_5(res):
log.msg("finished replace2")
self.failUnlessEqual(res, NEWERDATA)
- # make sure we can create empty files, this usually screws up the
- # segsize math
- d1 = self.clients[2].create_mutable_file("")
+ # Make sure we can create empty files -- this can screw up the
+ # segsize math.
+ d1 = self.clients[2].create_mutable_file("", wait_for_numpeers=self.numclients)
d1.addCallback(lambda newnode: newnode.download_to_data())
d1.addCallback(lambda res: self.failUnlessEqual("", res))
return d1
d.addCallback(_check_download_5)
- d.addCallback(lambda res: self.clients[0].create_empty_dirnode())
+ d.addCallback(lambda res: self.clients[0].create_empty_dirnode(wait_for_numpeers=self.numclients))
def _created_dirnode(dnode):
+ log.msg("_created_dirnode(%s)" % (dnode,))
d1 = dnode.list()
d1.addCallback(lambda children: self.failUnlessEqual(children, {}))
d1.addCallback(lambda res: dnode.has_child("edgar"))
d1.addCallback(lambda answer: self.failUnlessEqual(answer, False))
- d1.addCallback(lambda res: dnode.set_node("see recursive", dnode))
+ d1.addCallback(lambda res: dnode.set_node("see recursive", dnode, wait_for_numpeers=self.numclients))
d1.addCallback(lambda res: dnode.has_child("see recursive"))
d1.addCallback(lambda answer: self.failUnlessEqual(answer, True))
return d1
def test_vdrive(self):
self.basedir = "system/SystemTest/test_vdrive"
self.data = LARGE_DATA
- d = self.set_up_nodes()
+ d = self.set_up_nodes(createprivdir=True)
d.addCallback(self.log, "starting publish")
d.addCallback(self._do_publish1)
d.addCallback(self._test_runner)
d.addCallback(self._do_publish2)
- # at this point, we have the following global filesystem:
- # /
- # /subdir1
- # /subdir1/mydata567
- # /subdir1/subdir2/
- # /subdir1/subdir2/mydata992
+ # at this point, we have the following filesystem (where "R" denotes
+ # self._root_directory_uri):
+ # R
+ # R/subdir1
+ # R/subdir1/mydata567
+ # R/subdir1/subdir2/
+ # R/subdir1/subdir2/mydata992
d.addCallback(self._bounce_client0)
d.addCallback(self.log, "bounced client0")
d.addCallback(self.log, "did _check_publish2")
d.addCallback(self._do_publish_private)
d.addCallback(self.log, "did _do_publish_private")
- # now we also have:
- # ~client0/personal/sekrit data
- # ~client0/s2-rw -> /subdir1/subdir2/
- # ~client0/s2-ro -> /subdir1/subdir2/ (read-only)
+ # now we also have (where "P" denotes clients[0]'s automatic private
+ # dir):
+ # P/personal/sekrit data
+ # P/s2-rw -> /subdir1/subdir2/
+ # P/s2-ro -> /subdir1/subdir2/ (read-only)
d.addCallback(self._check_publish_private)
d.addCallback(self.log, "did _check_publish_private")
d.addCallback(self._test_web)
def _do_publish1(self, res):
ut = upload.Data(self.data)
c0 = self.clients[0]
- d = c0.getServiceNamed("vdrive").get_public_root()
- d.addCallback(lambda root: root.create_empty_directory("subdir1"))
+ d = c0.create_empty_dirnode(wait_for_numpeers=self.numclients)
+ def _made_root(new_dirnode):
+ log.msg("ZZZ %s -> %s" % (hasattr(self, '_root_directory_uri') and self._root_directory_uri, new_dirnode.get_uri(),))
+ self._root_directory_uri = new_dirnode.get_uri()
+ return c0.create_node_from_uri(self._root_directory_uri)
+ d.addCallback(_made_root)
+ d.addCallback(lambda root: root.create_empty_directory("subdir1", wait_for_numpeers=self.numclients))
def _made_subdir1(subdir1_node):
self._subdir1_node = subdir1_node
- d1 = subdir1_node.add_file("mydata567", ut)
+ d1 = subdir1_node.add_file("mydata567", ut, wait_for_numpeers=self.numclients)
d1.addCallback(self.log, "publish finished")
def _stash_uri(filenode):
self.uri = filenode.get_uri()
def _do_publish2(self, res):
ut = upload.Data(self.data)
- d = self._subdir1_node.create_empty_directory("subdir2")
- d.addCallback(lambda subdir2: subdir2.add_file("mydata992", ut))
+ d = self._subdir1_node.create_empty_directory("subdir2", wait_for_numpeers=self.numclients)
+ d.addCallback(lambda subdir2: subdir2.add_file("mydata992", ut, wait_for_numpeers=self.numclients))
return d
def _bounce_client0(self, res):
return d
def log(self, res, msg):
- #print "MSG: %s RES: %s" % (msg, res)
+ # print "MSG: %s RES: %s" % (msg, res)
log.msg(msg)
return res
return d
def _do_publish_private(self, res):
+ defer.setDebugging(True)
self.smalldata = "sssh, very secret stuff"
ut = upload.Data(self.smalldata)
- vdrive0 = self.clients[0].getServiceNamed("vdrive")
- d = vdrive0.get_node_at_path("~")
- d.addCallback(self.log, "GOT ~")
- def _got_root(rootnode):
- d1 = rootnode.create_empty_directory("personal")
- d1.addCallback(self.log, "made ~/personal")
- d1.addCallback(lambda node: node.add_file("sekrit data", ut))
- d1.addCallback(self.log, "made ~/personal/sekrit data")
- d1.addCallback(lambda res:
- vdrive0.get_node_at_path(["subdir1", "subdir2"]))
+ d = self.clients[0].get_private_uri()
+ d.addCallback(self.log, "GOT private directory")
+ def _got_root_uri(privuri):
+ assert privuri
+ privnode = self.clients[0].create_node_from_uri(privuri)
+ rootnode = self.clients[0].create_node_from_uri(self._root_directory_uri)
+ d1 = privnode.create_empty_directory("personal", wait_for_numpeers=self.numclients)
+ d1.addCallback(self.log, "made P/personal")
+ d1.addCallback(lambda node: node.add_file("sekrit data", ut, wait_for_numpeers=self.numclients))
+ d1.addCallback(self.log, "made P/personal/sekrit data")
+ d1.addCallback(lambda res: rootnode.get_child_at_path(["subdir1", "subdir2"]))
def _got_s2(s2node):
- d2 = rootnode.set_uri("s2-rw", s2node.get_uri())
- d2.addCallback(lambda node:
- rootnode.set_uri("s2-ro",
- s2node.get_immutable_uri()))
+ d2 = privnode.set_uri("s2-rw", s2node.get_uri(), wait_for_numpeers=self.numclients)
+ d2.addCallback(lambda node: privnode.set_uri("s2-ro", s2node.get_readonly_uri(), wait_for_numpeers=self.numclients))
return d2
d1.addCallback(_got_s2)
return d1
- d.addCallback(_got_root)
+ d.addCallback(_got_root_uri)
return d
def _check_publish1(self, res):
# this one uses the iterative API
c1 = self.clients[1]
- d = c1.getServiceNamed("vdrive").get_public_root()
+ d = defer.succeed(c1.create_node_from_uri(self._root_directory_uri))
d.addCallback(self.log, "check_publish1 got /")
d.addCallback(lambda root: root.get("subdir1"))
d.addCallback(lambda subdir1: subdir1.get("mydata567"))
def _check_publish2(self, res):
# this one uses the path-based API
- vdrive1 = self.clients[1].getServiceNamed("vdrive")
- get_path = vdrive1.get_node_at_path
- d = get_path("subdir1")
+ rootnode = self.clients[1].create_node_from_uri(self._root_directory_uri)
+ d = rootnode.get_child_at_path("subdir1")
d.addCallback(lambda dirnode:
self.failUnless(IDirectoryNode.providedBy(dirnode)))
- d.addCallback(lambda res: get_path("/subdir1/mydata567"))
+ d.addCallback(lambda res: rootnode.get_child_at_path("subdir1/mydata567"))
d.addCallback(lambda filenode: filenode.download_to_data())
d.addCallback(lambda data: self.failUnlessEqual(data, self.data))
- d.addCallback(lambda res: get_path("subdir1/mydata567"))
+ d.addCallback(lambda res: rootnode.get_child_at_path("subdir1/mydata567"))
def _got_filenode(filenode):
- d1 = vdrive1.get_node(filenode.get_uri())
- d1.addCallback(self.failUnlessEqual, filenode)
- return d1
+ fnode = self.clients[1].create_node_from_uri(filenode.get_uri())
+ assert fnode == filenode
d.addCallback(_got_filenode)
return d
def _check_publish_private(self, res):
# this one uses the path-based API
- def get_path(path):
- vdrive0 = self.clients[0].getServiceNamed("vdrive")
- return vdrive0.get_node_at_path(path)
- d = get_path("~/personal")
+ d = self.clients[0].get_private_uri()
+ def _got_private_uri(privateuri):
+ self._private_node = self.clients[0].create_node_from_uri(privateuri)
+ d.addCallback(_got_private_uri)
+
+ d.addCallback(lambda res: self._private_node.get_child_at_path("personal"))
def _got_personal(personal):
self._personal_node = personal
return personal
d.addCallback(_got_personal)
+
d.addCallback(lambda dirnode:
- self.failUnless(IDirectoryNode.providedBy(dirnode)))
- d.addCallback(lambda res: get_path("~/personal/sekrit data"))
+ self.failUnless(IDirectoryNode.providedBy(dirnode), dirnode))
+ def get_path(path):
+ return self._private_node.get_child_at_path(path)
+
+ d.addCallback(lambda res: get_path("personal/sekrit data"))
d.addCallback(lambda filenode: filenode.download_to_data())
d.addCallback(lambda data: self.failUnlessEqual(data, self.smalldata))
- d.addCallback(lambda res: get_path("~/s2-rw"))
+ d.addCallback(lambda res: get_path("s2-rw"))
d.addCallback(lambda dirnode: self.failUnless(dirnode.is_mutable()))
- d.addCallback(lambda res: get_path("~/s2-ro"))
+ d.addCallback(lambda res: get_path("s2-ro"))
def _got_s2ro(dirnode):
- self.failIf(dirnode.is_mutable())
+ self.failUnless(dirnode.is_mutable(), dirnode)
+ self.failUnless(dirnode.is_readonly(), dirnode)
d1 = defer.succeed(None)
d1.addCallback(lambda res: dirnode.list())
d1.addCallback(self.log, "dirnode.list")
- d1.addCallback(lambda res: dirnode.create_empty_directory("nope"))
- d1.addBoth(self.shouldFail, NotMutableError, "mkdir(nope)")
+
+ d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "mkdir(nope)", None, dirnode.create_empty_directory, "nope"))
+
d1.addCallback(self.log, "doing add_file(ro)")
ut = upload.Data("I will disappear, unrecorded and unobserved. The tragedy of my demise is made more poignant by its silence, but this beauty is not for you to ever know.")
- d1.addCallback(lambda res: dirnode.add_file("hope", ut))
- d1.addBoth(self.shouldFail, NotMutableError, "add_file(nope)")
+ d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "add_file(nope)", None, dirnode.add_file, "hope", ut))
d1.addCallback(self.log, "doing get(ro)")
d1.addCallback(lambda res: dirnode.get("mydata992"))
self.failUnless(IFileNode.providedBy(filenode)))
d1.addCallback(self.log, "doing delete(ro)")
- d1.addCallback(lambda res: dirnode.delete("mydata992"))
- d1.addBoth(self.shouldFail, NotMutableError, "delete(nope)")
+ d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "delete(nope)", None, dirnode.delete, "mydata992"))
- d1.addCallback(lambda res: dirnode.set_uri("hopeless", self.uri))
- d1.addBoth(self.shouldFail, NotMutableError, "set_uri(nope)")
+ d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "set_uri(nope)", None, dirnode.set_uri, "hopeless", self.uri))
- d1.addCallback(lambda res: dirnode.get("missing"))
- d1.addBoth(self.shouldFail, KeyError, "get(missing)",
- "unable to find child named 'missing'")
+ d1.addCallback(lambda res: self.shouldFail2(KeyError, "get(missing)", "'missing'", dirnode.get, "missing"))
- d1.addCallback(self.log, "doing move_child_to(ro)")
personal = self._personal_node
- d1.addCallback(lambda res:
- dirnode.move_child_to("mydata992",
- personal, "nope"))
- d1.addBoth(self.shouldFail, NotMutableError, "mv from readonly")
+ d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "mv from readonly", None, dirnode.move_child_to, "mydata992", personal, "nope"))
d1.addCallback(self.log, "doing move_child_to(ro)2")
- d1.addCallback(lambda res:
- personal.move_child_to("sekrit data",
- dirnode, "nope"))
- d1.addBoth(self.shouldFail, NotMutableError, "mv to readonly")
+ d1.addCallback(lambda res: self.shouldFail2(NotMutableError, "mv to readonly", None, personal.move_child_to, "sekrit data", dirnode, "nope"))
d1.addCallback(self.log, "finished with _got_s2ro")
return d1
d.addCallback(_got_s2ro)
- d.addCallback(lambda res: get_path("~"))
- def _got_home(home):
+ def _got_home(dummy):
+ home = self._private_node
personal = self._personal_node
d1 = defer.succeed(None)
- d1.addCallback(self.log, "mv '~/personal/sekrit data' to ~/sekrit")
+ d1.addCallback(self.log, "mv 'P/personal/sekrit data' to P/sekrit")
d1.addCallback(lambda res:
personal.move_child_to("sekrit data",home,"sekrit"))
- d1.addCallback(self.log, "mv ~/sekrit '~/sekrit data'")
+ d1.addCallback(self.log, "mv P/sekrit 'P/sekrit data'")
d1.addCallback(lambda res:
home.move_child_to("sekrit", home, "sekrit data"))
- d1.addCallback(self.log, "mv '~/sekret data' ~/personal/")
+ d1.addCallback(self.log, "mv 'P/sekret data' P/personal/")
d1.addCallback(lambda res:
home.move_child_to("sekrit data", personal))
d1.addCallback(lambda res: home.build_manifest())
d1.addCallback(self.log, "manifest")
# four items:
- # ~client0/personal/
- # ~client0/personal/sekrit data
- # ~client0/s2-rw (same as ~client/s2-ro)
- # ~client0/s2-rw/mydata992 (same as ~client/s2-rw/mydata992)
+ # P/personal/
+ # P/personal/sekrit data
+ # P/s2-rw (same as P/s2-ro)
+ # P/s2-rw/mydata992 (same as P/s2-rw/mydata992)
d1.addCallback(lambda manifest:
self.failUnlessEqual(len(manifest), 4))
return d1
self.fail("%s was supposed to raise %s, not get '%s'" %
(which, expected_failure, res))
+ def shouldFail2(self, expected_failure, which, substring, callable, *args, **kwargs):
+ assert substring is None or isinstance(substring, str)
+ d = defer.maybeDeferred(callable, *args, **kwargs)
+ def done(res):
+ if isinstance(res, Failure):
+ res.trap(expected_failure)
+ if substring:
+ self.failUnless(substring in str(res),
+ "substring '%s' not in '%s'"
+ % (substring, str(res)))
+ else:
+ self.fail("%s was supposed to raise %s, not get '%s'" %
+ (which, expected_failure, res))
+ d.addBoth(done)
+ return d
+
def PUT(self, urlpath, data):
url = self.webish_url + urlpath
return getPage(url, method="PUT", postdata=data)
def _test_web(self, res):
base = self.webish_url
+ public = "uri/" + self._root_directory_uri.replace("/", "!")
d = getPage(base)
def _got_welcome(page):
expected = "Connected Peers: <span>%d</span>" % (self.numclients)
"in: %s" % page)
d.addCallback(_got_welcome)
d.addCallback(self.log, "done with _got_welcome")
- d.addCallback(lambda res: getPage(base + "vdrive/global"))
- d.addCallback(lambda res: getPage(base + "vdrive/global/subdir1"))
+ d.addCallback(lambda res: getPage(base + public))
+ d.addCallback(lambda res: getPage(base + public + "/subdir1"))
def _got_subdir1(page):
# there ought to be an href for our file
self.failUnless(("<td>%d</td>" % len(self.data)) in page)
d.addCallback(_got_subdir1)
d.addCallback(self.log, "done with _got_subdir1")
d.addCallback(lambda res:
- getPage(base + "vdrive/global/subdir1/mydata567"))
+ getPage(base + public + "/subdir1/mydata567"))
def _got_data(page):
self.failUnlessEqual(page, self.data)
d.addCallback(_got_data)
def _get_from_uri2(res):
return getPage(base + "uri?uri=%s" % (self.uri,))
d.addCallback(_get_from_uri2)
- def _got_from_uri2(page):
- self.failUnlessEqual(page, self.data)
- d.addCallback(_got_from_uri2)
+ d.addCallback(_got_from_uri)
# download from a bogus URI, make sure we get a reasonable error
d.addCallback(self.log, "_get_from_bogus_uri")
# upload a file with PUT
d.addCallback(self.log, "about to try PUT")
- d.addCallback(lambda res: self.PUT("vdrive/global/subdir3/new.txt",
+ d.addCallback(lambda res: self.PUT(public + "/subdir3/new.txt",
"new.txt contents"))
- d.addCallback(lambda res: self.GET("vdrive/global/subdir3/new.txt"))
+ d.addCallback(lambda res: self.GET(public + "/subdir3/new.txt"))
d.addCallback(self.failUnlessEqual, "new.txt contents")
# and again with something large enough to use multiple segments,
# and hopefully trigger pauseProducing too
- d.addCallback(lambda res: self.PUT("vdrive/global/subdir3/big.txt",
+ d.addCallback(lambda res: self.PUT(public + "/subdir3/big.txt",
"big" * 500000)) # 1.5MB
- d.addCallback(lambda res: self.GET("vdrive/global/subdir3/big.txt"))
+ d.addCallback(lambda res: self.GET(public + "/subdir3/big.txt"))
d.addCallback(lambda res: self.failUnlessEqual(len(res), 1500000))
# can we replace files in place?
- d.addCallback(lambda res: self.PUT("vdrive/global/subdir3/new.txt",
+ d.addCallback(lambda res: self.PUT(public + "/subdir3/new.txt",
"NEWER contents"))
- d.addCallback(lambda res: self.GET("vdrive/global/subdir3/new.txt"))
+ d.addCallback(lambda res: self.GET(public + "/subdir3/new.txt"))
d.addCallback(self.failUnlessEqual, "NEWER contents")
# TODO: download a URI with a form
# TODO: create a directory by using a form
# TODO: upload by using a form on the directory page
- # url = base + "global_vdrive/subdir1/freeform_post!!upload"
+ # url = base + "somedir/subdir1/freeform_post!!upload"
# TODO: delete a file by using a button on the directory page
return d
self.failUnless(os.path.exists(startfile))
start_html = open(startfile, "r").read()
self.failUnless(self.webish_url in start_html)
- private_uri = self.clients[0].getServiceNamed("vdrive")._private_uri
- private_url = self.webish_url + "uri/" + private_uri.replace("/","!")
- self.failUnless(private_url in start_html)
+ d = self.clients[0].get_private_uri()
+ def done(private_uri):
+ private_url = self.webish_url + "uri/" + private_uri.replace("/","!")
+ self.failUnless(private_url in start_html)
+ d.addCallback(done)
+ return d
def _test_runner(self, res):
# exercise some of the diagnostic tools in runner.py
# we're sitting in .../storage/shares/$SINDEX , and there are
# sharefiles here
filename = os.path.join(dirpath, filenames[0])
- break
+ # peek at the magic to see if it is a chk share
+ magic = open(filename, "rb").read(4)
+ if magic == '\x00\x00\x00\x01':
+ break
else:
self.fail("unable to find any uri_extension files in %s"
% self.basedir)
self.failUnless("size: %d\n" % len(self.data) in output)
self.failUnless("num_segments: 1\n" in output)
# segment_size is always a multiple of needed_shares
- self.failUnless("segment_size: 114\n" in output)
+ self.failUnless("segment_size: %d\n" % mathutil.next_multiple(len(self.data), 3) in output)
self.failUnless("total_shares: 10\n" in output)
# keys which are supposed to be present
for key in ("size", "num_segments", "segment_size",
"codec_name", "codec_params", "tail_codec_params",
"plaintext_hash", "plaintext_root_hash",
"crypttext_hash", "crypttext_root_hash",
- "share_root_hash",
- ):
+ "share_root_hash",):
self.failUnless("%s: " % key in output, key)
def _test_control(self, res):
control_furl_file = os.path.join(c0.basedir, "control.furl")
control_furl = open(control_furl_file, "r").read().strip()
# it doesn't really matter which Tub we use to connect to the client,
- # so let's just use our Introducer's
- d = self.introducer_and_vdrive.tub.getReference(control_furl)
+ # so let's just use our IntroducerNode's
+ d = self.introducer.tub.getReference(control_furl)
d.addCallback(self._test_control2, control_furl_file)
return d
def _test_control2(self, rref, filename):
# run various CLI commands (in a thread, since they use blocking
# network calls)
- private_uri = self.clients[0].getServiceNamed("vdrive")._private_uri
- global_uri = self.clients[0].getServiceNamed("vdrive")._global_uri
+ private_uri = self._private_node.get_uri()
+ some_uri = self._root_directory_uri
+
nodeargs = [
"--node-url", self.webish_url,
"--root-uri", private_uri,
]
public_nodeargs = [
"--node-url", self.webish_url,
- "--root-uri", global_uri,
+ "--root-uri", some_uri,
]
TESTDATA = "I will not write the same thing over and over.\n" * 100
def _check_put((out,err)):
self.failUnless("200 OK" in out)
self.failUnlessEqual(err, "")
- vdrive0 = self.clients[0].getServiceNamed("vdrive")
- d = vdrive0.get_node_at_path("~/test_put/upload.txt")
+ d = self._private_node.get_child_at_path("test_put/upload.txt")
d.addCallback(lambda filenode: filenode.download_to_data())
def _check_put2(res):
self.failUnlessEqual(res, TESTDATA)
def _check_mv((out,err)):
self.failUnless("OK" in out)
self.failUnlessEqual(err, "")
- vdrive0 = self.clients[0].getServiceNamed("vdrive")
- d = defer.maybeDeferred(vdrive0.get_node_at_path,
- "~/test_put/upload.txt")
- d.addBoth(self.shouldFail, KeyError, "test_cli._check_rm",
- "unable to find child named 'upload.txt'")
+ d = self.shouldFail2(KeyError, "test_cli._check_rm", "'upload.txt'", self._private_node.get_child_at_path, "test_put/upload.txt")
+
d.addCallback(lambda res:
- vdrive0.get_node_at_path("~/test_put/moved.txt"))
+ self._private_node.get_child_at_path("test_put/moved.txt"))
d.addCallback(lambda filenode: filenode.download_to_data())
def _check_mv2(res):
self.failUnlessEqual(res, TESTDATA)
def _check_rm((out,err)):
self.failUnless("200 OK" in out)
self.failUnlessEqual(err, "")
- vdrive0 = self.clients[0].getServiceNamed("vdrive")
- d = defer.maybeDeferred(vdrive0.get_node_at_path,
- "~/test_put/moved.txt")
- d.addBoth(self.shouldFail, KeyError, "test_cli._check_rm",
- "unable to find child named 'moved.txt'")
+ d = self.shouldFail2(KeyError, "test_cli._check_rm", "'moved.txt'", self._private_node.get_child_at_path, "test_put/moved.txt")
return d
d.addCallback(_check_rm)
return d
return d
def _test_checker(self, res):
- vdrive0 = self.clients[0].getServiceNamed("vdrive")
- d = vdrive0.get_node_at_path("~")
- d.addCallback(lambda home: home.build_manifest())
+ d = self._private_node.build_manifest()
d.addCallback(self._test_checker_2)
return d
all_results = []
for si in manifest:
results = checker1.checker_results_for(si)
+ if not results:
+ # TODO: implement checker for mutable files and implement tests of that checker
+ continue
self.failUnlessEqual(len(results), 1)
when, those_results = results[0]
self.failUnless(isinstance(when, (int, float)))
return d
def _test_verifier(self, res):
- vdrive0 = self.clients[0].getServiceNamed("vdrive")
checker1 = self.clients[1].getServiceNamed("checker")
- d = vdrive0.get_node_at_path("~")
- d.addCallback(lambda home: home.build_manifest())
+ d = self._private_node.build_manifest()
def _check_all(manifest):
dl = []
for si in manifest:
self.failUnless(i is True)
d.addCallback(_done)
return d
-
precondition(not self.closed)
self.closed = True
+class FakeIntroducerClient:
+ def when_enough_peers(self, numpeers):
+ return defer.succeed(None)
+
class FakeClient:
def __init__(self, mode="good", num_servers=50):
self.mode = mode
self.num_servers = num_servers
+ self.introducer_client = FakeIntroducerClient()
def get_permuted_peers(self, storage_index, include_myself):
peers = [ ("%20d"%fakeid, "%20d"%fakeid, FakePeer(self.mode),)
for fakeid in range(self.num_servers) ]
self.u.parent = self.node
def _should_fail(self, f):
- self.failUnless(isinstance(f, Failure) and f.check(encode.NotEnoughPeersError))
+ self.failUnless(isinstance(f, Failure) and f.check(encode.NotEnoughPeersError), f)
def test_data_large(self):
data = DATA
from twisted.internet import defer
from twisted.web import client, error, http
from twisted.python import failure, log
-from allmydata import webish, interfaces, dirnode, uri, provisioning
+from allmydata import dirnode2, webish, interfaces, uri, provisioning, filenode
from allmydata.encode import NotEnoughPeersError
from allmydata.util import fileutil
import itertools
+import test_mutable
+
# create a fake uploader/downloader, and a couple of fake dirnodes, then
# create a webserver that works against them
def get_all_peerids(self):
return []
- def upload(self, uploadable):
+ def create_node_from_uri(self, uri):
+ return self.my_nodes[uri]
+
+ def create_empty_dirnode(self, wait_for_numpeers=None):
+ n = FakeDirectoryNode(self)
+ r = defer.succeed(n.fake_create(wait_for_numpeers=1))
+ self.my_nodes[n.get_uri()] = n
+ nro = FakeDirectoryNode(self)
+ nro.init_from_uri(n.get_readonly_uri())
+ nro._node.all_rw_friends[nro._node.get_uri()] = n._node.get_uri()
+ self.my_nodes[nro.get_uri()] = nro
+ return r
+
+ def upload(self, uploadable, wait_for_numpeers=None):
uploader = self.getServiceNamed("uploader")
- return uploader.upload(uploadable)
+ return uploader.upload(uploadable, wait_for_numpeers=wait_for_numpeers)
class MyDownloader(service.Service):
self.files = files
self.nodes = nodes
- def upload(self, uploadable):
+ def upload(self, uploadable, wait_for_numpeers=None):
d = uploadable.get_size()
d.addCallback(lambda size: uploadable.read(size))
d.addCallback(lambda data: "".join(data))
d.addCallback(_got_data)
return d
-class MyDirectoryNode(dirnode.MutableDirectoryNode):
-
- def __init__(self, nodes, files, client, myuri=None):
- self._my_nodes = nodes
- self._my_files = files
- self._my_client = client
- if myuri is None:
- u = uri.DirnodeURI("furl", "idx%s" % str(uri_counter.next()))
- myuri = u.to_string()
- self._uri = myuri
- self._my_nodes[self._uri] = self
- self.children = {}
- self._mutable = True
-
- def get(self, name):
- def _try():
- uri = self.children[name]
- if uri not in self._my_nodes:
- raise IndexError("this isn't supposed to happen")
- return self._my_nodes[uri]
- return defer.maybeDeferred(_try)
-
- def set_uri(self, name, child_uri):
- self.children[name] = child_uri
- return defer.succeed(None)
-
- def add_file(self, name, uploadable):
- d = uploadable.get_size()
- d.addCallback(lambda size: uploadable.read(size))
- d.addCallback(lambda data: "".join(data))
- def _got_data(data):
- newuri = make_newuri(data)
- self._my_files[newuri] = data
- self._my_nodes[newuri] = MyFileNode(newuri, self._my_client)
- self.children[name] = newuri
- uploadable.close()
- return self._my_nodes[newuri]
- d.addCallback(_got_data)
- return d
+def syncwrap(meth):
+ """
+ syncwrap invokes a method, assumes that it fired its deferred
+ synchronously, and returns the result. syncwrap is convenient to use as a
+ decorator in FakeDirectoryNode."""
+ def _syncwrapped_meth(self, *args, **kwargs):
+ l = []
+ d = meth(self, *args, **kwargs)
+ d.addCallback(l.append)
+ assert len(l) == 1, l
+ return l[0]
+ return _syncwrapped_meth
- def delete(self, name):
- def _try():
- del self.children[name]
- return defer.maybeDeferred(_try)
+class FakeDirectoryNode(dirnode2.NewDirectoryNode):
+ filenode_class = test_mutable.FakeFilenode
- def create_empty_directory(self, name):
- node = MyDirectoryNode(self._my_nodes, self._my_files, self._my_client)
- self.children[name] = node.get_uri()
- return defer.succeed(node)
+ @syncwrap
+ def fake_create(self, wait_for_numpeers=None):
+ return self.create(wait_for_numpeers=wait_for_numpeers)
- def list(self):
- kids = dict([(name, self._my_nodes[uri])
- for name,uri in self.children.iteritems()])
- return defer.succeed(kids)
+ @syncwrap
+ def fake_has_child(self, name):
+ return self.has_child(name)
-class MyFileNode(dirnode.FileNode):
- pass
+ @syncwrap
+ def fake_get(self, name):
+ return self.get(name)
+ @syncwrap
+ def fake_list(self):
+ return self.list()
-class MyVirtualDrive(service.Service):
- name = "vdrive"
- public_root = None
- private_root = None
- def __init__(self, nodes, files):
- self._my_nodes = nodes
- self._my_files = files
- def have_public_root(self):
- return bool(self.public_root)
- def have_private_root(self):
- return bool(self.private_root)
- def get_public_root(self):
- return defer.succeed(self.public_root)
- def get_private_root(self):
- return defer.succeed(self.private_root)
-
- def get_node(self, uri):
- def _try():
- return self._my_nodes[uri]
- return defer.maybeDeferred(_try)
-
- def create_directory(self):
- # the dirnode adds itself to self.nodes
- dirnode = MyDirectoryNode(self._my_nodes, self._my_files, self.parent)
- return defer.succeed(dirnode)
+ @syncwrap
+ def fake_set_uri(self, name, uri):
+ return self.set_uri(name, uri)
+class MyFileNode(filenode.FileNode):
+ pass
class WebMixin(object):
def setUp(self):
self.webish_url = "http://localhost:%d" % port
self.nodes = {} # maps URI to node
+ self.s.my_nodes = self.nodes
self.files = {} # maps file URI to contents
- v = MyVirtualDrive(self.nodes, self.files)
- v.setServiceParent(self.s)
-
dl = MyDownloader(self.files)
dl.setServiceParent(self.s)
ul = MyUploader(self.files, self.nodes)
ul.setServiceParent(self.s)
- v.public_root = self.makedir()
- self.public_root = v.public_root
- v.private_root = self.makedir()
- foo = self.makedir()
- self._foo_node = foo
- self._foo_uri = foo.get_uri()
- self._foo_readonly_uri = foo.get_immutable_uri()
- v.public_root.children["foo"] = foo.get_uri()
-
-
- self._bar_txt_uri = self.makefile(0)
- self.BAR_CONTENTS = self.files[self._bar_txt_uri]
- foo.children["bar.txt"] = self._bar_txt_uri
- foo.children["empty"] = self.makedir().get_uri()
- sub_uri = foo.children["sub"] = self.makedir().get_uri()
- sub = self.nodes[sub_uri]
-
- blocking_uri = self.make_smallfile(1)
- foo.children["blockingfile"] = blocking_uri
-
- baz_file = self.makefile(2)
- sub.children["baz.txt"] = baz_file
-
- self._bad_file_uri = self.makefile(3)
- del self.files[self._bad_file_uri]
-
- rodir = self.makedir()
- rodir._mutable = False
- v.public_root.children["readonly"] = rodir.get_uri()
- rodir.children["nor"] = baz_file
-
- # public/
- # public/foo/
- # public/foo/bar.txt
- # public/foo/blockingfile
- # public/foo/empty/
- # public/foo/sub/
- # public/foo/sub/baz.txt
- # public/readonly/
- # public/readonly/nor
- self.NEWFILE_CONTENTS = "newfile contents\n"
+ l = [ self.s.create_empty_dirnode() for x in range(6) ]
+ d = defer.DeferredList(l)
+ def _then(res):
+ self.public_root = res[0][1]
+ assert interfaces.IDirectoryNode.providedBy(self.public_root), res
+ self.public_url = "/uri/" + self.public_root.get_uri()
+ self.private_root = res[1][1]
+
+ foo = res[2][1]
+ self._foo_node = foo
+ self._foo_uri = foo.get_uri()
+ self._foo_readonly_uri = foo.get_readonly_uri()
+ self.public_root.set_uri("foo", foo.get_uri()) # ignore the deferred because we know the fake one does this synchronously
+
+ self._bar_txt_uri = self.makefile(0)
+ self.BAR_CONTENTS = self.files[self._bar_txt_uri]
+ foo.set_uri("bar.txt", self._bar_txt_uri)
+ foo.set_uri("empty", res[3][1].get_uri())
+ sub_uri = res[4][1].get_uri()
+ sub = foo.fake_set_uri("sub", sub_uri)
+
+ blocking_uri = self.make_smallfile(1)
+ foo.set_uri("blockingfile", blocking_uri)
+
+ baz_file = self.makefile(2)
+ sub.set_uri("baz.txt", baz_file)
+
+ self._bad_file_uri = self.makefile(3)
+ del self.files[self._bad_file_uri]
+
+ rodir = res[5][1]
+ self.public_root.set_uri("reedownlee", rodir.get_readonly_uri())
+ rodir.set_uri("nor", baz_file)
+
+ # public/
+ # public/foo/
+ # public/foo/bar.txt
+ # public/foo/blockingfile
+ # public/foo/empty/
+ # public/foo/sub/
+ # public/foo/sub/baz.txt
+ # public/reedownlee/
+ # public/reedownlee/nor
+ self.NEWFILE_CONTENTS = "newfile contents\n"
+ d.addCallback(_then)
+ return d
def makefile(self, number):
n = str(number)
+
assert len(n) == 1
newuri = uri.CHKFileURI(key="K" + n*15,
uri_extension_hash="EH" + n*30,
needed_shares=25,
total_shares=100,
size=123+number).to_string()
- assert newuri not in self.nodes
assert newuri not in self.files
node = MyFileNode(newuri, self.s)
- self.nodes[newuri] = node
contents = "contents of file %s\n" % n
self.files[newuri] = contents
+ self.nodes[newuri] = node
return newuri
def make_smallfile(self, number):
assert len(n) == 1
contents = "small data %s\n" % n
newuri = uri.LiteralFileURI(contents).to_string()
- assert newuri not in self.nodes
assert newuri not in self.files
node = MyFileNode(newuri, self.s)
- self.nodes[newuri] = node
self.files[newuri] = contents
+ self.nodes[newuri] = node
return newuri
- def makedir(self):
- node = MyDirectoryNode(self.nodes, self.files, self.s)
- return node
-
def tearDown(self):
return self.s.stopService()
def failUnlessIsFooJSON(self, res):
data = self.worlds_cheapest_json_decoder(res)
self.failUnless(isinstance(data, list))
- self.failUnlessEqual(data[0], "dirnode")
+ self.failUnlessEqual(data[0], "dirnode", res)
self.failUnless(isinstance(data[1], dict))
self.failUnless("rw_uri" in data[1]) # mutable
self.failUnlessEqual(data[1]["rw_uri"], self._foo_uri)
self.failUnlessEqual(data[1]["ro_uri"], self._foo_readonly_uri)
- kidnames = sorted(data[1]["children"].keys())
+
+ kidnames = sorted(data[1]["children"])
self.failUnlessEqual(kidnames,
["bar.txt", "blockingfile", "empty", "sub"])
kids = data[1]["children"]
def _check(res):
self.failUnless('Welcome To AllMyData' in res)
self.failUnless('Tahoe' in res)
- self.failUnless('View <a href="vdrive/global">the global shared filestore' in res, res)
self.failUnless('personal vdrive not available.' in res)
self.s.basedir = 'web/test_welcome'
self.failUnless(nodeurl.startswith("http://localhost"))
def test_GET_FILEURL(self):
- d = self.GET("/vdrive/global/foo/bar.txt")
+ d = self.GET(self.public_url + "/foo/bar.txt")
d.addCallback(self.failUnlessIsBarDotTxt)
return d
def test_GET_FILEURL_download(self):
- d = self.GET("/vdrive/global/foo/bar.txt?t=download")
+ d = self.GET(self.public_url + "/foo/bar.txt?t=download")
d.addCallback(self.failUnlessIsBarDotTxt)
return d
def test_GET_FILEURL_missing(self):
- d = self.GET("/vdrive/global/foo/missing")
+ d = self.GET(self.public_url + "/foo/missing")
d.addBoth(self.should404, "test_GET_FILEURL_missing")
return d
def test_PUT_NEWFILEURL(self):
- d = self.PUT("/vdrive/global/foo/new.txt", self.NEWFILE_CONTENTS)
+ d = self.PUT(self.public_url + "/foo/new.txt", self.NEWFILE_CONTENTS)
def _check(res):
# TODO: we lose the response code, so we can't check this
#self.failUnlessEqual(responsecode, 201)
- self.failUnless("new.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["new.txt"]
+ self.failUnless(self._foo_node.fake_has_child("new.txt"))
+ new_uri = self._foo_node.fake_get("new.txt").get_uri()
new_contents = self.files[new_uri]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
self.failUnlessEqual(res.strip(), new_uri)
return d
def test_PUT_NEWFILEURL_replace(self):
- d = self.PUT("/vdrive/global/foo/bar.txt", self.NEWFILE_CONTENTS)
+ d = self.PUT(self.public_url + "/foo/bar.txt", self.NEWFILE_CONTENTS)
def _check(res):
# TODO: we lose the response code, so we can't check this
#self.failUnlessEqual(responsecode, 200)
- self.failUnless("bar.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["bar.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("bar.txt"))
+ new_node = self._foo_node.fake_get("bar.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_PUT_NEWFILEURL_no_replace(self):
- d = self.PUT("/vdrive/global/foo/bar.txt?replace=false",
+ d = self.PUT(self.public_url + "/foo/bar.txt?replace=false",
self.NEWFILE_CONTENTS)
d.addBoth(self.shouldFail, error.Error, "PUT_NEWFILEURL_no_replace",
"409 Conflict",
return d
def test_PUT_NEWFILEURL_mkdirs(self):
- d = self.PUT("/vdrive/global/foo/newdir/new.txt", self.NEWFILE_CONTENTS)
+ d = self.PUT(self.public_url + "/foo/newdir/new.txt", self.NEWFILE_CONTENTS)
def _check(res):
- self.failIf("new.txt" in self._foo_node.children)
- self.failUnless("newdir" in self._foo_node.children)
- newdir_uri = self._foo_node.children["newdir"]
- newdir_node = self.nodes[newdir_uri]
- self.failUnless("new.txt" in newdir_node.children)
- new_uri = newdir_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failIf(self._foo_node.fake_has_child("new.txt"))
+ self.failUnless(self._foo_node.fake_has_child("newdir"))
+ newdir_node = self._foo_node.fake_get("newdir")
+ self.failUnless(newdir_node.fake_has_child("new.txt"))
+ new_node = newdir_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_PUT_NEWFILEURL_blocked(self):
- d = self.PUT("/vdrive/global/foo/blockingfile/new.txt",
+ d = self.PUT(self.public_url + "/foo/blockingfile/new.txt",
self.NEWFILE_CONTENTS)
d.addBoth(self.shouldFail, error.Error, "PUT_NEWFILEURL_blocked",
"400 Bad Request",
return d
def test_DELETE_FILEURL(self):
- d = self.DELETE("/vdrive/global/foo/bar.txt")
+ d = self.DELETE(self.public_url + "/foo/bar.txt")
def _check(res):
- self.failIf("bar.txt" in self._foo_node.children)
+ self.failIf(self._foo_node.fake_has_child("bar.txt"))
d.addCallback(_check)
return d
def test_DELETE_FILEURL_missing(self):
- d = self.DELETE("/vdrive/global/foo/missing")
+ d = self.DELETE(self.public_url + "/foo/missing")
d.addBoth(self.should404, "test_DELETE_FILEURL_missing")
return d
def test_DELETE_FILEURL_missing2(self):
- d = self.DELETE("/vdrive/global/missing/missing")
+ d = self.DELETE(self.public_url + "/missing/missing")
d.addBoth(self.should404, "test_DELETE_FILEURL_missing2")
return d
# I can't do "GET /path?json", I have to do "GET /path/t=json"
# instead. This may make it tricky to emulate the S3 interface
# completely.
- d = self.GET("/vdrive/global/foo/bar.txt?t=json")
+ d = self.GET(self.public_url + "/foo/bar.txt?t=json")
d.addCallback(self.failUnlessIsBarJSON)
return d
def test_GET_FILEURL_json_missing(self):
- d = self.GET("/vdrive/global/foo/missing?json")
+ d = self.GET(self.public_url + "/foo/missing?json")
d.addBoth(self.should404, "test_GET_FILEURL_json_missing")
return d
def test_GET_FILEURL_localfile(self):
localfile = os.path.abspath("web/GET_FILEURL_localfile")
- url = "/vdrive/global/foo/bar.txt?t=download&localfile=%s" % localfile
+ url = self.public_url + "/foo/bar.txt?t=download&localfile=%s" % localfile
fileutil.make_dirs("web")
d = self.GET(url)
def _done(res):
def test_GET_FILEURL_localfile_disabled(self):
localfile = os.path.abspath("web/GET_FILEURL_localfile_disabled")
- url = "/vdrive/global/foo/bar.txt?t=download&localfile=%s" % localfile
+ url = self.public_url + "/foo/bar.txt?t=download&localfile=%s" % localfile
fileutil.make_dirs("web")
self.disable_local_access()
d = self.GET(url)
webish.LOCALHOST = "127.0.0.2"
localfile = os.path.abspath("web/GET_FILEURL_localfile_nonlocal")
fileutil.make_dirs("web")
- d = self.GET("/vdrive/global/foo/bar.txt?t=download&localfile=%s"
+ d = self.GET(self.public_url + "/foo/bar.txt?t=download&localfile=%s"
% localfile)
d.addBoth(self.shouldFail, error.Error, "localfile non-local",
"403 Forbidden",
def test_GET_FILEURL_localfile_nonabsolute(self):
localfile = "web/nonabsolute/path"
fileutil.make_dirs("web/nonabsolute")
- d = self.GET("/vdrive/global/foo/bar.txt?t=download&localfile=%s"
+ d = self.GET(self.public_url + "/foo/bar.txt?t=download&localfile=%s"
% localfile)
d.addBoth(self.shouldFail, error.Error, "localfile non-absolute",
"403 Forbidden",
def test_PUT_NEWFILEURL_localfile(self):
localfile = os.path.abspath("web/PUT_NEWFILEURL_localfile")
- url = "/vdrive/global/foo/new.txt?t=upload&localfile=%s" % localfile
+ url = self.public_url + "/foo/new.txt?t=upload&localfile=%s" % localfile
fileutil.make_dirs("web")
f = open(localfile, "wb")
f.write(self.NEWFILE_CONTENTS)
f.close()
d = self.PUT(url, "")
def _check(res):
- self.failUnless("new.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("new.txt"))
+ new_node = self._foo_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_PUT_NEWFILEURL_localfile_disabled(self):
localfile = os.path.abspath("web/PUT_NEWFILEURL_localfile_disabled")
- url = "/vdrive/global/foo/new.txt?t=upload&localfile=%s" % localfile
+ url = self.public_url + "/foo/new.txt?t=upload&localfile=%s" % localfile
fileutil.make_dirs("web")
f = open(localfile, "wb")
f.write(self.NEWFILE_CONTENTS)
f = open(localfile, "wb")
f.write(self.NEWFILE_CONTENTS)
f.close()
- d = self.PUT("/vdrive/global/foo/newdir/new.txt?t=upload&localfile=%s"
+ d = self.PUT(self.public_url + "/foo/newdir/new.txt?t=upload&localfile=%s"
% localfile, "")
def _check(res):
- self.failIf("new.txt" in self._foo_node.children)
- self.failUnless("newdir" in self._foo_node.children)
- newdir_uri = self._foo_node.children["newdir"]
- newdir_node = self.nodes[newdir_uri]
- self.failUnless("new.txt" in newdir_node.children)
- new_uri = newdir_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failIf(self._foo_node.fake_has_child("new.txt"))
+ self.failUnless(self._foo_node.fake_has_child("newdir"))
+ newdir_node = self._foo_node.fake_get("newdir")
+ self.failUnless(newdir_node.fake_has_child("new.txt"))
+ new_node = newdir_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_GET_FILEURL_uri(self):
- d = self.GET("/vdrive/global/foo/bar.txt?t=uri")
+ d = self.GET(self.public_url + "/foo/bar.txt?t=uri")
def _check(res):
self.failUnlessEqual(res, self._bar_txt_uri)
d.addCallback(_check)
d.addCallback(lambda res:
- self.GET("/vdrive/global/foo/bar.txt?t=readonly-uri"))
+ self.GET(self.public_url + "/foo/bar.txt?t=readonly-uri"))
def _check2(res):
# for now, for files, uris and readonly-uris are the same
self.failUnlessEqual(res, self._bar_txt_uri)
return d
def test_GET_FILEURL_uri_missing(self):
- d = self.GET("/vdrive/global/foo/missing?t=uri")
+ d = self.GET(self.public_url + "/foo/missing?t=uri")
d.addBoth(self.should404, "test_GET_FILEURL_uri_missing")
return d
def test_GET_DIRURL(self):
# the addSlash means we get a redirect here
- d = self.GET("/vdrive/global/foo", followRedirect=True)
+ d = self.GET(self.public_url + "/foo", followRedirect=True)
def _check(res):
# the FILE reference points to a URI, but it should end in bar.txt
self.failUnless(re.search(r'<td>'
# look at a directory which is readonly
d.addCallback(lambda res:
- self.GET("/vdrive/global/readonly", followRedirect=True))
+ self.GET(self.public_url + "/reedownlee", followRedirect=True))
def _check2(res):
- self.failUnless("(readonly)" in res)
- self.failIf("Upload a file" in res)
+ self.failUnless("(readonly)" in res, res)
+ self.failIf("Upload a file" in res, res)
d.addCallback(_check2)
# and at a directory that contains a readonly directory
d.addCallback(lambda res:
- self.GET("/vdrive/global", followRedirect=True))
+ self.GET(self.public_url, followRedirect=True))
def _check3(res):
- self.failUnless(re.search(r'<td><a href="readonly">readonly</a>'
+ self.failUnless(re.search(r'<td><a href="reedownlee">reedownlee</a>'
'</td>\s+<td>DIR-RO</td>', res))
d.addCallback(_check3)
return d
def test_GET_DIRURL_json(self):
- d = self.GET("/vdrive/global/foo?t=json")
+ d = self.GET(self.public_url + "/foo?t=json")
d.addCallback(self.failUnlessIsFooJSON)
return d
def test_GET_DIRURL_manifest(self):
- d = self.GET("/vdrive/global/foo?t=manifest", followRedirect=True)
+ d = self.GET(self.public_url + "/foo?t=manifest", followRedirect=True)
def _got(manifest):
self.failUnless("Refresh Capabilities" in manifest)
d.addCallback(_got)
return d
def test_GET_DIRURL_uri(self):
- d = self.GET("/vdrive/global/foo?t=uri")
+ d = self.GET(self.public_url + "/foo?t=uri")
def _check(res):
self.failUnlessEqual(res, self._foo_uri)
d.addCallback(_check)
return d
def test_GET_DIRURL_readonly_uri(self):
- d = self.GET("/vdrive/global/foo?t=readonly-uri")
+ d = self.GET(self.public_url + "/foo?t=readonly-uri")
def _check(res):
self.failUnlessEqual(res, self._foo_readonly_uri)
d.addCallback(_check)
return d
def test_PUT_NEWDIRURL(self):
- d = self.PUT("/vdrive/global/foo/newdir?t=mkdir", "")
+ d = self.PUT(self.public_url + "/foo/newdir?t=mkdir", "")
def _check(res):
- self.failUnless("newdir" in self._foo_node.children)
- newdir_uri = self._foo_node.children["newdir"]
- newdir_node = self.nodes[newdir_uri]
- self.failIf(newdir_node.children)
+ self.failUnless(self._foo_node.fake_has_child("newdir"))
+ newdir_node = self._foo_node.fake_get("newdir")
+ self.failIf(newdir_node.fake_list())
d.addCallback(_check)
return d
def test_PUT_NEWDIRURL_replace(self):
- d = self.PUT("/vdrive/global/foo/sub?t=mkdir", "")
+ d = self.PUT(self.public_url + "/foo/sub?t=mkdir", "")
def _check(res):
- self.failUnless("sub" in self._foo_node.children)
- newdir_uri = self._foo_node.children["sub"]
- newdir_node = self.nodes[newdir_uri]
- self.failIf(newdir_node.children)
+ self.failUnless(self._foo_node.fake_has_child("sub"))
+ newdir_node = self._foo_node.fake_get("sub")
+ self.failIf(newdir_node.fake_list())
d.addCallback(_check)
return d
def test_PUT_NEWDIRURL_no_replace(self):
- d = self.PUT("/vdrive/global/foo/sub?t=mkdir&replace=false", "")
+ d = self.PUT(self.public_url + "/foo/sub?t=mkdir&replace=false", "")
d.addBoth(self.shouldFail, error.Error, "PUT_NEWDIRURL_no_replace",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
def _check(res):
- self.failUnless("sub" in self._foo_node.children)
- newdir_uri = self._foo_node.children["sub"]
- newdir_node = self.nodes[newdir_uri]
- self.failUnlessEqual(newdir_node.children.keys(), ["baz.txt"])
+ self.failUnless(self._foo_node.fake_has_child("sub"))
+ newdir_node = self._foo_node.fake_get("sub")
+ self.failUnlessEqual(newdir_node.fake_list().keys(), ["baz.txt"])
d.addCallback(_check)
return d
def test_PUT_NEWDIRURL_mkdirs(self):
- d = self.PUT("/vdrive/global/foo/subdir/newdir?t=mkdir", "")
+ d = self.PUT(self.public_url + "/foo/subdir/newdir?t=mkdir", "")
def _check(res):
- self.failIf("newdir" in self._foo_node.children)
- self.failUnless("subdir" in self._foo_node.children)
- subdir_node = self.nodes[self._foo_node.children["subdir"]]
- self.failUnless("newdir" in subdir_node.children)
- newdir_node = self.nodes[subdir_node.children["newdir"]]
- self.failIf(newdir_node.children)
+ self.failIf(self._foo_node.fake_has_child("newdir"))
+ self.failUnless(self._foo_node.fake_has_child("subdir"))
+ subdir_node = self._foo_node.fake_get("subdir")
+ self.failUnless(subdir_node.fake_has_child("newdir"))
+ newdir_node = subdir_node.fake_get("newdir")
+ self.failIf(newdir_node.fake_list())
d.addCallback(_check)
return d
def test_DELETE_DIRURL(self):
- d = self.DELETE("/vdrive/global/foo")
+ d = self.DELETE(self.public_url + "/foo")
def _check(res):
- self.failIf("foo" in self.public_root.children)
+ self.failIf(self.public_root.fake_has_child("foo"))
d.addCallback(_check)
return d
def test_DELETE_DIRURL_missing(self):
- d = self.DELETE("/vdrive/global/foo/missing")
+ d = self.DELETE(self.public_url + "/foo/missing")
d.addBoth(self.should404, "test_DELETE_DIRURL_missing")
def _check(res):
- self.failUnless("foo" in self.public_root.children)
+ self.failUnless(self.public_root.fake_has_child("foo"))
d.addCallback(_check)
return d
def test_DELETE_DIRURL_missing2(self):
- d = self.DELETE("/vdrive/global/missing")
+ d = self.DELETE(self.public_url + "/missing")
d.addBoth(self.should404, "test_DELETE_DIRURL_missing2")
return d
def test_walker(self):
out = []
- def _visitor(path, node):
+ def _visitor(path, node, metadata):
out.append((path, node))
return defer.succeed(None)
w = webish.DirnodeWalkerMixin()
('foo', 'empty'),
('foo', 'sub'),
('foo','sub','baz.txt'),
- ('readonly',),
- ('readonly', 'nor'),
+ ('reedownlee',),
+ ('reedownlee', 'nor'),
])
subindex = names.index( ('foo', 'sub') )
bazindex = names.index( ('foo', 'sub', 'baz.txt') )
def test_GET_DIRURL_localdir(self):
localdir = os.path.abspath("web/GET_DIRURL_localdir")
fileutil.make_dirs("web")
- d = self.GET("/vdrive/global/foo?t=download&localdir=%s" % localdir)
+ d = self.GET(self.public_url + "/foo?t=download&localdir=%s" % localdir)
def _check(res):
barfile = os.path.join(localdir, "bar.txt")
self.failUnless(os.path.exists(barfile))
localdir = os.path.abspath("web/GET_DIRURL_localdir_disabled")
fileutil.make_dirs("web")
self.disable_local_access()
- d = self.GET("/vdrive/global/foo?t=download&localdir=%s" % localdir)
+ d = self.GET(self.public_url + "/foo?t=download&localdir=%s" % localdir)
d.addBoth(self.shouldFail, error.Error, "localfile disabled",
"403 Forbidden",
"local file access is disabled")
def test_GET_DIRURL_localdir_nonabsolute(self):
localdir = "web/nonabsolute/dirpath"
fileutil.make_dirs("web/nonabsolute")
- d = self.GET("/vdrive/global/foo?t=download&localdir=%s" % localdir)
+ d = self.GET(self.public_url + "/foo?t=download&localdir=%s" % localdir)
d.addBoth(self.shouldFail, error.Error, "localdir non-absolute",
"403 Forbidden",
"localfile= or localdir= requires an absolute path")
def walk_mynodes(self, node, path=()):
yield path, node
if interfaces.IDirectoryNode.providedBy(node):
- for name in sorted(node.children.keys()):
- child_uri = node.children[name]
- childnode = self.nodes[child_uri]
+ for name in sorted(node.list()):
+ childnode = node.fake_get(name)
childpath = path + (name,)
for xpath,xnode in self.walk_mynodes(childnode, childpath):
yield xpath, xnode
self.touch(localdir, "three/bar.txt")
self.touch(localdir, "zap.zip")
- d = self.PUT("/vdrive/global/newdir?t=upload&localdir=%s"
+ d = self.PUT(self.public_url + "/newdir?t=upload&localdir=%s"
% localdir, "")
def _check(res):
- self.failUnless("newdir" in self.public_root.children)
- newnode = self.nodes[self.public_root.children["newdir"]]
- self.failUnlessEqual(sorted(newnode.children.keys()),
+ self.failUnless(self.public_root.fake_has_child("newdir"))
+ newnode = self.public_root.fake_get("newdir")
+ self.failUnlessEqual(sorted(newnode.fake_list().keys()),
sorted(["one", "two", "three", "zap.zip"]))
- onenode = self.nodes[newnode.children["one"]]
- self.failUnlessEqual(sorted(onenode.children.keys()),
+ onenode = newnode.fake_get("one")
+ self.failUnlessEqual(sorted(onenode.fake_list().keys()),
sorted(["sub"]))
- threenode = self.nodes[newnode.children["three"]]
- self.failUnlessEqual(sorted(threenode.children.keys()),
+ threenode = newnode.fake_get("three")
+ self.failUnlessEqual(sorted(threenode.fake_list().keys()),
sorted(["foo.txt", "bar.txt"]))
- barnode = self.nodes[threenode.children["bar.txt"]]
+ barnode = threenode.fake_get("bar.txt")
contents = self.files[barnode.get_uri()]
self.failUnlessEqual(contents, "contents of three/bar.txt\n")
d.addCallback(_check)
self.touch(localdir, "zap.zip")
self.disable_local_access()
- d = self.PUT("/vdrive/global/newdir?t=upload&localdir=%s"
+ d = self.PUT(self.public_url + "/newdir?t=upload&localdir=%s"
% localdir, "")
d.addBoth(self.shouldFail, error.Error, "localfile disabled",
"403 Forbidden",
self.touch(localdir, "three/bar.txt")
self.touch(localdir, "zap.zip")
- d = self.PUT("/vdrive/global/foo/subdir/newdir?t=upload&localdir=%s"
+ d = self.PUT(self.public_url + "/foo/subdir/newdir?t=upload&localdir=%s"
% localdir,
"")
def _check(res):
- self.failUnless("subdir" in self._foo_node.children)
- subnode = self.nodes[self._foo_node.children["subdir"]]
- self.failUnless("newdir" in subnode.children)
- newnode = self.nodes[subnode.children["newdir"]]
- self.failUnlessEqual(sorted(newnode.children.keys()),
+ self.failUnless(self._foo_node.fake_has_child("subdir"))
+ subnode = self._foo_node.fake_get("subdir")
+ self.failUnless(subnode.fake_has_child("newdir"))
+ newnode = subnode.fake_get("newdir")
+ self.failUnlessEqual(sorted(newnode.fake_list()),
sorted(["one", "two", "three", "zap.zip"]))
- onenode = self.nodes[newnode.children["one"]]
- self.failUnlessEqual(sorted(onenode.children.keys()),
+ onenode = newnode.fake_get("one")
+ self.failUnlessEqual(sorted(onenode.fake_list()),
sorted(["sub"]))
- threenode = self.nodes[newnode.children["three"]]
- self.failUnlessEqual(sorted(threenode.children.keys()),
+ threenode = newnode.fake_get("three")
+ self.failUnlessEqual(sorted(threenode.fake_list()),
sorted(["foo.txt", "bar.txt"]))
- barnode = self.nodes[threenode.children["bar.txt"]]
+ barnode = threenode.fake_get("bar.txt")
contents = self.files[barnode.get_uri()]
self.failUnlessEqual(contents, "contents of three/bar.txt\n")
d.addCallback(_check)
return d
def test_POST_upload(self):
- d = self.POST("/vdrive/global/foo", t="upload",
+ d = self.POST(self.public_url + "/foo", t="upload",
file=("new.txt", self.NEWFILE_CONTENTS))
def _check(res):
- self.failUnless("new.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("new.txt"))
+ new_node = self._foo_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_POST_upload_replace(self):
- d = self.POST("/vdrive/global/foo", t="upload",
+ d = self.POST(self.public_url + "/foo", t="upload",
file=("bar.txt", self.NEWFILE_CONTENTS))
def _check(res):
- self.failUnless("bar.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["bar.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("bar.txt"))
+ new_node = self._foo_node.fake_get("bar.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_POST_upload_no_replace_queryarg(self):
- d = self.POST("/vdrive/global/foo?replace=false", t="upload",
+ d = self.POST(self.public_url + "/foo?replace=false", t="upload",
file=("bar.txt", self.NEWFILE_CONTENTS))
d.addBoth(self.shouldFail, error.Error,
"POST_upload_no_replace_queryarg",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/bar.txt"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/bar.txt"))
d.addCallback(self.failUnlessIsBarDotTxt)
return d
def test_POST_upload_no_replace_field(self):
- d = self.POST("/vdrive/global/foo", t="upload", replace="false",
+ d = self.POST(self.public_url + "/foo", t="upload", replace="false",
file=("bar.txt", self.NEWFILE_CONTENTS))
d.addBoth(self.shouldFail, error.Error, "POST_upload_no_replace_field",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/bar.txt"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/bar.txt"))
d.addCallback(self.failUnlessIsBarDotTxt)
return d
def test_POST_upload_whendone(self):
- d = self.POST("/vdrive/global/foo", t="upload", when_done="/THERE",
+ d = self.POST(self.public_url + "/foo", t="upload", when_done="/THERE",
file=("new.txt", self.NEWFILE_CONTENTS))
d.addBoth(self.shouldRedirect, "/THERE")
def _check(res):
- self.failUnless("new.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("new.txt"))
+ new_node = self._foo_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
d.addCallback(_check)
return d
def test_POST_upload_named(self):
- d = self.POST("/vdrive/global/foo", t="upload",
+ d = self.POST(self.public_url + "/foo", t="upload",
name="new.txt", file=self.NEWFILE_CONTENTS)
def _check(res):
- self.failUnless("new.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("new.txt"))
+ new_node = self._foo_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.NEWFILE_CONTENTS)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_POST_upload_named_badfilename(self):
- d = self.POST("/vdrive/global/foo", t="upload",
+ d = self.POST(self.public_url + "/foo", t="upload",
name="slashes/are/bad.txt", file=self.NEWFILE_CONTENTS)
d.addBoth(self.shouldFail, error.Error,
"test_POST_upload_named_badfilename",
)
def _check(res):
# make sure that nothing was added
- kids = sorted(self._foo_node.children.keys())
+ kids = sorted(self._foo_node.fake_list())
self.failUnlessEqual(sorted(["bar.txt", "blockingfile",
"empty", "sub"]),
kids)
return d
def test_POST_mkdir(self): # return value?
- d = self.POST("/vdrive/global/foo", t="mkdir", name="newdir")
+ d = self.POST(self.public_url + "/foo", t="mkdir", name="newdir")
def _check(res):
- self.failUnless("newdir" in self._foo_node.children)
- newdir_uri = self._foo_node.children["newdir"]
- newdir_node = self.nodes[newdir_uri]
- self.failIf(newdir_node.children)
+ self.failUnless(self._foo_node.fake_has_child("newdir"))
+ newdir_node = self._foo_node.fake_get("newdir")
+ self.failIf(newdir_node.fake_list())
d.addCallback(_check)
return d
def test_POST_mkdir_replace(self): # return value?
- d = self.POST("/vdrive/global/foo", t="mkdir", name="sub")
+ d = self.POST(self.public_url + "/foo", t="mkdir", name="sub")
def _check(res):
- self.failUnless("sub" in self._foo_node.children)
- newdir_uri = self._foo_node.children["sub"]
- newdir_node = self.nodes[newdir_uri]
- self.failIf(newdir_node.children)
+ self.failUnless(self._foo_node.fake_has_child("sub"))
+ newdir_node = self._foo_node.fake_get("sub")
+ self.failIf(newdir_node.fake_list())
d.addCallback(_check)
return d
def test_POST_mkdir_no_replace_queryarg(self): # return value?
- d = self.POST("/vdrive/global/foo?replace=false", t="mkdir", name="sub")
+ d = self.POST(self.public_url + "/foo?replace=false", t="mkdir", name="sub")
d.addBoth(self.shouldFail, error.Error,
"POST_mkdir_no_replace_queryarg",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
def _check(res):
- self.failUnless("sub" in self._foo_node.children)
- newdir_uri = self._foo_node.children["sub"]
- newdir_node = self.nodes[newdir_uri]
- self.failUnlessEqual(newdir_node.children.keys(), ["baz.txt"])
+ self.failUnless(self._foo_node.fake_has_child("sub"))
+ newdir_node = self._foo_node.fake_get("sub")
+ self.failUnlessEqual(newdir_node.fake_list().keys(), ["baz.txt"])
d.addCallback(_check)
return d
def test_POST_mkdir_no_replace_field(self): # return value?
- d = self.POST("/vdrive/global/foo", t="mkdir", name="sub",
+ d = self.POST(self.public_url + "/foo", t="mkdir", name="sub",
replace="false")
d.addBoth(self.shouldFail, error.Error, "POST_mkdir_no_replace_field",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
def _check(res):
- self.failUnless("sub" in self._foo_node.children)
- newdir_uri = self._foo_node.children["sub"]
- newdir_node = self.nodes[newdir_uri]
- self.failUnlessEqual(newdir_node.children.keys(), ["baz.txt"])
+ self.failUnless(self._foo_node.fake_has_child("sub"))
+ newdir_node = self._foo_node.fake_get("sub")
+ self.failUnlessEqual(newdir_node.fake_list().keys(), ["baz.txt"])
d.addCallback(_check)
return d
def test_POST_mkdir_whendone_field(self):
- d = self.POST("/vdrive/global/foo",
+ d = self.POST(self.public_url + "/foo",
t="mkdir", name="newdir", when_done="/THERE")
d.addBoth(self.shouldRedirect, "/THERE")
def _check(res):
- self.failUnless("newdir" in self._foo_node.children)
- newdir_uri = self._foo_node.children["newdir"]
- newdir_node = self.nodes[newdir_uri]
- self.failIf(newdir_node.children)
+ self.failUnless(self._foo_node.fake_has_child("newdir"))
+ newdir_node = self._foo_node.fake_get("newdir")
+ self.failIf(newdir_node.fake_list())
d.addCallback(_check)
return d
def test_POST_mkdir_whendone_queryarg(self):
- d = self.POST("/vdrive/global/foo?when_done=/THERE",
+ d = self.POST(self.public_url + "/foo?when_done=/THERE",
t="mkdir", name="newdir")
d.addBoth(self.shouldRedirect, "/THERE")
def _check(res):
- self.failUnless("newdir" in self._foo_node.children)
- newdir_uri = self._foo_node.children["newdir"]
- newdir_node = self.nodes[newdir_uri]
- self.failIf(newdir_node.children)
+ self.failUnless(self._foo_node.fake_has_child("newdir"))
+ newdir_node = self._foo_node.fake_get("newdir")
+ self.failIf(newdir_node.fake_list())
d.addCallback(_check)
return d
def test_POST_put_uri(self):
newuri = self.makefile(8)
contents = self.files[newuri]
- d = self.POST("/vdrive/global/foo", t="uri", name="new.txt", uri=newuri)
+ d = self.POST(self.public_url + "/foo", t="uri", name="new.txt", uri=newuri)
def _check(res):
- self.failUnless("new.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("new.txt"))
+ new_node = self._foo_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, contents)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_POST_put_uri_replace(self):
newuri = self.makefile(8)
contents = self.files[newuri]
- d = self.POST("/vdrive/global/foo", t="uri", name="bar.txt", uri=newuri)
+ d = self.POST(self.public_url + "/foo", t="uri", name="bar.txt", uri=newuri)
def _check(res):
- self.failUnless("bar.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["bar.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("bar.txt"))
+ new_node = self._foo_node.fake_get("bar.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, contents)
- self.failUnlessEqual(res.strip(), new_uri)
+ self.failUnlessEqual(res.strip(), new_node.get_uri())
d.addCallback(_check)
return d
def test_POST_put_uri_no_replace_queryarg(self):
newuri = self.makefile(8)
contents = self.files[newuri]
- d = self.POST("/vdrive/global/foo?replace=false", t="uri",
+ d = self.POST(self.public_url + "/foo?replace=false", t="uri",
name="bar.txt", uri=newuri)
d.addBoth(self.shouldFail, error.Error,
"POST_put_uri_no_replace_queryarg",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/bar.txt"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/bar.txt"))
d.addCallback(self.failUnlessIsBarDotTxt)
return d
def test_POST_put_uri_no_replace_field(self):
newuri = self.makefile(8)
contents = self.files[newuri]
- d = self.POST("/vdrive/global/foo", t="uri", replace="false",
+ d = self.POST(self.public_url + "/foo", t="uri", replace="false",
name="bar.txt", uri=newuri)
d.addBoth(self.shouldFail, error.Error,
"POST_put_uri_no_replace_field",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/bar.txt"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/bar.txt"))
d.addCallback(self.failUnlessIsBarDotTxt)
return d
def test_POST_delete(self):
- d = self.POST("/vdrive/global/foo", t="delete", name="bar.txt")
+ d = self.POST(self.public_url + "/foo", t="delete", name="bar.txt")
def _check(res):
- self.failIf("bar.txt" in self._foo_node.children)
+ self.failIf(self._foo_node.fake_has_child("bar.txt"))
d.addCallback(_check)
return d
def test_POST_rename_file(self):
- d = self.POST("/vdrive/global/foo", t="rename",
+ d = self.POST(self.public_url + "/foo", t="rename",
from_name="bar.txt", to_name='wibble.txt')
def _check(res):
- self.failIf("bar.txt" in self._foo_node.children)
- self.failUnless("wibble.txt" in self._foo_node.children)
+ self.failIf(self._foo_node.fake_has_child("bar.txt"))
+ self.failUnless(self._foo_node.fake_has_child("wibble.txt"))
d.addCallback(_check)
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/wibble.txt"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/wibble.txt"))
d.addCallback(self.failUnlessIsBarDotTxt)
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/wibble.txt?t=json"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/wibble.txt?t=json"))
d.addCallback(self.failUnlessIsBarJSON)
return d
def test_POST_rename_file_replace(self):
# rename a file and replace a directory with it
- d = self.POST("/vdrive/global/foo", t="rename",
+ d = self.POST(self.public_url + "/foo", t="rename",
from_name="bar.txt", to_name='empty')
def _check(res):
- self.failIf("bar.txt" in self._foo_node.children)
- self.failUnless("empty" in self._foo_node.children)
+ self.failIf(self._foo_node.fake_has_child("bar.txt"))
+ self.failUnless(self._foo_node.fake_has_child("empty"))
d.addCallback(_check)
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/empty"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/empty"))
d.addCallback(self.failUnlessIsBarDotTxt)
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/empty?t=json"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/empty?t=json"))
d.addCallback(self.failUnlessIsBarJSON)
return d
def test_POST_rename_file_no_replace_queryarg(self):
# rename a file and replace a directory with it
- d = self.POST("/vdrive/global/foo?replace=false", t="rename",
+ d = self.POST(self.public_url + "/foo?replace=false", t="rename",
from_name="bar.txt", to_name='empty')
d.addBoth(self.shouldFail, error.Error,
"POST_rename_file_no_replace_queryarg",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/empty?t=json"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/empty?t=json"))
d.addCallback(self.failUnlessIsEmptyJSON)
return d
def test_POST_rename_file_no_replace_field(self):
# rename a file and replace a directory with it
- d = self.POST("/vdrive/global/foo", t="rename", replace="false",
+ d = self.POST(self.public_url + "/foo", t="rename", replace="false",
from_name="bar.txt", to_name='empty')
d.addBoth(self.shouldFail, error.Error,
"POST_rename_file_no_replace_field",
"409 Conflict",
"There was already a child by that name, and you asked me "
"to not replace it")
- d.addCallback(lambda res: self.GET("/vdrive/global/foo/empty?t=json"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo/empty?t=json"))
d.addCallback(self.failUnlessIsEmptyJSON)
return d
def failUnlessIsEmptyJSON(self, res):
data = self.worlds_cheapest_json_decoder(res)
- self.failUnlessEqual(data[0], "dirnode")
+ self.failUnlessEqual(data[0], "dirnode", data)
self.failUnlessEqual(len(data[1]["children"]), 0)
def test_POST_rename_file_slash_fail(self):
- d = self.POST("/vdrive/global/foo", t="rename",
+ d = self.POST(self.public_url + "/foo", t="rename",
from_name="bar.txt", to_name='kirk/spock.txt')
d.addBoth(self.shouldFail, error.Error,
"test_POST_rename_file_slash_fail",
"to_name= may not contain a slash",
)
def _check1(res):
- self.failUnless("bar.txt" in self._foo_node.children)
+ self.failUnless(self._foo_node.fake_has_child("bar.txt"))
d.addCallback(_check1)
- d.addCallback(lambda res: self.POST("/vdrive/global", t="rename",
+ d.addCallback(lambda res: self.POST(self.public_url, t="rename",
from_name="foo/bar.txt", to_name='george.txt'))
d.addBoth(self.shouldFail, error.Error,
"test_POST_rename_file_slash_fail",
"from_name= may not contain a slash",
)
def _check2(res):
- self.failUnless("foo" in self.public_root.children)
- self.failIf("george.txt" in self.public_root.children)
- self.failUnless("bar.txt" in self._foo_node.children)
+ self.failUnless(self.public_root.fake_has_child("foo"))
+ self.failIf(self.public_root.fake_has_child("george.txt"))
+ self.failUnless(self._foo_node.fake_has_child("bar.txt"))
d.addCallback(_check2)
- d.addCallback(lambda res: self.GET("/vdrive/global/foo?t=json"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/foo?t=json"))
d.addCallback(self.failUnlessIsFooJSON)
return d
def test_POST_rename_dir(self):
- d = self.POST("/vdrive/global", t="rename",
+ d = self.POST(self.public_url, t="rename",
from_name="foo", to_name='plunk')
def _check(res):
- self.failIf("foo" in self.public_root.children)
- self.failUnless("plunk" in self.public_root.children)
+ self.failIf(self.public_root.fake_has_child("foo"))
+ self.failUnless(self.public_root.fake_has_child("plunk"))
d.addCallback(_check)
- d.addCallback(lambda res: self.GET("/vdrive/global/plunk?t=json"))
+ d.addCallback(lambda res: self.GET(self.public_url + "/plunk?t=json"))
d.addCallback(self.failUnlessIsFooJSON)
return d
return d
def test_GET_rename_form(self):
- d = self.GET("/vdrive/global/foo?t=rename-form&name=bar.txt",
+ d = self.GET(self.public_url + "/foo?t=rename-form&name=bar.txt",
followRedirect=True) # XXX [ ] todo: figure out why '.../foo' doesn't work
def _check(res):
- self.failUnless(re.search(r'name="when_done" value=".*vdrive/global/foo/', res))
+ self.failUnless(re.search(r'name="when_done" value=".*%s/foo/' % (urllib.quote(self.public_url),), res), (r'name="when_done" value=".*%s/foo/' % (urllib.quote(self.public_url),), res,))
self.failUnless(re.search(r'name="from_name" value="bar\.txt"', res))
d.addCallback(_check)
return d
def test_PUT_NEWFILEURL_uri(self):
new_uri = self.makefile(8)
- d = self.PUT("/vdrive/global/foo/new.txt?t=uri", new_uri)
+ d = self.PUT(self.public_url + "/foo/new.txt?t=uri", new_uri)
def _check(res):
- self.failUnless("new.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["new.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("new.txt"))
+ new_node = self._foo_node.fake_get("new.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.files[new_uri])
self.failUnlessEqual(res.strip(), new_uri)
d.addCallback(_check)
def test_PUT_NEWFILEURL_uri_replace(self):
new_uri = self.makefile(8)
- d = self.PUT("/vdrive/global/foo/bar.txt?t=uri", new_uri)
+ d = self.PUT(self.public_url + "/foo/bar.txt?t=uri", new_uri)
def _check(res):
- self.failUnless("bar.txt" in self._foo_node.children)
- new_uri = self._foo_node.children["bar.txt"]
- new_contents = self.files[new_uri]
+ self.failUnless(self._foo_node.fake_has_child("bar.txt"))
+ new_node = self._foo_node.fake_get("bar.txt")
+ new_contents = self.files[new_node.get_uri()]
self.failUnlessEqual(new_contents, self.files[new_uri])
self.failUnlessEqual(res.strip(), new_uri)
d.addCallback(_check)
def test_PUT_NEWFILEURL_uri_no_replace(self):
new_uri = self.makefile(8)
- d = self.PUT("/vdrive/global/foo/bar.txt?t=uri&replace=false", new_uri)
+ d = self.PUT(self.public_url + "/foo/bar.txt?t=uri&replace=false", new_uri)
d.addBoth(self.shouldFail, error.Error, "PUT_NEWFILEURL_uri_no_replace",
"409 Conflict",
"There was already a child by that name, and you asked me "
d = self.PUT("/uri", file_contents)
def _check(uri):
self.failUnless(uri in self.files)
- self.failUnless(uri in self.nodes)
self.failUnlessEqual(self.files[uri], file_contents)
return self.GET("/uri/%s" % uri.replace("/","!"))
d.addCallback(_check)
d = self.PUT("/uri?t=mkdir", "")
def _check(uri):
self.failUnless(uri in self.nodes)
- self.failUnless(isinstance(self.nodes[uri], MyDirectoryNode))
+ self.failUnless(isinstance(self.nodes[uri], FakeDirectoryNode))
return self.GET("/uri/%s?t=json" % uri.replace("/","!"))
d.addCallback(_check)
d.addCallback(self.failUnlessIsEmptyJSON)
class CHKUploader:
peer_selector_class = Tahoe2PeerSelector
- def __init__(self, client, options={}):
+ def __init__(self, client, options={}, wait_for_numpeers=None):
+ assert wait_for_numpeers is None or isinstance(wait_for_numpeers, int), wait_for_numpeers
self._client = client
+ self._wait_for_numpeers = wait_for_numpeers
self._options = options
def set_params(self, encoding_parameters):
e = encode.Encoder(self._options)
e.set_params(self._encoding_parameters)
d = e.set_encrypted_uploadable(eu)
+ def _wait_for_peers(res):
+ wait_for_numpeers = self._wait_for_numpeers
+ if wait_for_numpeers is None:
+ # wait_for_numpeers = e.get_param("share_counts")[0] # XXX
+ wait_for_numpeers = 1
+
+ d1 = self._client.introducer_client.when_enough_peers(wait_for_numpeers)
+ d1.addCallback(lambda dummy: res)
+ return d1
+ d.addCallback(_wait_for_peers)
d.addCallback(self.locate_all_shareholders)
d.addCallback(self.set_shareholders, e)
d.addCallback(lambda res: e.start())
class LiteralUploader:
- def __init__(self, client, options={}):
+ def __init__(self, client, wait_for_numpeers, options={}):
self._client = client
self._options = options
# 'total' is the total number of shares created by encoding. If everybody
# has room then this is is how many we will upload.
- def upload(self, uploadable, options={}):
+ def upload(self, uploadable, options={}, wait_for_numpeers=None):
+ assert wait_for_numpeers is None or isinstance(wait_for_numpeers, int), wait_for_numpeers
# this returns the URI
assert self.parent
assert self.running
uploader_class = self.uploader_class
if size <= self.URI_LIT_SIZE_THRESHOLD:
uploader_class = LiteralUploader
- uploader = uploader_class(self.parent, options)
+ uploader = uploader_class(self.parent, options, wait_for_numpeers)
uploader.set_params(self.parent.get_encoding_parameters()
or self.DEFAULT_ENCODING_PARAMETERS)
return uploader.start(uploadable)
from twisted.python.components import registerAdapter
from allmydata.util import idlib, hashutil
from allmydata.interfaces import IURI, IDirnodeURI, IFileURI, IVerifierURI, \
- IMutableFileURI, INewDirectoryURI
+ IMutableFileURI, INewDirectoryURI, IReadonlyNewDirectoryURI
# the URI shall be an ascii representation of the file. It shall contain
# enough information to retrieve and validate the contents. It shall be
return "URI:SSK:%s:%s" % (idlib.b2a(self.writekey),
idlib.b2a(self.fingerprint))
+ def __repr__(self):
+ return "<%s %s>" % (self.__class__.__name__, self.abbrev())
+
+ def abbrev(self):
+ return idlib.b2a(self.writekey[:5])
+
def is_readonly(self):
return False
def is_mutable(self):
return "URI:SSK-RO:%s:%s" % (idlib.b2a(self.readkey),
idlib.b2a(self.fingerprint))
+ def __repr__(self):
+ return "<%s %s>" % (self.__class__.__name__, self.abbrev())
+
+ def abbrev(self):
+ return idlib.b2a(self.readkey[:5])
+
def is_readonly(self):
return True
def is_mutable(self):
return "URI:SSK-Verifier:%s:%s" % (idlib.b2a(self.storage_index),
idlib.b2a(self.fingerprint))
-class NewDirectoryURI(_BaseURI):
- implements(IURI, IDirnodeURI, INewDirectoryURI)
+class _NewDirectoryBaseURI(_BaseURI):
+ implements(IURI, IDirnodeURI)
+ def __init__(self, filenode_uri=None):
+ self._filenode_uri = filenode_uri
+
+ def __repr__(self):
+ return "<%s %s>" % (self.__class__.__name__, self.abbrev())
+
+ def abbrev(self):
+ return self._filenode_uri.to_string().split(':')[2][:5]
+ def get_filenode_uri(self):
+ return self._filenode_uri
+
+ def is_mutable(self):
+ return True
+
+ def get_verifier(self):
+ return NewDirectoryURIVerifier(self._filenode_uri.get_verifier())
+
+class NewDirectoryURI(_NewDirectoryBaseURI):
+ implements(INewDirectoryURI)
def __init__(self, filenode_uri=None):
if filenode_uri:
assert not filenode_uri.is_readonly()
- self._filenode_uri = filenode_uri
+ _NewDirectoryBaseURI.__init__(self, filenode_uri)
def init_from_string(self, uri):
assert uri.startswith("URI:DIR2:")
(header_uri, header_ssk, bits) = fn_u.split(":", 2)
return "URI:DIR2:" + bits
- def get_filenode_uri(self):
- return self._filenode_uri
-
def is_readonly(self):
return False
- def is_mutable(self):
- return True
+
def get_readonly(self):
return ReadonlyNewDirectoryURI(self._filenode_uri.get_readonly())
- def get_verifier(self):
- return NewDirectoryURIVerifier(self._filenode_uri.get_verifier())
-
-class ReadonlyNewDirectoryURI(_BaseURI):
- implements(IURI, IDirnodeURI)
+class ReadonlyNewDirectoryURI(_NewDirectoryBaseURI):
+ implements(IReadonlyNewDirectoryURI)
def __init__(self, filenode_uri=None):
if filenode_uri:
assert filenode_uri.is_readonly()
- self._filenode_uri = filenode_uri
+ _NewDirectoryBaseURI.__init__(self, filenode_uri)
def init_from_string(self, uri):
assert uri.startswith("URI:DIR2-RO:")
(header_uri, header_ssk, bits) = fn_u.split(":", 2)
return "URI:DIR2-RO:" + bits
- def get_filenode_uri(self):
- return self._filenode_uri
-
def is_readonly(self):
return True
- def is_mutable(self):
- return True
+
def get_readonly(self):
return self
- def get_verifier(self):
- return NewDirectoryURIVerifier(self._filenode_uri.get_verifier())
class NewDirectoryURIVerifier(_BaseURI):
implements(IVerifierURI)
registerAdapter(from_string_dirnode, str, IDirnodeURI)
+def is_string_newdirnode_rw(s):
+ if not s.startswith("URI:DIR2:"):
+ return False
+ try:
+ (header_uri, header_dir2, writekey_s, fingerprint_s) = s.split(":", 2)
+ except ValueError:
+ return False
+ return idlib.could_be_base32_encoded(writekey_s) and idlib.could_be_base32_encoded(fingerprint_s)
+
def from_string_filenode(s):
u = from_string(s)
assert IFileURI.providedBy(u)
from pycryptopp.hash.sha256 import SHA256
import os
+class IntegrityCheckError(Exception):
+ pass
+
def netstring(s):
+ assert isinstance(s, str), s
return "%d:%s," % (len(s), s,)
def tagged_hash(tag, val):
+++ /dev/null
-
-import os
-from zope.interface import implements
-from twisted.python import log
-from twisted.application import service
-from twisted.internet import defer
-from allmydata.interfaces import IVirtualDrive, IDirnodeURI, IURI
-from allmydata.util import observer
-from allmydata import dirnode
-from allmydata.dirnode2 import INewDirectoryURI
-
-class NoGlobalVirtualDriveError(Exception):
- pass
-class NoPrivateVirtualDriveError(Exception):
- pass
-
-class VirtualDrive(service.MultiService):
- implements(IVirtualDrive)
- name = "vdrive"
-
- GLOBAL_VDRIVE_FURL_FILE = "vdrive.furl"
-
- GLOBAL_VDRIVE_URI_FILE = "global_root.uri"
- MY_VDRIVE_URI_FILE = "my_vdrive.uri"
-
- def __init__(self):
- service.MultiService.__init__(self)
- self._global_uri = None
- self._private_uri = None
- self._private_root_observer = observer.OneShotObserverList()
-
- def log(self, msg):
- self.parent.log(msg)
-
- def startService(self):
- service.MultiService.startService(self)
- basedir = self.parent.basedir
- tub = self.parent.tub
-
- global_uri_file = os.path.join(basedir, self.GLOBAL_VDRIVE_URI_FILE)
- if os.path.exists(global_uri_file):
- f = open(global_uri_file)
- self._global_uri = f.read().strip()
- f.close()
- self.log("using global vdrive uri %s" % self._global_uri)
-
- private_uri_file = os.path.join(basedir, self.MY_VDRIVE_URI_FILE)
- if os.path.exists(private_uri_file):
- f = open(private_uri_file)
- self._private_uri = f.read().strip()
- f.close()
- self.log("using private vdrive uri %s" % self._private_uri)
- self._private_root_observer.fire(self._private_uri)
-
- furl_file = os.path.join(basedir, self.GLOBAL_VDRIVE_FURL_FILE)
- if os.path.exists(furl_file):
- f = open(furl_file, "r")
- global_vdrive_furl = f.read().strip()
- f.close()
- else:
- self.log("no %s, cannot access global or private dirnodes"
- % furl_file)
- return
-
- self.global_vdrive_furl = global_vdrive_furl
- tub.connectTo(global_vdrive_furl,
- self._got_vdrive_server, global_vdrive_furl)
-
- if not self._global_uri:
- self.log("will fetch global_uri when we attach to the "
- "vdrive server")
-
- if not self._private_uri:
- self.log("will create private_uri when we attach to the "
- "vdrive server")
-
-
- def _got_vdrive_server(self, vdrive_server, global_vdrive_furl):
- self.log("connected to vdrive server")
- basedir = self.parent.basedir
- global_uri_file = os.path.join(basedir, self.GLOBAL_VDRIVE_URI_FILE)
- private_uri_file = os.path.join(basedir, self.MY_VDRIVE_URI_FILE)
-
- d = vdrive_server.callRemote("get_public_root_uri")
- def _got_global_uri(global_uri):
- self.log("got global_uri: %s" % global_uri)
- self._global_uri = global_uri
- f = open(global_uri_file, "w")
- f.write(self._global_uri + "\n")
- f.close()
- d.addCallback(_got_global_uri)
-
- if not self._private_uri:
- d.addCallback(lambda res:
- dirnode.create_directory(self.parent,
- global_vdrive_furl))
- def _got_directory(dirnode):
- self.log("creating private uri")
- self._private_uri = dirnode.get_uri()
- f = open(private_uri_file, "w")
- f.write(self._private_uri + "\n")
- f.close()
- self._private_root_observer.fire(self._private_uri)
- d.addCallback(_got_directory)
-
- def _oops(f):
- self.log("error getting URIs from vdrive server")
- log.err(f)
- d.addErrback(_oops)
-
-
- def have_public_root(self):
- return bool(self._global_uri)
- def get_public_root(self):
- if not self._global_uri:
- return defer.fail(NoGlobalVirtualDriveError())
- return self.get_node(self._global_uri)
-
- def when_private_root_available(self):
- """Return a Deferred that will fire with the URI of the private
- vdrive root, when it is available.
-
- This might be right away if the private vdrive was already present.
- The first time the node is started, this will take a bit longer.
- """
- return self._private_root_observer.when_fired()
-
- def have_private_root(self):
- return bool(self._private_uri)
- def get_private_root(self):
- if not self._private_uri:
- return defer.fail(NoPrivateVirtualDriveError())
- return self.get_node(self._private_uri)
-
- def get_node(self, node_uri):
- node_uri = IURI(node_uri)
- if (IDirnodeURI.providedBy(node_uri)
- and not INewDirectoryURI.providedBy(node_uri)):
- return dirnode.create_directory_node(self.parent, node_uri)
- else:
- return defer.succeed(self.parent.create_node_from_uri(node_uri))
-
-
- def get_node_at_path(self, path, root=None):
- if not isinstance(path, (list, tuple)):
- assert isinstance(path, (str, unicode))
- if path[0] == "/":
- path = path[1:]
- path = path.split("/")
- assert isinstance(path, (list, tuple))
-
- if root is None:
- if path and path[0] == "~":
- d = self.get_private_root()
- d.addCallback(lambda node:
- self.get_node_at_path(path[1:], node))
- return d
- d = self.get_public_root()
- d.addCallback(lambda node: self.get_node_at_path(path, node))
- return d
-
- if path:
- assert path[0] != ""
- d = root.get(path[0])
- d.addCallback(lambda node: self.get_node_at_path(path[1:], node))
- return d
-
- return root
-
- def create_directory(self):
- # return a new+empty+unlinked dirnode
- assert self.global_vdrive_furl
- d = dirnode.create_directory(self.parent, self.global_vdrive_furl)
- return d
<td><n:slot name="size"/></td>
<td><n:slot name="data"/></td>
<td><n:slot name="delete"/></td>
-
<td><div n:render="overwrite"/></td>
+ <td><div n:render="overwrite"/></td>
<td><n:slot name="rename"/></td>
<td><n:slot name="check"/></td>
<h1>Welcome To Your AllMyData "Tahoe" Node!</h1>
-<div>View <a href="%(base_url)s/vdrive/global">the global shared filestore</a>.</div>
-
-<div>View <a href="%(base_url)s/uri/%(private_uri)s">your personal private non-shared filestore</a>.</div>
+<div>%(link_to_private_uri)s</div>
<div>View <a href="%(base_url)s/">this node's status page</a>.</div>
<h1>Welcome To AllMyData "Tahoe"!</h1>
-<div n:render="global_vdrive" />
<div n:render="private_vdrive" />
<div>Please visit the <a href="http://allmydata.org">Tahoe home page</a> for
<div>My version: <span n:render="string" n:data="version" /></div>
<div>Introducer: <span n:render="string" n:data="introducer_furl" /></div>
<div>Connected to introducer?: <span n:render="string" n:data="connected_to_introducer" /></div>
-<div>Connected to vdrive?: <span n:render="string" n:data="connected_to_vdrive" /></div>
<div>Known+Connected Peers: <span n:render="string" n:data="num_peers" /></div>
<div>
header.append("/")
header.append("'")
- if not self._dirnode.is_mutable():
+ if self._dirnode.is_readonly():
header.append(" (readonly)")
header.append(":")
return ctx.tag[header]
return d
def render_row(self, ctx, data):
- name, target = data
+ name, (target, metadata) = data
- if self._dirnode.is_mutable():
+ if self._dirnode.is_readonly():
+ delete = "-"
+ rename = "-"
+ else:
# this creates a button which will cause our child__delete method
# to be invoked, which deletes the file and then redirects the
# browser back to this directory
T.input(type='hidden', name='when_done', value=url.here),
T.input(type='submit', value='rename', name="rename"),
]
- else:
- delete = "-"
- rename = "-"
ctx.fillSlots("delete", delete)
ctx.fillSlots("rename", rename)
uri_link += '?%s' % (urllib.urlencode({'filename': name}),)
# to prevent javascript in displayed .html files from stealing a
- # secret vdrive URI from the URL, send the browser to a URI-based
- # page that doesn't know about the vdrive at all
+ # secret directory URI from the URL, send the browser to a URI-based
+ # page that doesn't know about the directory at all
#dlurl = urllib.quote(name)
dlurl = uri_link
uri_link += '?%s' % (urllib.urlencode({'filename': name}),)
# to prevent javascript in displayed .html files from stealing a
- # secret vdrive URI from the URL, send the browser to a URI-based
- # page that doesn't know about the vdrive at all
+ # secret directory URI from the URL, send the browser to a URI-based
+ # page that doesn't know about the directory at all
#dlurl = urllib.quote(name)
dlurl = uri_link
subdir_url = urllib.quote(name)
ctx.fillSlots("filename",
T.a(href=subdir_url)[html.escape(name)])
- if target.is_mutable():
- dirtype = "DIR"
- else:
+ if target.is_readonly():
dirtype = "DIR-RO"
+ else:
+ dirtype = "DIR"
ctx.fillSlots("type", dirtype)
ctx.fillSlots("size", "-")
text_plain_tag = None
return ctx.tag
def render_forms(self, ctx, data):
- if not self._dirnode.is_mutable():
- return T.div["No upload forms: directory is immutable"]
+ if self._dirnode.is_readonly():
+ return T.div["No upload forms: directory is read-only"]
mkdir = T.form(action=".", method="post",
enctype="multipart/form-data")[
T.fieldset[
def _handle_items(self, items, visitor, rootpath):
if not items:
return
- childname, childnode = items[0]
+ childname, (childnode, metadata) = items[0]
childpath = rootpath + (childname,)
- d = defer.maybeDeferred(visitor, childpath, childnode)
+ d = defer.maybeDeferred(visitor, childpath, childnode, metadata)
if IDirectoryNode.providedBy(childnode):
d.addCallback(lambda res: self.walk(childnode, visitor, childpath))
d.addCallback(lambda res:
self._dirnode = dirnode
self._localdir = localdir
- def _handle(self, path, node):
+ def _handle(self, path, node, metadata):
localfile = os.path.join(self._localdir, os.sep.join(path))
if IDirectoryNode.providedBy(node):
fileutil.make_dirs(localfile)
d = node.list()
def _got(children):
kids = {}
- for name, childnode in children.iteritems():
+ for name, (childnode, metadata) in children.iteritems():
if IFileNode.providedBy(childnode):
kiduri = childnode.get_uri()
kiddata = ("filenode",
'size': childnode.get_size(),
})
else:
- assert IDirectoryNode.providedBy(childnode)
+ assert IDirectoryNode.providedBy(childnode), (childnode, children,)
kiddata = ("dirnode",
- {'ro_uri': childnode.get_immutable_uri(),
+ {'ro_uri': childnode.get_readonly_uri(),
})
- if childnode.is_mutable():
+ if not childnode.is_readonly():
kiddata[1]['rw_uri'] = childnode.get_uri()
kids[name] = kiddata
contents = { 'children': kids,
- 'ro_uri': node.get_immutable_uri(),
+ 'ro_uri': node.get_readonly_uri(),
}
- if node.is_mutable():
+ if not node.is_readonly():
contents['rw_uri'] = node.get_uri()
data = ("dirnode", contents)
return simplejson.dumps(data, indent=1)
class DirectoryReadonlyURI(DirectoryJSONMetadata):
def renderNode(self, node):
- return node.get_immutable_uri()
+ return node.get_readonly_uri()
class RenameForm(rend.Page):
addSlash = True
"/".join(self._dirpath),
"':", ]
- if not self._dirnode.is_mutable():
+ if self._dirnode.is_readonly():
header.append(" (readonly)")
return ctx.tag[header]
method = req.method
path = segments
- # when we're pointing at a directory (like /vdrive/public/my_pix),
- # Directory.addSlash causes a redirect to /vdrive/public/my_pix/,
+ # when we're pointing at a directory (like /uri/$DIR_URI/my_pix),
+ # Directory.addSlash causes a redirect to /uri/$DIR_URI/my_pix/,
# which appears here as ['my_pix', '']. This is supposed to hit the
# same Directory as ['my_pix'].
if path and path[-1] == '':
return d
if t == "mkdir":
- # "PUT /uri?t=mkdir", to create an unlinked directory. We use the
- # public vdriveserver to create the dirnode.
- vdrive = IClient(ctx).getServiceNamed("vdrive")
- d = vdrive.create_directory()
- # TODO: switch to new-style dirnodes and replace this with:
- #d = IClient(ctx).create_empty_dirnode()
+ # "PUT /uri?t=mkdir", to create an unlinked directory.
+ d = IClient(ctx).create_empty_dirnode()
d.addCallback(lambda dirnode: dirnode.get_uri())
return d
def locateChild(self, ctx, segments):
client = IClient(ctx)
req = inevow.IRequest(ctx)
- vdrive = client.getServiceNamed("vdrive")
- if segments[0] == "vdrive":
- if len(segments) < 2:
- return rend.NotFound
- if segments[1] == "global":
- d = vdrive.get_public_root()
- name = "public vdrive"
- else:
- return rend.NotFound
- d.addCallback(lambda dirnode: VDrive(dirnode, name))
- d.addCallback(lambda vd: vd.locateChild(ctx, segments[2:]))
- return d
- elif segments[0] == "uri":
+ if segments[0] == "uri":
if len(segments) == 1 or segments[1] == '':
if "uri" in req.args:
uri = req.args["uri"][0].replace("/", "!")
if len(segments) < 2:
return rend.NotFound
uri = segments[1].replace("!", "/")
- d = vdrive.get_node(uri)
+ d = defer.maybeDeferred(client.create_node_from_uri, uri)
d.addCallback(lambda node: VDrive(node, "from-uri"))
d.addCallback(lambda vd: vd.locateChild(ctx, segments[2:]))
def _trap_KeyError(f):
d.addErrback(_trap_KeyError)
return d
elif segments[0] == "xmlrpc":
- pass # TODO
+ raise NotYetImplementedError()
return rend.Page.locateChild(self, ctx, segments)
child_webform_css = webform.defaultCSS
if IClient(ctx).connected_to_introducer():
return "yes"
return "no"
- def data_connected_to_vdrive(self, ctx, data):
- if IClient(ctx).getServiceNamed("vdrive").have_public_root():
- return "yes"
- return "no"
def data_num_peers(self, ctx, data):
#client = inevow.ISite(ctx)._client
client = IClient(ctx)
ctx.fillSlots("peerid", nodeid_a)
return ctx.tag
- def render_global_vdrive(self, ctx, data):
- if IClient(ctx).getServiceNamed("vdrive").have_public_root():
- return T.p["View ",
- T.a(href="vdrive/global")["the global shared filestore"],
- "."
- ]
- return T.p["vdrive.furl not specified (or vdrive server not "
- "responding), no vdrive available."]
-
def render_private_vdrive(self, ctx, data):
basedir = IClient(ctx).basedir
start_html = os.path.abspath(os.path.join(basedir, "start.html"))
s = strports.service(webport, site)
s.setServiceParent(self)
self.listener = s # stash it so the tests can query for the portnum
+ self._started = defer.Deferred()
def allow_local_access(self, enable=True):
self.allow_local.local_access = enable
# I thought you could do the same with an existing interface, but
# apparently 'ISite' does not exist
#self.site._client = self.parent
+ self._started.callback(None)
def create_start_html(self, private_uri, startfile, nodeurl_file):
+ """
+ Returns a deferred that eventually fires once the start.html page has
+ been created.
+ """
+ self._started.addCallback(self._create_start_html, private_uri, startfile, nodeurl_file)
+ return self._started
+
+ def _create_start_html(self, dummy, private_uri, startfile, nodeurl_file):
f = open(startfile, "w")
os.chmod(startfile, 0600)
template = open(util.sibpath(__file__, "web/start.html"), "r").read()
base_url = "UNKNOWN" # this will break the href
# TODO: emit a start.html that explains that we don't know
# how to create a suitable URL
- fields = {"private_uri": private_uri.replace("/","!"),
- "base_url": base_url,
- }
+ if private_uri:
+ private_uri = private_uri.replace("/","!")
+ link_to_private_uri = "View <a href=\"%s/uri/%s\">your personal private non-shared filestore</a>." % (base_url, private_uri)
+ fields = {"link_to_private_uri": link_to_private_uri,
+ "base_url": base_url,
+ }
+ else:
+ fields = {"link_to_private_uri": "",
+ "base_url": base_url,
+ }
f.write(template % fields)
f.close()
# this file is world-readable
f.write(base_url + "\n")
f.close()
-
projects / tahoe-lafs / tahoe-lafs.git / commitdiff