From: Brian Warner Date: Wed, 16 May 2007 00:33:52 +0000 (-0700) Subject: update to foolscap-0.1.4 X-Git-Tag: allmydata-tahoe-0.3.0~81 X-Git-Url: https://git.rkrishnan.org/components/com_hotproperty/frontends/%3C?a=commitdiff_plain;h=267a068fe4d45bffe5f38e93498b80066e837e8c;p=tahoe-lafs%2Ftahoe-lafs.git update to foolscap-0.1.4 --- diff --git a/src/foolscap/ChangeLog b/src/foolscap/ChangeLog index 06af6258..bade5f83 100644 --- a/src/foolscap/ChangeLog +++ b/src/foolscap/ChangeLog @@ -1,3 +1,112 @@ +2007-05-14 Brian Warner + + * foolscap/__init__.py: release Foolscap-0.1.4 + * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same, also + remove a bunch of old between-release version numbers + +2007-05-14 Brian Warner + + * NEWS: update for the upcoming release + + * doc/using-foolscap.xhtml: rename from doc/using-pb.xhtml + + * doc/using-pb.xhtml: replace all uses of 'PB URL' with 'FURL' + + * foolscap/pb.py (Tub.getReference): if getReference() is called + before Tub.startService(), queue the request until startup. + (Tub.connectTo): same for connectTo(). + (Tub.startService): launch pending getReference() and connectTo() + requests. There are all fired with eventual-sends. + * foolscap/reconnector.py (Reconnector): don't automatically start + the Reconnector in __init__, rather wait for the Tub to start it. + * foolscap/test/test_tub.py (QueuedStartup): test it + * doc/using-pb.xhtml: update docs to match + + * foolscap/test/test_call.py (TestCall.testCall1): replace an + arbitrary delay with a polling loop, to make the test more + reliable under load + + * foolscap/referenceable.py (SturdyRef.asLiveRef): remove a method + that was never used, didn't work, and is of dubious utility + anyways. + (_AsLiveRef): remove this too + + * misc/testutils/figleaf.py (CodeTracer.start): remove leftover + debug logging + + * foolscap/remoteinterface.py (RemoteInterfaceConstraint): accept + gifts too: allow sending of RemoteReferences on the outbound side, + and accept their-reference sequences on the inbound side. + * foolscap/test/test_gifts.py (Gifts.test_constraint): test it + * foolscap/test/test_schema.py (Interfaces.test_remotereference): + update test, since now we allow RemoteReferences to be sent on the + outbound side + + * foolscap/remoteinterface.py (getRemoteInterface): improve the + error message reported when a Referenceable class implements + multiple RemoteInterfaces + + * foolscap/remoteinterface.py (RemoteMethodSchema.initFromMethod): + properly handle methods like 'def foo(nodefault)' that are missing + *all* default values. Previously this resulted in an unhelpful + exception (since typeList==None), now it gives a sensible + InvalidRemoteInterface exception. + * foolscap/test/test_schema.py (Arguments.test_bad_arguments): + test it + +2007-05-11 Brian Warner + + * foolscap/slicers/set.py (FrozenSetSlicer): finally acknowledge + our dependence on python2.4 or newer, by using the built-in 'set' + and 'frozenset' types by default. We'll serialize the old sets.Set + and sets.ImmutableSet too, but they'll emerge as a set/frozenset. + This will cause problems for code that was written to be + compatible with python2.3 (by using sets.Set) and wasn't changed + when moved to 2.4, if it tries to mingle sets.Set with the data + coming out of Foolscap. Unfortunate, but doing it this way + preserves both sanity and good behavior for modern 2.4-or-later + apps. + (SetUnslicer): fix handling of children that were unreferenceable + during construction, fix handling of children that are not ready + for use (i.e. gifts). + (FrozenSetUnslicer): base this off of TupleUnslicer, since + previously the cycle-handling logic was completely broken. I'm not + entirely sure this is necessary, since I think the contents of + sets must be transitively immutable (or at least transitively + hashable), but it good to review and clean it up anyways. + * foolscap/slicers/allslicers.py: match name change + + * foolscap/slicers/tuple.py (TupleUnslicer.receiveClose): fix + handling of unready children (i.e. gifts), previously gifts inside + containers were completely broken. + * foolscap/slicers/list.py (ListUnslicer.receiveClose): same + * foolscap/slicers/dict.py (DictUnslicer.receiveClose): same + + * foolscap/call.py: add debug log messages (disabled) + + * foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose): + gifts must declare themselves 'unready' until the RemoteReference + resolves, since we might be inside a container of some sort. + Without this fix, methods would be invoked too early, before the + RemoteReference was really available. + + * foolscap/test/test_banana.py (ThereAndBackAgain.test_set): match + new set/sets.Set behavior + (ThereAndBackAgain.test_cycles_1): test some of the cycles + (ThereAndBackAgain.test_cycles_3): add (disabled) test for + checking cycles that involve sets. I think these tests are + non-sensical, since sets can't really participate in the sorts of + cycles we worry about, but I left the (disabled) test code in + place in case it becomes useful again. + + * foolscap/test/test_gifts.py (Gifts.testContainers): validate + that gifts can appear in all sorts of containers successfully. + +2007-05-11 Brian Warner + + * foolscap/__init__.py: bump revision to 0.1.3+ while between releases + * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same + 2007-05-02 Brian Warner * foolscap/__init__.py: release Foolscap-0.1.3 diff --git a/src/foolscap/NEWS b/src/foolscap/NEWS index 8ef246fb..f45ecb20 100644 --- a/src/foolscap/NEWS +++ b/src/foolscap/NEWS @@ -1,5 +1,69 @@ User visible changes in Foolscap (aka newpb/pb2). -*- outline -*- +* Release 0.1.4 (14 May 2007) + +** Compatibility + +This release is fully compatible with 0.1.3 . + +** getReference/connectTo can be called before Tub.startService() + +The Tub.startService changes that were suggested in the 0.1.3 release notes +have been implemented. Calling getReference() or connectTo() before the Tub +has been started is now allowed, however no action will take place until the +Tub is running. Don't forget to start the Tub, or you'll be left wondering +why your Deferred or callback is never fired. (A log message is emitted when +these calls are made before the Tub is started, in the hopes of helping +developers find this mistake faster). + +** constraint improvements + +The RIFoo -style constraint now accepts gifts (third-party references). This +also means that using RIFoo on the outbound side will accept either a +Referenceable that implements the given RemoteInterface or a RemoteReference +that points to a Referenceable that implements the given RemoteInterface. +There is a situation (sending a RemoteReference back to its owner) that will +pass the outbound constraint but be rejected by the inbound constraint on the +other end. It remains to be seen how this will be fixed. + +** foolscap now deserializes into python2.4-native 'set' and 'frozenset' types + +Since Foolscap is dependent upon python2.4 or newer anyways, it now +unconditionally creates built-in 'set' and 'frozenset' instances when +deserializing 'set'/'immutable-set' banana sequences. The pre-python2.4 +'sets' module has non-built-in set classes named sets.Set and +sets.ImmutableSet, and these are serialized just like the built-in forms. + +Unfortunately this means that Set and ImmutableSet will not survive a +round-trip: they'll be turned into set and frozenset, respectively. Worse +yet, 'set' and 'sets.Set' are not entirely compatible. This may cause a +problem for older applications that were written to be compatible with both +python-2.3 and python-2.4 (by using sets.Set/sets.ImmutableSet), for which +the compatibility code is still in place (i.e. they are not using +set/frozenset). These applications may experience problems when set objects +that traverse the wire via Foolscap are brought into close proximity with set +objects that remained local. This is unfortunate, but it's the cleanest way +to support modern applications that use the native types exclusively. + +** bug fixes + +Gifts inside containers (lists, tuples, dicts, sets) were broken: the target +method was frequently invoked before the gift had properly resolved into a +RemoteReference. Constraints involving gifts inside containers were broken +too. The constraints may be too loose right now, but I don't think they +should cause false negatives. + +The unused SturdyRef.asLiveRef method was removed, since it didn't work +anyways. + +** terminology shift: FURL + +The preferred name for the sort of URL that you get back from +registerReference (and hand to getReference or connectTo) has changed from +"PB URL" to "FURL" (short for Foolscap URL). They still start with 'pb:', +however. Documentation is slowly being changed to use this term. + + * Release 0.1.3 (02 May 2007) ** Incompatibility Warning diff --git a/src/foolscap/doc/using-foolscap.xhtml b/src/foolscap/doc/using-foolscap.xhtml new file mode 100644 index 00000000..9ddc88ce --- /dev/null +++ b/src/foolscap/doc/using-foolscap.xhtml @@ -0,0 +1,978 @@ + + +Introduction to Foolscap + + + + +

Introduction to Foolscap

+ +

Introduction

+ +

Suppose you find yourself in control of both ends of the wire: you have +two programs that need to talk to each other, and you get to use any protocol +you want. If you can think of your problem in terms of objects that need to +make method calls on each other, then chances are good that you can use the +Foolscap protocol rather than trying to shoehorn your needs into something +like HTTP, or implementing yet another RPC mechanism.

+ +

Foolscap is based upon a few central concepts:

+ +
    + +
  • serialization: taking fairly arbitrary objects and types, + turning them into a chunk of bytes, sending them over a wire, then + reconstituting them on the other end. By keeping careful track of object + ids, the serialized objects can contain references to other objects and the + remote copy will still be useful.
    • + +
  • remote method calls: doing something to a local proxy and + causing a method to get run on a distant object. The local proxy is called + a RemoteReference, + and you do something by running its .callRemote method. + The distant object is called a Referenceable, and it has methods like + remote_foo that will be invoked.
    • + +
+ +

Foolscap is the descendant of Perspective Broker (which lived in the +twisted.spread package). For many years it was known as "newpb". A lot of the +API still has the name "PB" in it somewhere. These will probably go away +sooner or later.

+ +

A "foolscap" is a size of paper, probably measuring 17 by 13.5 inches. A +twisted foolscap of paper makes a good fool's cap. Also, "cap" makes me think +of capabilities, and Foolscap is a protocol to implement a distributed +object-capabilities model in python.

+ + +

Getting Started

+ +

Any Foolscap application has at least two sides: one which hosts a +remotely-callable object, and another which calls (remotely) the methods of +that object. We'll start with a simple example that demostrates both ends. +Later, we'll add more features like RemoteInterface declarations, and +transferring object references.

+ +

The most common way to make an object with remotely-callable methods is to +subclass Referenceable. Let's create a simple +server which does basic arithmetic. You might use such a service to perform +difficult mathematical operations, like addition, on a remote machine which +is faster and more capable than your ownalthough +really, if your client machine is too slow to perform this kind of math, it +is probably too slow to run python or use a network, so you should seriously +consider a hardware upgrade.

+ +
+from foolscap import Referenceable
+
+class MathServer(Referenceable):
+    def remote_add(self, a, b):
+        return a+b
+    def remote_subtract(self, a, b):
+        return a-b
+    def remote_sum(self, args):
+        total = 0
+        for a in args: total += a
+        return total
+
+myserver = MathServer()
+
+ +

On the other end of the wire (which you might call the client +side), the code will have a RemoteReference to this object. The +RemoteReference has a method named callRemote which you +will use to invoke the method. It always returns a Deferred, which will fire +with the result of the method. Assuming you've already acquired the +RemoteReference, you would invoke the method like this:

+ +
+def gotAnswer(result):
+    print "result is", result
+def gotError(err):
+    print "error:", err
+d = remote.callRemote("add", 1, 2)
+d.addCallbacks(gotAnswer, gotError)
+
+ +

Ok, now how do you acquire that RemoteReference? How do you +make the Referenceable available to the outside world? For this, +we'll need to discuss the Tub, and the concept of a FURL +URL.

+ +

Tubs: The Foolscap Service

+ +

The Tub is the container that +you use to publish Referenceables, and is the middle-man you use +to access Referenceables on other systems. It is known as the +Tub, since it provides similar naming and identification properties as +the E language's Vatbut they do not provide quite the same insulation against +other objects as E's Vats do. In this sense, Tubs are leaky Vats.. If +you want to make a Referenceable available to the world, you +create a Tub, tell it to listen on a TCP port, and then register the +Referenceable with it under a name of your choosing. If you want +to access a remote Referenceable, you create a Tub and ask it to +acquire a RemoteReference using that same name.

+ +

The Tub is a Twisted Service subclass, so you use it in +the same way: once you've created one, you attach it to a parent Service or +Application object. Once the top-level Application object has been started, +the Tub will start listening on any network ports you've requested. When the +Tub is shut down, it will stop listening and drop any connections it had +established since last startup. If you have no parent to attach it to, you +can use startService and stopService on the Tub +directly.

+ +

Note that no network activity will occur until the Tub's +startService method has been called. This means that any +getReference or connectTo requests that occur +before the Tub is started will be deferred until startup. If the program +forgets to start the Tub, these requests will never be serviced. A message to +this effect is added to the twistd.log file to help developers discover this +kind of problem.

+ +

Making your Tub remotely accessible

+ +

To make any of your Referenceables available, you must make +your Tub available. There are three parts: give it an identity, have it +listen on a port, and tell it the protocol/hostname/portnumber at which that +port is accessibly to the outside world.

+ +

In general, the Tub will generate its own identity, the TubID, by +creating an SSL private key certificate and hashing it into a suitably-long +random-looking string. This is the primary identifier of the Tub: everything +else is just a location hint that suggests how the Tub might be +reached. The fact that the TubID is tied to the private key allows FURLs to +be secure references (meaning that no third party can cause you to +connect to the wrong reference). You can also create a Tub with a +pre-existing certificate, which is how Tubs can retain a persistent identity +over multiple executions.

+ +

You can also create an UnauthenticatedTub, which has an empty +TubID. Hosting and connecting to unauthenticated Tubs do not require the +pyOpenSSL library, but do not provide privacy, authentication, connection +redirection, or shared listening ports. The FURLs that point to +unauthenticated Tubs have a distinct form (starting with pbu: +instead of pb:) to make sure they are not mistaken for +authenticated Tubs. Foolscap uses authenticated Tubs by default.

+ +

Having the Tub listen on a TCP port is as simple as calling listenOn with a strports-formatted port specification +string. The simplest such string would be tcp:12345, to listen on port +12345 on all interfaces. Using tcp:12345:interface=127.0.0.1 would +cause it to only listen on the localhost interface, making it available only +to other processes on the same host. The strports module +provides many other possibilities.

+ +

The Tub needs to be told how it can be reached, so it knows what host and +port to put into the URLs it creates. This location is simply a string in the +format host:port, using the host name by which that TCP port you've +just opened can be reached. Foolscap cannot, in general, guess what this name +is, especially if there are NAT boxes or port-forwarding devices in the way. +If your machine is reachable directly over the internet as +myhost.example.com, then you could use something like this:

+ +
+from foolscap import Tub
+
+tub = Tub()
+tub.listenOn("tcp:12345")  # start listening on TCP port 12345
+tub.setLocation("myhost.example.com:12345")
+
+ +

Registering the Referenceable

+ +

Once the Tub has a Listener and a location, you can publish your +Referenceable to the entire world by picking a name and +registering it:

+ +
+url = tub.registerReference(myserver, "math-service")
+
+ +

This returns the FURL for your Referenceable. Remote +systems will use this URL to access your newly-published object. The +registration just maps a per-Tub name to the Referenceable: +technically the same Referenceable could be published multiple +times, under different names, or even be published by multiple Tubs in the +same application. But in general, each program will have exactly one Tub, and +each object will be registered under only one name.

+ +

In this example (if we pretend the generated TubID was ABCD), the +URL returned by registerReference would be +"pb://ABCD@myhost.example.com:12345/math-service".

+ +

If you do not provide a name, a random (and unguessable) name will be +generated for you. This is useful when you want to give access to your +Referenceable to someone specific, but do not want to make it +possible for someone else to acquire it by guessing the name.

+ +

To use an unauthenticated Tub instead, you would do the following:

+
+from foolscap import UnauthenticatedTub
+
+tub = UnauthenticatedTub()
+tub.listenOn("tcp:12345")  # start listening on TCP port 12345
+tub.setLocation("myhost.example.com:12345")
+url = tub.registerReference(myserver, "math-service")
+
+ +

In this case, the URL would be +"pbu://myhost.example.com:12345/math-service". The deterministic +nature of this form makes it slightly easier to throw together +quick-and-dirty Foolscap applications, since you only need to hard-code the +target host and port into the client side program. However any serious +application should just used the default authenticated form and use a full +URL as their starting point. Note that the URL can come from anywhere: typed +in by the user, retrieved from a web page, or hardcoded into the +application.

+ +

Using a persistent certificate

+ +

The Tub uses a TLS private-key certificate as the base of all its +cryptographic operations. If you don't give it one when you create the Tub, +it will generate a brand-new one.

+ +

The TubID is simply the hash of this certificate, so if you are writing an +application that should have a stable long-term identity, you will need to +insure that the Tub uses the same certificate every time your app starts. The +easiest way to do this is to pass the certFile= argument into +your Tub() constructor call. This argument provides a filename +where you want the Tub to store its certificate. The first time the Tub is +started (when this file does not exist), the Tub will generate a new +certificate and store it here. On subsequent invocations, the Tub will read +the earlier certificate from this location. Make sure this filename points to +a writable location, and that you pass the same filename to +Tub() each time.

+ +

Retrieving a RemoteReference

+ +

On the client side, you also need to create a Tub, although you +don't need to perform the (listenOn, setLocation, +registerReference) sequence unless you are also publishing +Referenceables to the world. To acquire a reference to somebody +else's object, just use getReference:

+ +
+from foolscap import Tub
+
+tub = Tub()
+tub.startService()
+d = tub.getReference("pb://ABCD@myhost.example.com:12345/math-service")
+def gotReference(remote):
+    print "Got the RemoteReference:", remote
+def gotError(err):
+    print "error:", err
+d.addCallbacks(gotReference, gotError)
+
+ +

getReference returns a Deferred which will fire with a +RemoteReference that is connected to the remote +Referenceable named by the URL. It will use an existing +connection, if one is available, and it will return an existing +RemoteReference, it one has already been acquired.

+ +

Since getReference requests are queued until the Tub starts, +the following will work too. But don't forget to call +tub.startService() eventually, otherwise your program will hang +forever.

+ +
+from foolscap import Tub
+
+tub = Tub()
+d = tub.getReference("pb://ABCD@myhost.example.com:12345/math-service")
+def gotReference(remote):
+    print "Got the RemoteReference:", remote
+def gotError(err):
+    print "error:", err
+d.addCallbacks(gotReference, gotError)
+tub.startService()
+
+ + +

Complete example

+ +

Here are two programs, one implementing the server side of our +remote-addition protocol, the other behaving as a client. This first example +uses an unauthenticated Tub so you don't have to manually copy a URL from the +server to the client. Both of these are standalone programs (you just run +them), but normally you would create an Application object and pass the +file to twistd -noy. An example of that usage will be provided +later.

+ +pb1server.py + +pb1client.py + +
+% doc/listings/pb1server.py
+the object is available at: pbu://localhost:12345/math-service
+
+ +
+% doc/listings/pb1client.py
+got a RemoteReference
+asking it to add 1+2
+the answer is 3
+%
+
+ +

The second example uses authenticated Tubs. When running this example, you +must copy the URL printed by the server and provide it as an argument to the +client.

+ +pb2server.py + +pb2client.py + +
+% doc/listings/pb2server.py
+the object is available at: pb://abcd123@localhost:12345/math-service
+
+ +
+% doc/listings/pb2client.py pb://abcd123@localhost:12345/math-service
+got a RemoteReference
+asking it to add 1+2
+the answer is 3
+%
+
+ + +

FURLs

+ +

In Foolscap, each world-accessible Referenceable has one or more URLs +which are secure, where we use the capability-security definition of +the term, meaning those URLs have the following properties:

+ +
    +
  • The only way to acquire the URL is either to get it from someone else + who already has it, or to be the person who published it in the first + place.
    • + +
  • Only that original creator of the URL gets to determine which + Referenceable it will connect to. If your + tub.getReference(url) call succeeds, the Referenceable you + will be connected to will be the right one.
    • +
+ +

To accomplish the first goal, FURLs must be unguessable. You can register +the reference with a human-readable name if your intention is to make it +available to the world, but in general you will let +tub.registerReference generate a random name for you, preserving +the unguessability property.

+ +

To accomplish the second goal, the cryptographically-secure TubID is used +as the primary identifier, and the location hints are just that: +hints. If DNS has been subverted to point the hostname at a different +machine, or if a man-in-the-middle attack causes you to connect to the wrong +box, the TubID will not match the remote end, and the connection will be +dropped. These attacks can cause a denial-of-service, but they cannot cause +you to mistakenly connect to the wrong target.

+ +

Obviously this second property only holds if you use SSL. If you choose to +use unauthenticated Tubs, all security properties are lost.

+ +

The format of a FURL, like +pb://abcd123@example.com:5901,backup.example.com:8800/math-server, +is as followsnote that the FURL uses the same format +as an HTTPSY +URL:

+ +
    +
  1. The literal string pb://
    2. +
  2. The TubID (as a base32-encoded hash of the SSL certificate)
    3. +
  3. A literal @ sign
    4. + +
  4. A comma-separated list of location hints. Each is one of the + following: +
      +
    • TCP over IPv4 via DNS: HOSTNAME:PORTNUM
      • +
    • TCP over IPv4 without DNS: A.B.C.D:PORTNUM
      • +
    • TCP over IPv6: (TODO, maybe tcp6:HOSTNAME:PORTNUM ?
      • +
    • TCP over IPv6 w/o DNS: (TODO, + maybe tcp6:[X:Y::Z]:PORTNUM
      • +
    • Unix-domain socket: (TODO)
      • +
    + + Each location hint is attempted in turn. Servers can return a + redirect, which will cause the client to insert the provided + redirect targets into the hint list and start trying them before continuing + with the original list.
    5. + +
  5. A literal / character
    6. +
  6. The reference's name
    7. +
+ +

(Unix-domain sockets are represented with only a single location hint, in +the format pb://ABCD@unix/path/to/socket/NAME, but this needs +some work)

+ +

FURLs for unauthenticated Tubs, like +pbu://example.com:8700/math-server, are formatted as +follows:

+ +
    +
  1. The literal string pbu://
    2. +
  2. A single location hint
    3. +
  3. A literal / character
    4. +
  4. The reference's name
    5. +
+ +

Clients vs Servers, Names and Capabilities

+ +

It is worthwhile to point out that Foolscap is a symmetric protocol. +Referenceable instances can live on either side of a wire, and +the only difference between client and server is who publishes +the object and who initiates the network connection.

+ +

In any Foolscap-using system, the very first object exchanged must be +acquired with a tub.getReference(url) callin fact, the very very first object exchanged is a +special implicit RemoteReference to the remote Tub itself, which implements +an internal protocol that includes a method named +remote_getReference. The tub.getReference(url) call +is turned into one step that connects to the remote Tub, and a second step +which invokes remotetub.callRemote("getReference", refname) on the +result, which means it must have been published with a call to +tub.registerReference(ref, name). After that, other objects can +be passed as an argument to (or a return value from) a remotely-invoked +method of that first object. Any suitable Referenceable object +that is passed over the wire will appear on the other side as a corresponding +RemoteReference. It is not necessary to +registerReference something to let it pass over the wire.

+ +

The converse of this property is thus: if you do not +registerReference a particular Referenceable, and +you do not give it to anyone else (by passing it in an argument to +somebody's remote method, or return it from one of your own), then nobody +else will be able to get access to that Referenceable. This +property means the Referenceable is a capability, as +holding a corresponding RemoteReference gives someone a power +that they cannot acquire in any other wayof course, +the Foolscap connections must be secured with SSL (otherwise an eavesdropper +or man-in-the-middle could get access), and the registered name must be +unguessable (or someone else could acquire a reference), but both of these +are the default.

+ +

In the following example, the first program creates an RPN-style +Calculator object which responds to push, pop, +add, and subtract messages from the user. The user can also +register an Observer, to which the Calculator sends an +event message each time something happens to the calculator's +state. When you consider the Calculator object, the first +program is the server and the second program is the client. When you think +about the Observer object, the first program is a client and the +second program is the server. It also happens that the first program is +listening on a socket, while the second program initiated a network +connection to the first. It also happens that the first program +published an object under some well-known name, while the second program has +not published any objects. These are all independent properties.

+ +

Also note that the Calculator side of the example is implemented using a +Application +object, which is the way you'd normally build a real-world application. You +therefore use twistd to launch the program. The User side is +written with the same reactor.run() style as the earlier +example.

+ +

The server registers the Calculator instance and prints the FURL at which +it is listening. You need to pass this FURL to the client program so it knows +how to contact the servre. If you have a modern version of Twisted (2.5 or +later) and the right encryption libraries installed, you'll get an +authenticated Tub (for which the FURL will start with "pb:" and will be +fairly long). If you don't, you'll get an unauthenticated Tub (with a +relatively short FURL that starts with "pbu:").

+ +pb3calculator.py + +pb3user.py + +
+% twistd -noy doc/listings/pb3calculator.py 
+15:46 PDT [-] Log opened.
+15:46 PDT [-] twistd 2.4.0 (/usr/bin/python 2.4.4) starting up
+15:46 PDT [-] reactor class: twisted.internet.selectreactor.SelectReactor
+15:46 PDT [-] Loading doc/listings/pb3calculator.py...
+15:46 PDT [-] the object is available at:
+              pb://5ojw4cv4u4d5cenxxekjukrogzytnhop@localhost:12345/calculator
+15:46 PDT [-] Loaded.
+15:46 PDT [-] foolscap.pb.Listener starting on 12345
+15:46 PDT [-] Starting factory <Listener at 0x4869c0f4 on tcp:12345
+              with tubs None>
+
+ +
+% doc/listings/pb3user.py \
+   pb://5ojw4cv4u4d5cenxxekjukrogzytnhop@localhost:12345/calculator
+event: push(2)
+event: push(3)
+event: add
+event: pop
+the result is 5
+%
+
+ + +

Invoking Methods, Method Arguments

+ +

As you've probably already guessed, all the methods with names that begin +with remote_ will be available to anyone who manages to acquire +a corresponding RemoteReference. remote_foo matches +a ref.callRemote("foo"), etc. This name lookup can be changed by +overriding Referenceable (or, perhaps more usefully, +implementing an IRemotelyCallable adapter).

+ +

The arguments of a remote method may be passed as either positional +parameters (foo(1,2)), or as keyword args +(foo(a=1,b=2)), or a mixture of both. The usual python rules +about not duplicating parameters apply.

+ +

You can pass all sorts of normal objects to a remote method: strings, +numbers, tuples, lists, and dictionaries. The serialization of these objects +is handled by Banana, which knows +how to convey arbitrary object graphs over the wire. Things like containers +which contain multiple references to the same object, and recursive +references (cycles in the object graph) are all handled correctlyyou may not want to accept shared objects in your method +arguments, as it could lead to surprising behavior depending upon how you +have written your method. The Shared constraint will let you express this, +and is described in the Constraints section of +this document.

+ +

Passing instances is handled specially. Foolscap will not send anything +over the wire that it does not know how to serialize, and (unlike the +standard pickle module) it will not make assumptions about how +to handle classes that that have not been explicitly marked as serializable. +This is for security, both for the sender (making you you don't pass anything +over the wire that you didn't intend to let out of your security perimeter), +and for the recipient (making sure outsiders aren't allowed to create +arbitrary instances inside your memory space, and therefore letting them run +somewhat arbitrary code inside your perimeter).

+ +

Sending Referenceables is straightforward: they always appear +as a corresponding RemoteReference on the other side. You can +send the same Referenceable as many times as you like, and it +will always show up as the same RemoteReference instance. A +distributed reference count is maintained, so as long as the remote side +hasn't forgotten about the RemoteReference, the original +Referenceable will be kept alive.

+ +

Sending RemoteReferences fall into two categories. If you are +sending a RemoteReference back to the Tub that you got it from, +they will see their original Referenceable. If you send it to +some other Tub, they will (eventually) see a RemoteReference of +their own. This last feature is called an introduction, and has a few +additional requirements: see the Introductions +section of this document for details.

+ +

Sending instances of other classes requires that you tell Banana how they +should be serialized. Referenceable is good for +copy-by-reference semanticsIn fact, if all you want is +referenceability (and not callability), you can use OnlyReferenceable. Strictly speaking, +Referenceable is both Referenceable (meaning it is sent +over the wire using pass-by-reference semantics, and it survives a round +trip) and Callable (meaning you can invoke remote methods on it). +Referenceable should really be named Callable, but +the existing name has a lot of historical weight behind it.. For +copy-by-value semantics, the easiest route is to subclass Copyable. See the Copyable section for details. Note that you can also +register an ICopyable +adapter on third-party classes to avoid subclassing. You will need to +register the Copyable's name on the receiving end too, otherwise +Banana will not know how to unserialize the incoming data stream.

+ +

When returning a value from a remote method, you can do all these things, +plus two more. If you raise an exception, the caller's Deferred will have the +errback fired instead of the callback, with a CopiedFailure instance that describes what went +wrong. The CopiedFailure is not quite as useful as a local Failure object would be: to +send it over the wire, some contents are replaced with strings, and the +actual Exception object (f.value) is replaced with its string +representation. But you can still use it to find out what went wrong. The +CopiedFailure may reveal more information about the internals of +your program than you want: you can set the unsafeTracebacks +flag on the Tub to limit outgoing CopiedFailures to contain only +the exception type (and none of the stack trace information that would reveal +lines of your source code to the remote end).

+ +

The other alternative is for your method to return a Deferred. If this happens, the caller +will not actually get a response until you fire that Deferred. This is useful +when the remote operation being requested cannot complete right away. The +caller's Deferred with fire with whatever value you eventually fire your own +Deferred with. If your Deferred is errbacked, their Deferred will be +errbacked with a CopiedFailure.

+ + +

Constraints and RemoteInterfaces

+ +

One major feature introduced by Foolscap (relative to oldpb) is the +serialization Constraint. +This lets you place limits on what kind of data you are willing to accept, +which enables safer distributed programming. Typically python uses duck +typing, wherein you usually just throw some arguments at the method and +see what happens. When you are less sure of the origin of those arguments, +you may want to be more circumspect. Enforcing type checking at the boundary +between your code and the outside world may make it safer to use duck typing +inside those boundaries. The type specifications also form a convenient +remote API reference you can publish for prospective clients of your +remotely-invokable service.

+ +

In addition, these Constraints are enforced on each token as it arrives +over the wire. This means that you can calculate a (small) upper bound on how +much received data your program will store before it decides to hang up on +the violator, minimizing your exposure to DoS attacks that involve sending +random junk at you.

+ +

There are three pieces you need to know about: Tokens, Constraints, and +RemoteInterfaces.

+ +

Tokens

+ +

The fundamental unit of serialization is the Banana Token. These are +thoroughly documented in the Banana +Specification, but what you need to know here is that each piece of +non-container data, like a string or a number, is represented by a single +token. Containers (like lists and dictionaries) are represented by a special +OPEN token, followed by tokens for everything that is in the container, +followed by the CLOSE token. Everything Banana does is in terms of these +nested OPEN/stuff/stuff/CLOSE sequences of tokens.

+ +

Each token consists of a header, a type byte, and an optional body. The +header is always a base-128 number with a maximum of 64 digits, and the type +byte is always a single byte. The body (if present) has a length dictated by +the magnitude of the header.

+ +

The length-first token format means that the receiving system never has to +accept more than 65 bytes before it knows the type and size of the token, at +which point it can make a decision about accepting or rejecting the rest of +it.

+ +

Constraints

+ +

The schema foolscap.schema module has a variety of Constraint classes that can be +applied to incoming data. Most of them correspond to typical Python types, +e.g. ListOf matches a list, +with a certain maximum length, and a child Constraint that gets +applied to the contents of the list. You can nest Constraints in +this way to describe the shape of the object graph that you are +willing to accept.

+ +

At any given time, the receiving Banana protocol has single +Constraint object that it enforces against the inbound data +streamto be precise, each Unslicer on the +receive stack has a Constraint, and the idea is that all of them +get to pass judgement on the inbound token. A useful syntax to describe this +sort of thing is still being worked out..

+ +

RemoteInterfaces

+ +

The RemoteInterface is how you describe +your constraints. You can provide a constraint for each argument of each +method, as well as one for the return value. You can also specify addtional +flags on the methods. The convention (which is actually enforced by the code) +is to name RemoteInterface objects with an RI prefix, +like RIFoo.

+ +

RemoteInterfaces are created and used a lot like the usual +zope.interface-style Interface. They look like +class definitions, inheriting from RemoteInterface. For each +method, the default value of each argument is used to create a +Constraint for that argument. Basic types (int, +str, bool) are converted into a +Constraint subclass (IntegerConstraint, StringConstraint, BooleanConstraint). You can also use +instances of other Constraint subclasses, like ListOf and DictOf. This Constraint will be +enforced against the value for the given argument. Unless you specify +otherwise, remote callers must match all the Constraints you +specify, all arguments listed in the RemoteInterface must be present, and no +arguments outside that list will be accepted.

+ +

Note that, like zope.interface, these methods should not include +self in their argument list. This is because you are +documenting how other people invoke your methods. self +is an implementation detail. RemoteInterface will complain if +you forget.

+ +

The methods in a RemoteInterface should return a +single value with the same format as the default arguments: either a basic +type (int, str, etc) or a Constraint +subclass. This Constraint is enforced on the return value of the +method. If you are calling a method in somebody else's process, the argument +constraints will be applied as a courtesy (be conservative in what you +send), and the return value constraint will be applied to prevent the +server from doing evil things to you. If you are running a method on behalf +of a remote client, the argument constraints will be enforced to protect +you, while the return value constraint will be applied as a +courtesy.

+ +

Attempting to send a value that does not satisfy the Constraint will +result in a Violation exception +being raised.

+ +

You can also specify methods by defining attributes of the same name in +the RemoteInterface object. Each attribute value should be an +instance of RemoteMethodSchemaalthough technically it can be any object which implements +the IRemoteMethodConstraint +interface. This approach is more flexible: there are some constraints +that are not easy to express with the default-argument syntax, and this is +the only way to set per-method flags. Note that all such method-defining +attributes must be set in the RemoteInterface body itself, +rather than being set on it after the fact (i.e. RIFoo.doBar = +stuff). This is required because the RemoteInterface +metaclass magic processes all of these attributes only once, immediately +after the RemoteInterface body has been evaluated.

+ +

The RemoteInterface class has a name. Normally this is +the (short) classnameRIFoo.__class__.__name__, if +RemoteInterfaces were actually classes, which they're +not. You can override this +name by setting a special __remote_name__ attribute on the +RemoteInterface (again, in the body). This name is important +because it is externally visible: all RemoteReferences that +point at your Referenceables will remember the name of the +RemoteInterfaces it implements. This is what enables the +type-checking to be performed on both ends of the wire.

+ +

In the future, this ought to default to the fully-qualified +classname (like package.module.RIFoo), so that two +RemoteInterfaces with the same name in different modules can co-exist. In the +current release, these two RemoteInterfaces will collide (and provoke an +import-time error message complaining about the duplicate name). As a result, +if you have such classes (e.g. foo.RIBar and +baz.RIBar), you must use __remote_name__ to +distinguish them (by naming one of them something other than +RIBar to avoid this error. + +Hopefully this will be improved in a future version, but it looks like a +difficult change to implement, so the standing recommendation is to use +__remote_name__ on all your RemoteInterfaces, and set it to a +suitably unique string (like a URI).

+ +

Here's an example:

+ +
+from foolscap import RemoteInterface, schema
+
+class RIMath(RemoteInterface):
+    __remote_name__ = "RIMath.using-foolscap.docs.foolscap.twistedmatrix.com"
+    def add(a=int, b=int):
+        return int
+    # declare it with an attribute instead of a function definition
+    subtract = schema.RemoteMethodSchema(a=int, b=int, _response=int)
+    def sum(args=schema.ListOf(int)):
+        return int
+
+ + +

Using RemoteInterface

+ +

To declare that your Referenceable responds to a particular +RemoteInterface, use the normal implements() +annotation:

+ +
+class MathServer(foolscap.Referenceable):
+    implements(RIMath)
+
+    def remote_add(self, a, b):
+        return a+b
+    def remote_subtract(self, a, b):
+        return a-b
+    def remote_sum(self, args):
+        total = 0
+        for a in args: total += a
+        return total
+
+ +

To enforce constraints everywhere, both sides will need to know about the +RemoteInterface, and both must know it by the same name. It is a +good idea to put the RemoteInterface in a common file that is +imported into the programs running on both sides. It is up to you to make +sure that both sides agree on the interface. Future versions of Foolscap may +implement some sort of checksum-verification or Interface-serialization as a +failsafe, but fundamentally the RemoteInterface that +you are using defines what your program is prepared to +handle. There is no difference between an old client accidentaly using a +different version of the RemoteInterface by mistake, and a malicious attacker +actively trying to confuse your code. The only promise that Foolscap can make +is that the constraints you provide in the RemoteInterface will be faithfully +applied to the incoming data stream, so that you don't need to do the type +checking yourself inside the method.

+ +

When making a remote method call, you use the RemoteInterface +to identify the method instead of a string. This scopes the method name to +the RemoteInterface:

+ +
+d = remote.callRemote(RIMath["add"], a=1, b=2)
+# or
+d = remote.callRemote(RIMath["add"], 1, 2)
+
+ +

Pass-By-Copy

+ +

You can pass (nearly) arbitrary instances over the wire. Foolscap knows +how to serialize all of Python's native data types already: numbers, strings, +unicode strings, booleans, lists, tuples, dictionaries, sets, and the None +object. You can teach it how to serialize instances of other types too. +Foolscap will not serialize (or deserialize) any class that you haven't +taught it about, both for security and because it refuses the temptation to +guess your intentions about how these unknown classes ought to be +serialized.

+ +

The simplest possible way to pass things by copy is demonstrated in the +following code fragment:

+ +
+from foolscap import Copyable, RemoteCopy
+
+class MyPassByCopy(Copyable, RemoteCopy):
+    typeToCopy = copytype = "MyPassByCopy"
+    def __init__(self):
+        # RemoteCopy subclasses may not accept any __init__ arguments
+        pass
+    def setCopyableState(self, state):
+        self.__dict__ = state
+
+ +

If the code on both sides of the wire import this class, then any +instances of MyPassByCopy that are present in the arguments of a +remote method call (or returned as the result of a remote method call) will +be serialized and reconstituted into an equivalent instance on the other +side.

+ +

For more complicated things to do with pass-by-copy, see the documentation +on Copyable. This explains the difference between +Copyable and RemoteCopy, how to control the +serialization and deserialization process, and how to arrange for +serialization of third-party classes that are not subclasses of +Copyable.

+ + +

Third-party References

+ +

Another new feature of Foolscap is the ability to send +RemoteReferences to third parties. The classic scenario for this +is illustrated by the three-party +Granovetter diagram. One party (Alice) has RemoteReferences to two other +objects named Bob and Carol. She wants to share her reference to Carol with +Bob, by including it in a message she sends to Bob (i.e. by using it as an +argument when she invokes one of Bob's remote methods). The Foolscap code for +doing this would look like:

+ +
+bobref.callRemote("foo", intro=carolref)
+
+ +

When Bob receives this message (i.e. when his remote_foo +method is invoked), he will discover that he's holding a fully-functional +RemoteReference to the object named Caroland if everyone involved is using authenticated Tubs, then +Foolscap offers a guarantee, in the cryptographic sense, that Bob will wind +up with a reference to the same object that Alice intended. The authenticated +FURLs prevent DNS-spoofing and man-in-the-middle attacks.. He can +start using this RemoteReference right away:

+ +
+class Bob(foolscap.Referenceable):
+    def remote_foo(self, intro):
+        self.carol = intro
+        carol.callRemote("howdy", msg="Pleased to meet you", you=intro)
+        return carol
+
+ +

If Bob sends this RemoteReference back to Alice, her method +will see the same RemoteReference that she sent to Bob. In this +example, Bob sends the reference by returning it from the original +remote_foo method call, but he could almost as easily send it in +a separate method call.

+ +
+class Alice(foolscap.Referenceable):
+    def start(self, carol):
+        self.carol = carol
+        d = self.bob.callRemote("foo", intro=carol)
+        d.addCallback(self.didFoo)
+    def didFoo(self, result):
+        assert result is self.carol  # this will be true
+
+ +

Moreover, if Bob sends it back to Carol (completing the +three-party round trip), Carol will see it as her original +Referenceable.

+ +
+class Carol(foolscap.Referenceable):
+    def remote_howdy(self, msg, you):
+        assert you is self  # this will be true
+
+ +

In addition to this, in the four-party introduction sequence as used by +the Grant +Matcher Puzzle, when a Referenceable is sent to the same destination +through multiple paths, the recipient will receive the same +RemoteReference object from both sides.

+ +

For a RemoteReference to be transferrable to third-parties in +this fashion, the original Referenceable must live in a Tub +which has a working listening port, and an established base URL. It is not +necessary for the Referenceable to have been published with +registerReference first: if it is sent over the wire before a +name has been associated with it, it will be registered under a new random +and unguessable name. The RemoteReference will contain the +resulting URL, enabling it to be sent to third parties.

+ +

When this introduction is made, the receiving system must establish a +connection with the Tub that holds the original Referenceable, and acquire +its own RemoteReference. These steps must take place before the remote method +can be invoked, and other method calls might arrive before they do. All +subsequent method calls are queued until the one that involved the +introduction is performed. Foolscap guarantees (by default) that the messages +sent to a given Referenceable will be delivered in the same order. In the +future there may be options to relax this guarantee, in exchange for higher +performance, reduced memory consumption, multiple priority queues, limited +latency, or other features. There might even be an option to turn off +introductions altogether.

+ +

Also note that enabling this capability means any of your communication +peers can make you create TCP connections to hosts and port numbers of their +choosing. The fact that those connections can only speak the Foolscap +protocol may reduce the security risk presented, but it still lets other +people be annoying.

+ + + diff --git a/src/foolscap/doc/using-pb.xhtml b/src/foolscap/doc/using-pb.xhtml deleted file mode 100644 index 97005d01..00000000 --- a/src/foolscap/doc/using-pb.xhtml +++ /dev/null @@ -1,958 +0,0 @@ - - -Introduction to Foolscap - - - - -

Introduction to Foolscap

- -

Introduction

- -

Suppose you find yourself in control of both ends of the wire: you have -two programs that need to talk to each other, and you get to use any protocol -you want. If you can think of your problem in terms of objects that need to -make method calls on each other, then chances are good that you can use the -Foolscap protocol rather than trying to shoehorn your needs into something -like HTTP, or implementing yet another RPC mechanism.

- -

Foolscap is based upon a few central concepts:

- -
    - -
  • serialization: taking fairly arbitrary objects and types, - turning them into a chunk of bytes, sending them over a wire, then - reconstituting them on the other end. By keeping careful track of object - ids, the serialized objects can contain references to other objects and the - remote copy will still be useful.
    • - -
  • remote method calls: doing something to a local proxy and - causing a method to get run on a distant object. The local proxy is called - a RemoteReference, - and you do something by running its .callRemote method. - The distant object is called a Referenceable, and it has methods like - remote_foo that will be invoked.
    • - -
- -

Foolscap is the descendant of Perspective Broker (which lived in the -twisted.spread package). For many years it was known as "newpb". A lot of the -API still has the name "PB" in it somewhere. These will probably go away -sooner or later.

- -

A "foolscap" is a size of paper, probably measuring 17 by 13.5 inches. A -twisted foolscap of paper makes a good fool's cap. Also, "cap" makes me think -of capabilities, and Foolscap is a protocol to implement a distributed -object-capabilities model in python.

- - -

Getting Started

- -

Any Foolscap application has at least two sides: one which hosts a -remotely-callable object, and another which calls (remotely) the methods of -that object. We'll start with a simple example that demostrates both ends. -Later, we'll add more features like RemoteInterface declarations, and -transferring object references.

- -

The most common way to make an object with remotely-callable methods is to -subclass Referenceable. Let's create a simple -server which does basic arithmetic. You might use such a service to perform -difficult mathematical operations, like addition, on a remote machine which -is faster and more capable than your ownalthough -really, if your client machine is too slow to perform this kind of math, it -is probably too slow to run python or use a network, so you should seriously -consider a hardware upgrade.

- -
-from foolscap import Referenceable
-
-class MathServer(Referenceable):
-    def remote_add(self, a, b):
-        return a+b
-    def remote_subtract(self, a, b):
-        return a-b
-    def remote_sum(self, args):
-        total = 0
-        for a in args: total += a
-        return total
-
-myserver = MathServer()
-
- -

On the other end of the wire (which you might call the client -side), the code will have a RemoteReference to this object. The -RemoteReference has a method named callRemote which you -will use to invoke the method. It always returns a Deferred, which will fire -with the result of the method. Assuming you've already acquired the -RemoteReference, you would invoke the method like this:

- -
-def gotAnswer(result):
-    print "result is", result
-def gotError(err):
-    print "error:", err
-d = remote.callRemote("add", 1, 2)
-d.addCallbacks(gotAnswer, gotError)
-
- -

Ok, now how do you acquire that RemoteReference? How do you -make the Referenceable available to the outside world? For this, -we'll need to discuss the Tub, and the concept of a PB URL.

- -

Tubs: The Foolscap Service

- -

The Tub is the container that -you use to publish Referenceables, and is the middle-man you use -to access Referenceables on other systems. It is known as the -Tub, since it provides similar naming and identification properties as -the E language's Vatbut they do not provide quite the same insulation against -other objects as E's Vats do. In this sense, Tubs are leaky Vats.. If -you want to make a Referenceable available to the world, you -create a Tub, tell it to listen on a TCP port, and then register the -Referenceable with it under a name of your choosing. If you want -to access a remote Referenceable, you create a Tub and ask it to -acquire a RemoteReference using that same name.

- -

The Tub is a Twisted Service subclass, so you use it in -the same way: once you've created one, you attach it to a parent Service or -Application object. Once the top-level Application object has been started, -the Tub will start listening on any network ports you've requested. When the -Tub is shut down, it will stop listening and drop any connections it had -established since last startup. If you have no parent to attach it to, you -can use startService and stopService on the Tub -directly.

- -

Note that you must start the Tub before calling getReference -or connectTo, since both of these trigger network activity, and -Tubs are supposed to be silent until they are started. In a future release -this requirement may be relaxed, but the principle of "no network activity -until the Tub is started" will be maintained, probably by queueing the -getReference calls and handling them after the Tub been -started.

- -

Making your Tub remotely accessible

- -

To make any of your Referenceables available, you must make -your Tub available. There are three parts: give it an identity, have it -listen on a port, and tell it the protocol/hostname/portnumber at which that -port is accessibly to the outside world.

- -

In general, the Tub will generate its own identity, the TubID, by -creating an SSL private key certificate and hashing it into a suitably-long -random-looking string. This is the primary identifier of the Tub: everything -else is just a location hint that suggests how the Tub might be -reached. The fact that the TubID is tied to the private key allows PB URLs to -be secure references (meaning that no third party can cause you to -connect to the wrong reference). You can also create a Tub with a -pre-existing certificate, which is how Tubs can retain a persistent identity -over multiple executions.

- -

You can also create an UnauthenticatedTub, which has an empty -TubID. Hosting and connecting to unauthenticated Tubs do not require the -pyOpenSSL library, but do not provide privacy, authentication, connection -redirection, or shared listening ports. The PB-URLs that point to -unauthenticated Tubs have a distinct form (starting with pbu: -instead of pb:) to make sure they are not mistaken for -authenticated Tubs. PB uses authenticated Tubs by default.

- -

Having the Tub listen on a TCP port is as simple as calling listenOn with a strports-formatted port specification -string. The simplest such string would be tcp:12345, to listen on port -12345 on all interfaces. Using tcp:12345:interface=127.0.0.1 would -cause it to only listen on the localhost interface, making it available only -to other processes on the same host. The strports module -provides many other possibilities.

- -

The Tub needs to be told how it can be reached, so it knows what host and -port to put into the URLs it creates. This location is simply a string in the -format host:port, using the host name by which that TCP port you've -just opened can be reached. Foolscap cannot, in general, guess what this name -is, especially if there are NAT boxes or port-forwarding devices in the way. -If your machine is reachable directly over the internet as -myhost.example.com, then you could use something like this:

- -
-from foolscap import Tub
-
-tub = Tub()
-tub.listenOn("tcp:12345")  # start listening on TCP port 12345
-tub.setLocation("myhost.example.com:12345")
-
- -

Registering the Referenceable

- -

Once the Tub has a Listener and a location, you can publish your -Referenceable to the entire world by picking a name and -registering it:

- -
-url = tub.registerReference(myserver, "math-service")
-
- -

This returns the PB URL for your Referenceable. Remote -systems will use this URL to access your newly-published object. The -registration just maps a per-Tub name to the Referenceable: -technically the same Referenceable could be published multiple -times, under different names, or even be published by multiple Tubs in the -same application. But in general, each program will have exactly one Tub, and -each object will be registered under only one name.

- -

In this example (if we pretend the generated TubID was ABCD), the -URL returned by registerReference would be -"pb://ABCD@myhost.example.com:12345/math-service".

- -

If you do not provide a name, a random (and unguessable) name will be -generated for you. This is useful when you want to give access to your -Referenceable to someone specific, but do not want to make it -possible for someone else to acquire it by guessing the name.

- -

To use an unauthenticated Tub instead, you would do the following:

-
-from foolscap import UnauthenticatedTub
-
-tub = UnauthenticatedTub()
-tub.listenOn("tcp:12345")  # start listening on TCP port 12345
-tub.setLocation("myhost.example.com:12345")
-url = tub.registerReference(myserver, "math-service")
-
- -

In this case, the URL would be -"pbu://myhost.example.com:12345/math-service". The deterministic -nature of this form makes it slightly easier to throw together -quick-and-dirty Foolscap applications, since you only need to hard-code the -target host and port into the client side program. However any serious -application should just used the default authenticated form and use a full -URL as their starting point. Note that the URL can come from anywhere: typed -in by the user, retrieved from a web page, or hardcoded into the -application.

- -

Using a persistent certificate

- -

The Tub uses a TLS private-key certificate as the base of all its -cryptographic operations. If you don't give it one when you create the Tub, -it will generate a brand-new one.

- -

The TubID is simply the hash of this certificate, so if you are writing an -application that should have a stable long-term identity, you will need to -insure that the Tub uses the same certificate every time your app starts. The -easiest way to do this is to pass the certFile= argument into -your Tub() constructor call. This argument provides a filename -where you want the Tub to store its certificate. The first time the Tub is -started (when this file does not exist), the Tub will generate a new -certificate and store it here. On subsequent invocations, the Tub will read -the earlier certificate from this location. Make sure this filename points to -a writable location, and that you pass the same filename to -Tub() each time.

- -

Retrieving a RemoteReference

- -

On the client side, you also need to create a Tub, although you -don't need to perform the (listenOn, setLocation, -registerReference) sequence unless you are also publishing -Referenceables to the world. To acquire a reference to somebody -else's object, just use getReference:

- -
-from foolscap import Tub
-
-tub = Tub()
-tub.startService()
-d = tub.getReference("pb://ABCD@myhost.example.com:12345/math-service")
-def gotReference(remote):
-    print "Got the RemoteReference:", remote
-def gotError(err):
-    print "error:", err
-d.addCallbacks(gotReference, gotError)
-
- -

getReference returns a Deferred which will fire with a -RemoteReference that is connected to the remote -Referenceable named by the URL. It will use an existing -connection, if one is available, and it will return an existing -RemoteReference, it one has already been acquired.

- -

Complete example

- -

Here are two programs, one implementing the server side of our -remote-addition protocol, the other behaving as a client. This first example -uses an unauthenticated Tub so you don't have to manually copy a URL from the -server to the client. Both of these are standalone programs (you just run -them), but normally you would create an Application object and pass the -file to twistd -noy. An example of that usage will be provided -later.

- -pb1server.py - -pb1client.py - -
-% doc/listings/pb1server.py
-the object is available at: pbu://localhost:12345/math-service
-
- -
-% doc/listings/pb1client.py
-got a RemoteReference
-asking it to add 1+2
-the answer is 3
-%
-
- -

The second example uses authenticated Tubs. When running this example, you -must copy the URL printed by the server and provide it as an argument to the -client.

- -pb2server.py - -pb2client.py - -
-% doc/listings/pb2server.py
-the object is available at: pb://abcd123@localhost:12345/math-service
-
- -
-% doc/listings/pb2client.py pb://abcd123@localhost:12345/math-service
-got a RemoteReference
-asking it to add 1+2
-the answer is 3
-%
-
- - -

PB URLs

- -

In Foolscap, each world-accessible Referenceable has one or more URLs -which are secure, where we use the capability-security definition of -the term, meaning those URLs have the following properties:

- -
    -
  • The only way to acquire the URL is either to get it from someone else - who already has it, or to be the person who published it in the first - place.
    • - -
  • Only that original creator of the URL gets to determine which - Referenceable it will connect to. If your - tub.getReference(url) call succeeds, the Referenceable you - will be connected to will be the right one.
    • -
- -

To accomplish the first goal, PB URLs must be unguessable. You can -register the reference with a human-readable name if your intention is to -make it available to the world, but in general you will let -tub.registerReference generate a random name for you, preserving -the unguessability property.

- -

To accomplish the second goal, the cryptographically-secure TubID is used -as the primary identifier, and the location hints are just that: -hints. If DNS has been subverted to point the hostname at a different -machine, or if a man-in-the-middle attack causes you to connect to the wrong -box, the TubID will not match the remote end, and the connection will be -dropped. These attacks can cause a denial-of-service, but they cannot cause -you to mistakenly connect to the wrong target.

- -

Obviously this second property only holds if you use SSL. If you choose to -use unauthenticated Tubs, all security properties are lost.

- -

The format of a PB URL, like -pb://abcd123@example.com:5901,backup.example.com:8800/math-server, -is as followsnote that the PB URL uses the same format -as an HTTPSY -URL:

- -
    -
  1. The literal string pb://
    2. -
  2. The TubID (as a base32-encoded hash of the SSL certificate)
    3. -
  3. A literal @ sign
    4. - -
  4. A comma-separated list of location hints. Each is one of the - following: -
      -
    • TCP over IPv4 via DNS: HOSTNAME:PORTNUM
      • -
    • TCP over IPv4 without DNS: A.B.C.D:PORTNUM
      • -
    • TCP over IPv6: (TODO, maybe tcp6:HOSTNAME:PORTNUM ?
      • -
    • TCP over IPv6 w/o DNS: (TODO, - maybe tcp6:[X:Y::Z]:PORTNUM
      • -
    • Unix-domain socket: (TODO)
      • -
    - - Each location hint is attempted in turn. Servers can return a - redirect, which will cause the client to insert the provided - redirect targets into the hint list and start trying them before continuing - with the original list.
    5. - -
  5. A literal / character
    6. -
  6. The reference's name
    7. -
- -

(Unix-domain sockets are represented with only a single location hint, in -the format pb://ABCD@unix/path/to/socket/NAME, but this needs -some work)

- -

PB URLs for unauthenticated Tubs, like -pbu://example.com:8700/math-server, are formatted as -follows:

- -
    -
  1. The literal string pbu://
    2. -
  2. A single location hint
    3. -
  3. A literal / character
    4. -
  4. The reference's name
    5. -
- -

Clients vs Servers, Names and Capabilities

- -

It is worthwhile to point out that Foolscap is a symmetric protocol. -Referenceable instances can live on either side of a wire, and -the only difference between client and server is who publishes -the object and who initiates the network connection.

- -

In any Foolscap-using system, the very first object exchanged must be -acquired with a tub.getReference(url) callin fact, the very very first object exchanged is a -special implicit RemoteReference to the remote Tub itself, which implements -an internal protocol that includes a method named -remote_getReference. The tub.getReference(url) call -is turned into one step that connects to the remote Tub, and a second step -which invokes remotetub.callRemote("getReference", refname) on the -result, which means it must have been published with a call to -tub.registerReference(ref, name). After that, other objects can -be passed as an argument to (or a return value from) a remotely-invoked -method of that first object. Any suitable Referenceable object -that is passed over the wire will appear on the other side as a corresponding -RemoteReference. It is not necessary to -registerReference something to let it pass over the wire.

- -

The converse of this property is thus: if you do not -registerReference a particular Referenceable, and -you do not give it to anyone else (by passing it in an argument to -somebody's remote method, or return it from one of your own), then nobody -else will be able to get access to that Referenceable. This -property means the Referenceable is a capability, as -holding a corresponding RemoteReference gives someone a power -that they cannot acquire in any other wayof course, -the Foolscap connections must be secured with SSL (otherwise an eavesdropper -or man-in-the-middle could get access), and the registered name must be -unguessable (or someone else could acquire a reference), but both of these -are the default.

- -

In the following example, the first program creates an RPN-style -Calculator object which responds to push, pop, -add, and subtract messages from the user. The user can also -register an Observer, to which the Calculator sends an -event message each time something happens to the calculator's -state. When you consider the Calculator object, the first -program is the server and the second program is the client. When you think -about the Observer object, the first program is a client and the -second program is the server. It also happens that the first program is -listening on a socket, while the second program initiated a network -connection to the first. It also happens that the first program -published an object under some well-known name, while the second program has -not published any objects. These are all independent properties.

- -

Also note that the Calculator side of the example is implemented using a -Application -object, which is the way you'd normally build a real-world application. You -therefore use twistd to launch the program. The User side is -written with the same reactor.run() style as the earlier -example.

- -

The server registers the Calculator instance and prints the PBURL at which -it is listening. You need to pass this PBURL to the client program so it -knows how to contact the servre. If you have a modern version of Twisted (2.5 -or later) and the right encryption libraries installed, you'll get an -authenticated Tub (for which the PBURL will start with "pb:" and will be -fairly long). If you don't, you'll get an unauthenticated Tub (with a -relatively short PBURL that starts with "pbu:").

- -pb3calculator.py - -pb3user.py - -
-% twistd -noy doc/listings/pb3calculator.py 
-15:46 PDT [-] Log opened.
-15:46 PDT [-] twistd 2.4.0 (/usr/bin/python 2.4.4) starting up
-15:46 PDT [-] reactor class: twisted.internet.selectreactor.SelectReactor
-15:46 PDT [-] Loading doc/listings/pb3calculator.py...
-15:46 PDT [-] the object is available at:
-              pb://5ojw4cv4u4d5cenxxekjukrogzytnhop@localhost:12345/calculator
-15:46 PDT [-] Loaded.
-15:46 PDT [-] foolscap.pb.Listener starting on 12345
-15:46 PDT [-] Starting factory <Listener at 0x4869c0f4 on tcp:12345
-              with tubs None>
-
- -
-% doc/listings/pb3user.py \
-   pb://5ojw4cv4u4d5cenxxekjukrogzytnhop@localhost:12345/calculator
-event: push(2)
-event: push(3)
-event: add
-event: pop
-the result is 5
-%
-
- - -

Invoking Methods, Method Arguments

- -

As you've probably already guessed, all the methods with names that begin -with remote_ will be available to anyone who manages to acquire -a corresponding RemoteReference. remote_foo matches -a ref.callRemote("foo"), etc. This name lookup can be changed by -overriding Referenceable (or, perhaps more usefully, -implementing an IRemotelyCallable adapter).

- -

The arguments of a remote method may be passed as either positional -parameters (foo(1,2)), or as keyword args -(foo(a=1,b=2)), or a mixture of both. The usual python rules -about not duplicating parameters apply.

- -

You can pass all sorts of normal objects to a remote method: strings, -numbers, tuples, lists, and dictionaries. The serialization of these objects -is handled by Banana, which knows -how to convey arbitrary object graphs over the wire. Things like containers -which contain multiple references to the same object, and recursive -references (cycles in the object graph) are all handled correctlyyou may not want to accept shared objects in your method -arguments, as it could lead to surprising behavior depending upon how you -have written your method. The Shared constraint will let you express this, -and is described in the Constraints section of -this document.

- -

Passing instances is handled specially. Foolscap will not send anything -over the wire that it does not know how to serialize, and (unlike the -standard pickle module) it will not make assumptions about how -to handle classes that that have not been explicitly marked as serializable. -This is for security, both for the sender (making you you don't pass anything -over the wire that you didn't intend to let out of your security perimeter), -and for the recipient (making sure outsiders aren't allowed to create -arbitrary instances inside your memory space, and therefore letting them run -somewhat arbitrary code inside your perimeter).

- -

Sending Referenceables is straightforward: they always appear -as a corresponding RemoteReference on the other side. You can -send the same Referenceable as many times as you like, and it -will always show up as the same RemoteReference instance. A -distributed reference count is maintained, so as long as the remote side -hasn't forgotten about the RemoteReference, the original -Referenceable will be kept alive.

- -

Sending RemoteReferences fall into two categories. If you are -sending a RemoteReference back to the Tub that you got it from, -they will see their original Referenceable. If you send it to -some other Tub, they will (eventually) see a RemoteReference of -their own. This last feature is called an introduction, and has a few -additional requirements: see the Introductions -section of this document for details.

- -

Sending instances of other classes requires that you tell Banana how they -should be serialized. Referenceable is good for -copy-by-reference semanticsIn fact, if all you want is -referenceability (and not callability), you can use OnlyReferenceable. Strictly speaking, -Referenceable is both Referenceable (meaning it is sent -over the wire using pass-by-reference semantics, and it survives a round -trip) and Callable (meaning you can invoke remote methods on it). -Referenceable should really be named Callable, but -the existing name has a lot of historical weight behind it.. For -copy-by-value semantics, the easiest route is to subclass Copyable. See the Copyable section for details. Note that you can also -register an ICopyable -adapter on third-party classes to avoid subclassing. You will need to -register the Copyable's name on the receiving end too, otherwise -Banana will not know how to unserialize the incoming data stream.

- -

When returning a value from a remote method, you can do all these things, -plus two more. If you raise an exception, the caller's Deferred will have the -errback fired instead of the callback, with a CopiedFailure instance that describes what went -wrong. The CopiedFailure is not quite as useful as a local Failure object would be: to -send it over the wire, some contents are replaced with strings, and the -actual Exception object (f.value) is replaced with its string -representation. But you can still use it to find out what went wrong. The -CopiedFailure may reveal more information about the internals of -your program than you want: you can set the unsafeTracebacks -flag on the Tub to limit outgoing CopiedFailures to contain only -the exception type (and none of the stack trace information that would reveal -lines of your source code to the remote end).

- -

The other alternative is for your method to return a Deferred. If this happens, the caller -will not actually get a response until you fire that Deferred. This is useful -when the remote operation being requested cannot complete right away. The -caller's Deferred with fire with whatever value you eventually fire your own -Deferred with. If your Deferred is errbacked, their Deferred will be -errbacked with a CopiedFailure.

- - -

Constraints and RemoteInterfaces

- -

One major feature introduced by Foolscap (relative to oldpb) is the -serialization Constraint. -This lets you place limits on what kind of data you are willing to accept, -which enables safer distributed programming. Typically python uses duck -typing, wherein you usually just throw some arguments at the method and -see what happens. When you are less sure of the origin of those arguments, -you may want to be more circumspect. Enforcing type checking at the boundary -between your code and the outside world may make it safer to use duck typing -inside those boundaries. The type specifications also form a convenient -remote API reference you can publish for prospective clients of your -remotely-invokable service.

- -

In addition, these Constraints are enforced on each token as it arrives -over the wire. This means that you can calculate a (small) upper bound on how -much received data your program will store before it decides to hang up on -the violator, minimizing your exposure to DoS attacks that involve sending -random junk at you.

- -

There are three pieces you need to know about: Tokens, Constraints, and -RemoteInterfaces.

- -

Tokens

- -

The fundamental unit of serialization is the Banana Token. These are -thoroughly documented in the Banana -Specification, but what you need to know here is that each piece of -non-container data, like a string or a number, is represented by a single -token. Containers (like lists and dictionaries) are represented by a special -OPEN token, followed by tokens for everything that is in the container, -followed by the CLOSE token. Everything Banana does is in terms of these -nested OPEN/stuff/stuff/CLOSE sequences of tokens.

- -

Each token consists of a header, a type byte, and an optional body. The -header is always a base-128 number with a maximum of 64 digits, and the type -byte is always a single byte. The body (if present) has a length dictated by -the magnitude of the header.

- -

The length-first token format means that the receiving system never has to -accept more than 65 bytes before it knows the type and size of the token, at -which point it can make a decision about accepting or rejecting the rest of -it.

- -

Constraints

- -

The schema foolscap.schema module has a variety of Constraint classes that can be -applied to incoming data. Most of them correspond to typical Python types, -e.g. ListOf matches a list, -with a certain maximum length, and a child Constraint that gets -applied to the contents of the list. You can nest Constraints in -this way to describe the shape of the object graph that you are -willing to accept.

- -

At any given time, the receiving Banana protocol has single -Constraint object that it enforces against the inbound data -streamto be precise, each Unslicer on the -receive stack has a Constraint, and the idea is that all of them -get to pass judgement on the inbound token. A useful syntax to describe this -sort of thing is still being worked out..

- -

RemoteInterfaces

- -

The RemoteInterface is how you describe -your constraints. You can provide a constraint for each argument of each -method, as well as one for the return value. You can also specify addtional -flags on the methods. The convention (which is actually enforced by the code) -is to name RemoteInterface objects with an RI prefix, -like RIFoo.

- -

RemoteInterfaces are created and used a lot like the usual -zope.interface-style Interface. They look like -class definitions, inheriting from RemoteInterface. For each -method, the default value of each argument is used to create a -Constraint for that argument. Basic types (int, -str, bool) are converted into a -Constraint subclass (IntegerConstraint, StringConstraint, BooleanConstraint). You can also use -instances of other Constraint subclasses, like ListOf and DictOf. This Constraint will be -enforced against the value for the given argument. Unless you specify -otherwise, remote callers must match all the Constraints you -specify, all arguments listed in the RemoteInterface must be present, and no -arguments outside that list will be accepted.

- -

Note that, like zope.interface, these methods should not include -self in their argument list. This is because you are -documenting how other people invoke your methods. self -is an implementation detail. RemoteInterface will complain if -you forget.

- -

The methods in a RemoteInterface should return a -single value with the same format as the default arguments: either a basic -type (int, str, etc) or a Constraint -subclass. This Constraint is enforced on the return value of the -method. If you are calling a method in somebody else's process, the argument -constraints will be applied as a courtesy (be conservative in what you -send), and the return value constraint will be applied to prevent the -server from doing evil things to you. If you are running a method on behalf -of a remote client, the argument constraints will be enforced to protect -you, while the return value constraint will be applied as a -courtesy.

- -

Attempting to send a value that does not satisfy the Constraint will -result in a Violation exception -being raised.

- -

You can also specify methods by defining attributes of the same name in -the RemoteInterface object. Each attribute value should be an -instance of RemoteMethodSchemaalthough technically it can be any object which implements -the IRemoteMethodConstraint -interface. This approach is more flexible: there are some constraints -that are not easy to express with the default-argument syntax, and this is -the only way to set per-method flags. Note that all such method-defining -attributes must be set in the RemoteInterface body itself, -rather than being set on it after the fact (i.e. RIFoo.doBar = -stuff). This is required because the RemoteInterface -metaclass magic processes all of these attributes only once, immediately -after the RemoteInterface body has been evaluated.

- -

The RemoteInterface class has a name. Normally this is -the (short) classnameRIFoo.__class__.__name__, if -RemoteInterfaces were actually classes, which they're -not. You can override this -name by setting a special __remote_name__ attribute on the -RemoteInterface (again, in the body). This name is important -because it is externally visible: all RemoteReferences that -point at your Referenceables will remember the name of the -RemoteInterfaces it implements. This is what enables the -type-checking to be performed on both ends of the wire.

- -

In the future, this ought to default to the fully-qualified -classname (like package.module.RIFoo), so that two -RemoteInterfaces with the same name in different modules can co-exist. In the -current release, these two RemoteInterfaces will collide (and provoke an -import-time error message complaining about the duplicate name). As a result, -if you have such classes (e.g. foo.RIBar and -baz.RIBar), you must use __remote_name__ to -distinguish them (by naming one of them something other than -RIBar to avoid this error. - -Hopefully this will be improved in a future version, but it looks like a -difficult change to implement, so the standing recommendation is to use -__remote_name__ on all your RemoteInterfaces, and set it to a -suitably unique string (like a URI).

- -

Here's an example:

- -
-from foolscap import RemoteInterface, schema
-
-class RIMath(RemoteInterface):
-    __remote_name__ = "RIMath.using-pb.docs.foolscap.twistedmatrix.com"
-    def add(a=int, b=int):
-        return int
-    # declare it with an attribute instead of a function definition
-    subtract = schema.RemoteMethodSchema(a=int, b=int, _response=int)
-    def sum(args=schema.ListOf(int)):
-        return int
-
- - -

Using RemoteInterface

- -

To declare that your Referenceable responds to a particular -RemoteInterface, use the normal implements() -annotation:

- -
-class MathServer(foolscap.Referenceable):
-    implements(RIMath)
-
-    def remote_add(self, a, b):
-        return a+b
-    def remote_subtract(self, a, b):
-        return a-b
-    def remote_sum(self, args):
-        total = 0
-        for a in args: total += a
-        return total
-
- -

To enforce constraints everywhere, both sides will need to know about the -RemoteInterface, and both must know it by the same name. It is a -good idea to put the RemoteInterface in a common file that is -imported into the programs running on both sides. It is up to you to make -sure that both sides agree on the interface. Future versions of Foolscap may -implement some sort of checksum-verification or Interface-serialization as a -failsafe, but fundamentally the RemoteInterface that -you are using defines what your program is prepared to -handle. There is no difference between an old client accidentaly using a -different version of the RemoteInterface by mistake, and a malicious attacker -actively trying to confuse your code. The only promise that Foolscap can make -is that the constraints you provide in the RemoteInterface will be faithfully -applied to the incoming data stream, so that you don't need to do the type -checking yourself inside the method.

- -

When making a remote method call, you use the RemoteInterface -to identify the method instead of a string. This scopes the method name to -the RemoteInterface:

- -
-d = remote.callRemote(RIMath["add"], a=1, b=2)
-# or
-d = remote.callRemote(RIMath["add"], 1, 2)
-
- -

Pass-By-Copy

- -

You can pass (nearly) arbitrary instances over the wire. Foolscap knows -how to serialize all of Python's native data types already: numbers, strings, -unicode strings, booleans, lists, tuples, dictionaries, sets, and the None -object. You can teach it how to serialize instances of other types too. -Foolscap will not serialize (or deserialize) any class that you haven't -taught it about, both for security and because it refuses the temptation to -guess your intentions about how these unknown classes ought to be -serialized.

- -

The simplest possible way to pass things by copy is demonstrated in the -following code fragment:

- -
-from foolscap import Copyable, RemoteCopy
-
-class MyPassByCopy(Copyable, RemoteCopy):
-    typeToCopy = copytype = "MyPassByCopy"
-    def __init__(self):
-        # RemoteCopy subclasses may not accept any __init__ arguments
-        pass
-    def setCopyableState(self, state):
-        self.__dict__ = state
-
- -

If the code on both sides of the wire import this class, then any -instances of MyPassByCopy that are present in the arguments of a -remote method call (or returned as the result of a remote method call) will -be serialized and reconstituted into an equivalent instance on the other -side.

- -

For more complicated things to do with pass-by-copy, see the documentation -on Copyable. This explains the difference between -Copyable and RemoteCopy, how to control the -serialization and deserialization process, and how to arrange for -serialization of third-party classes that are not subclasses of -Copyable.

- - -

Third-party References

- -

Another new feature of Foolscap is the ability to send -RemoteReferences to third parties. The classic scenario for this -is illustrated by the three-party -Granovetter diagram. One party (Alice) has RemoteReferences to two other -objects named Bob and Carol. She wants to share her reference to Carol with -Bob, by including it in a message she sends to Bob (i.e. by using it as an -argument when she invokes one of Bob's remote methods). The Foolscap code for -doing this would look like:

- -
-bobref.callRemote("foo", intro=carolref)
-
- -

When Bob receives this message (i.e. when his remote_foo -method is invoked), he will discover that he's holding a fully-functional -RemoteReference to the object named Caroland if everyone involved is using authenticated Tubs, then -Foolscap offers a guarantee, in the cryptographic sense, that Bob will wind -up with a reference to the same object that Alice intended. The authenticated -PBURLs prevent DNS-spoofing and man-in-the-middle attacks.. He can -start using this RemoteReference right away:

- -
-class Bob(foolscap.Referenceable):
-    def remote_foo(self, intro):
-        self.carol = intro
-        carol.callRemote("howdy", msg="Pleased to meet you", you=intro)
-        return carol
-
- -

If Bob sends this RemoteReference back to Alice, her method -will see the same RemoteReference that she sent to Bob. In this -example, Bob sends the reference by returning it from the original -remote_foo method call, but he could almost as easily send it in -a separate method call.

- -
-class Alice(foolscap.Referenceable):
-    def start(self, carol):
-        self.carol = carol
-        d = self.bob.callRemote("foo", intro=carol)
-        d.addCallback(self.didFoo)
-    def didFoo(self, result):
-        assert result is self.carol  # this will be true
-
- -

Moreover, if Bob sends it back to Carol (completing the -three-party round trip), Carol will see it as her original -Referenceable.

- -
-class Carol(foolscap.Referenceable):
-    def remote_howdy(self, msg, you):
-        assert you is self  # this will be true
-
- -

In addition to this, in the four-party introduction sequence as used by -the Grant -Matcher Puzzle, when a Referenceable is sent to the same destination -through multiple paths, the recipient will receive the same -RemoteReference object from both sides.

- -

For a RemoteReference to be transferrable to third-parties in -this fashion, the original Referenceable must live in a Tub -which has a working listening port, and an established base URL. It is not -necessary for the Referenceable to have been published with -registerReference first: if it is sent over the wire before a -name has been associated with it, it will be registered under a new random -and unguessable name. The RemoteReference will contain the -resulting URL, enabling it to be sent to third parties.

- -

When this introduction is made, the receiving system must establish a -connection with the Tub that holds the original Referenceable, and acquire -its own RemoteReference. These steps must take place before the remote method -can be invoked, and other method calls might arrive before they do. All -subsequent method calls are queued until the one that involved the -introduction is performed. Foolscap guarantees (by default) that the messages -sent to a given Referenceable will be delivered in the same order. In the -future there may be options to relax this guarantee, in exchange for higher -performance, reduced memory consumption, multiple priority queues, limited -latency, or other features. There might even be an option to turn off -introductions altogether.

- -

Also note that enabling this capability means any of your communication -peers can make you create TCP connections to hosts and port numbers of their -choosing. The fact that those connections can only speak the Foolscap -protocol may reduce the security risk presented, but it still lets other -people be annoying.

- - - diff --git a/src/foolscap/foolscap/__init__.py b/src/foolscap/foolscap/__init__.py index e2243f42..ab58d559 100644 --- a/src/foolscap/foolscap/__init__.py +++ b/src/foolscap/foolscap/__init__.py @@ -1,6 +1,6 @@ """Foolscap""" -__version__ = "0.1.3" +__version__ = "0.1.4" # here are the primary entry points from foolscap.pb import Tub, UnauthenticatedTub, getRemoteURL_TCP diff --git a/src/foolscap/foolscap/call.py b/src/foolscap/foolscap/call.py index 8ed3af4b..2f1ccd6a 100644 --- a/src/foolscap/foolscap/call.py +++ b/src/foolscap/foolscap/call.py @@ -197,11 +197,14 @@ class InboundDelivery: class ArgumentUnslicer(slicer.ScopedUnslicer): methodSchema = None + debug = False def setConstraint(self, methodSchema): self.methodSchema = methodSchema def start(self, count): + if self.debug: + log.msg("%s.start: %s" % (self, count)) self.numargs = None self.args = [] self.kwargs = {} @@ -242,6 +245,11 @@ class ArgumentUnslicer(slicer.ScopedUnslicer): return unslicer def receiveChild(self, token, ready_deferred=None): + if self.debug: + log.msg("%s.receiveChild: %s %s %s %s %s args=%s kwargs=%s" % + (self, self.closed, self.num_unreferenceable_children, + self.num_unready_children, token, ready_deferred, + self.args, self.kwargs)) if self.numargs is None: # this token is the number of positional arguments assert isinstance(token, int) @@ -261,10 +269,14 @@ class ArgumentUnslicer(slicer.ScopedUnslicer): argpos = len(self.args) self.args.append(argvalue) if isinstance(argvalue, defer.Deferred): + # this may occur if the child is a gift which has not + # resolved yet. self.num_unreferenceable_children += 1 argvalue.addCallback(self.updateChild, argpos) argvalue.addErrback(self.explode) if ready_deferred: + if self.debug: + log.msg("%s.receiveChild got an unready posarg" % self) self.num_unready_children += 1 ready_deferred.addCallback(self.childReady) if len(self.args) < self.numargs: @@ -298,11 +310,13 @@ class ArgumentUnslicer(slicer.ScopedUnslicer): argvalue.addCallback(self.updateChild, self.argname) argvalue.addErrback(self.explode) if ready_deferred: + if self.debug: + log.msg("%s.receiveChild got an unready kwarg" % self) self.num_unready_children += 1 ready_deferred.addCallback(self.childReady) self.argname = None return - + def updateChild(self, obj, which): # one of our arguments has just now become referenceable. Normal # types can't trigger this (since the arguments to a method form a @@ -311,6 +325,9 @@ class ArgumentUnslicer(slicer.ScopedUnslicer): # RemoteReference, but for now all we get is a Deferred as a # placeholder. + if self.debug: + log.msg("%s.updateChild, [%s] became referenceable: %s" % + (self, which, obj)) if isinstance(which, int): self.args[which] = obj else: @@ -321,12 +338,22 @@ class ArgumentUnslicer(slicer.ScopedUnslicer): def childReady(self, obj): self.num_unready_children -= 1 + if self.debug: + log.msg("%s.childReady, now %d left" % + (self, self.num_unready_children)) + log.msg(" obj=%s, args=%s, kwargs=%s" % + (obj, self.args, self.kwargs)) self.checkComplete() return obj def checkComplete(self): # this is called each time one of our children gets updated or # becomes ready (like when a Gift is finally resolved) + if self.debug: + log.msg("%s.checkComplete: %s %s %s args=%s kwargs=%s" % + (self, self.closed, self.num_unreferenceable_children, + self.num_unready_children, self.args, self.kwargs)) + if not self.closed: return if self.num_unreferenceable_children: @@ -334,17 +361,25 @@ class ArgumentUnslicer(slicer.ScopedUnslicer): if self.num_unready_children: return # yup, we're done. Notify anyone who is still waiting + if self.debug: + log.msg(" we are ready") for d in self.watchers: eventually(d.callback, self) del self.watchers def receiveClose(self): + if self.debug: + log.msg("%s.receiveClose: %s %s %s" % + (self, self.closed, self.num_unreferenceable_children, + self.num_unready_children)) if (self.numargs is None or len(self.args) < self.numargs or self.argname is not None): raise BananaError("'arguments' sequence ended too early") self.closed = True self.watchers = [] + # we don't return a ready_deferred. Instead, the InboundDelivery + # object queries our isReady() method directly. return self, None def isReady(self): @@ -385,6 +420,8 @@ class ArgumentUnslicer(slicer.ScopedUnslicer): class CallUnslicer(slicer.ScopedUnslicer): + debug = False + def start(self, count): # start=0:reqID, 1:objID, 2:methodname, 3: arguments self.stage = 0 @@ -436,7 +473,9 @@ class CallUnslicer(slicer.ScopedUnslicer): def receiveChild(self, token, ready_deferred=None): assert not isinstance(token, defer.Deferred) assert ready_deferred is None - #print "CallUnslicer.receiveChild [s%d]" % self.stage, repr(token) + if self.debug: + log.msg("%s.receiveChild [s%d]: %s" % + (self, self.stage, repr(token))) if self.stage == 0: # reqID # we don't yet know which reqID to send any failure to diff --git a/src/foolscap/foolscap/pb.py b/src/foolscap/foolscap/pb.py index 9f7cfb9d..735f1dbc 100644 --- a/src/foolscap/foolscap/pb.py +++ b/src/foolscap/foolscap/pb.py @@ -6,7 +6,7 @@ from twisted.internet import defer, protocol from twisted.application import service, strports from twisted.python import log -from foolscap import ipb, base32, negotiate, broker, observer +from foolscap import ipb, base32, negotiate, broker, observer, eventual from foolscap.referenceable import SturdyRef from foolscap.tokens import PBError, BananaError from foolscap.reconnector import Reconnector @@ -242,6 +242,8 @@ class Tub(service.MultiService): self._activeConnectors = [] self._allConnectorsAreFinished = observer.OneShotObserverList() + self._pending_getReferences = [] # list of (d, furl) pairs + def setOption(self, name, value): if name == "logLocalFailures": # log (with log.err) any exceptions that occur during the @@ -376,6 +378,15 @@ class Tub(service.MultiService): if not self.running and not self._activeConnectors: self._allConnectorsAreFinished.fire(self) + def startService(self): + service.MultiService.startService(self) + for d,sturdy in self._pending_getReferences: + d1 = eventual.fireEventually(sturdy) + d1.addCallback(self.getReference) + d1.addBoth(lambda res, d=d: d.callback(res)) + del self._pending_getReferences + for rc in self.reconnectors: + eventual.eventually(rc.startConnecting, self) def _tubsAreNotRestartable(self): raise RuntimeError("Sorry, but Tubs cannot be restarted.") @@ -386,6 +397,7 @@ class Tub(service.MultiService): # note that once you stopService a Tub, I cannot be restarted. (at # least this code is not designed to make that possible.. it might be # doable in the future). + assert self.running self.startService = self._tubsAreNotRestartable self.getReference = self._tubHasBeenShutDown self.connectTo = self._tubHasBeenShutDown @@ -523,8 +535,6 @@ class Tub(service.MultiService): @return: a Deferred that fires with the RemoteReference """ - assert self.running - if isinstance(sturdyOrURL, SturdyRef): sturdy = sturdyOrURL else: @@ -538,12 +548,21 @@ class Tub(service.MultiService): "we cannot handle encrypted PB-URLs like %s" % sturdy.getURL()) return defer.fail(e) + + if not self.running: + # queue their request for service once the Tub actually starts + log.msg("Tub.getReference(%s) queued until Tub.startService called" + % sturdy) + d = defer.Deferred() + self._pending_getReferences.append((d, sturdy)) + return d + name = sturdy.name d = self.getBrokerForTubRef(sturdy.getTubRef()) d.addCallback(lambda b: b.getYourReferenceByName(name)) return d - def connectTo(self, sturdyOrURL, cb, *args, **kwargs): + def connectTo(self, _sturdyOrURL, _cb, *args, **kwargs): """Establish (and maintain) a connection to a given PBURL. I establish a connection to the PBURL and run a callback to inform @@ -582,8 +601,12 @@ class Tub(service.MultiService): rc.stopConnecting() # later """ - assert self.running - rc = Reconnector(self, sturdyOrURL, cb, *args, **kwargs) + rc = Reconnector(_sturdyOrURL, _cb, args, kwargs) + if self.running: + rc.startConnecting(self) + else: + log.msg("Tub.connectTo(%s) queued until Tub.startService called" + % _sturdyOrURL) self.reconnectors.append(rc) return rc diff --git a/src/foolscap/foolscap/reconnector.py b/src/foolscap/foolscap/reconnector.py index d22ab3e8..eace104e 100644 --- a/src/foolscap/foolscap/reconnector.py +++ b/src/foolscap/foolscap/reconnector.py @@ -42,17 +42,17 @@ class Reconnector: jitter = 0.11962656492 # molar Planck constant times c, Joule meter/mole verbose = False - def __init__(self, tub, url, cb, *args, **kwargs): - self._tub = tub + def __init__(self, url, cb, args, kwargs): self._url = url self._active = False self._observer = (cb, args, kwargs) self._delay = self.initialDelay self._retries = 0 self._timer = None - self.startConnecting() + self._tub = None - def startConnecting(self): + def startConnecting(self, tub): + self._tub = tub if self.verbose: log.msg("Reconnector starting for %s" % self._url) self._active = True @@ -65,7 +65,8 @@ class Reconnector: if self._timer: self._timer.cancel() self._timer = False - self._tub._removeReconnector(self) + if self._tub: + self._tub._removeReconnector(self) def _connect(self): d = self._tub.getReference(self._url) diff --git a/src/foolscap/foolscap/referenceable.py b/src/foolscap/foolscap/referenceable.py index 283705a7..d940429f 100644 --- a/src/foolscap/foolscap/referenceable.py +++ b/src/foolscap/foolscap/referenceable.py @@ -630,8 +630,18 @@ class TheirReferenceUnslicer(slicer.LeafUnslicer): d.addBoth(self.ackGift) # we return a Deferred that will fire with the RemoteReference when # it becomes available. The RemoteReference is not even referenceable - # until then. - return d,None + # until then. In addition, we provide a ready_deferred, since any + # mutable container which holds the gift will be referenceable early + # but the message delivery must still wait for the getReference to + # complete. See to it that we fire the object deferred before we fire + # the ready_deferred. + obj_deferred, ready_deferred = defer.Deferred(), defer.Deferred() + def _ready(rref): + obj_deferred.callback(rref) + ready_deferred.callback(rref) + d.addCallback(_ready) + + return obj_deferred, ready_deferred def ackGift(self, rref): rb = self.broker.remote_broker @@ -727,26 +737,6 @@ class SturdyRef(Copyable, RemoteCopy): cmp(self.__class__, them.__class__) or cmp(self._distinguishers(), them._distinguishers())) - def asLiveRef(self): - """Return an object that can be sent over the wire and unserialized - as a live RemoteReference on the far end. Use this when you have a - SturdyRef and want to give someone a reference to its target, but - when you haven't bothered to acquire your own live reference to it.""" - - return _AsLiveRef(self) - -class _AsLiveRef: - implements(ipb.ISlicer) - - def __init__(self, sturdy): - self.target = sturdy - - def slice(self, streamable, banana): - yield 'their-reference' - yield giftID - yield self.target.getURL() - yield [] # interfacenames - class TubRef: """This is a little helper class which provides a comparable identifier diff --git a/src/foolscap/foolscap/remoteinterface.py b/src/foolscap/foolscap/remoteinterface.py index 46df3616..f3fbdc09 100644 --- a/src/foolscap/foolscap/remoteinterface.py +++ b/src/foolscap/foolscap/remoteinterface.py @@ -105,7 +105,8 @@ def getRemoteInterface(obj): if isinstance(i, RemoteInterfaceClass): if i not in ilist: ilist.append(i) - assert len(ilist) <= 1, "don't use multiple RemoteInterfaces! %s" % (obj,) + assert len(ilist) <= 1, ("don't use multiple RemoteInterfaces! %s uses %s" + % (obj, ilist)) if ilist: return ilist[0] return None @@ -199,7 +200,8 @@ class RemoteMethodSchema: raise InvalidRemoteInterface(why) if not names: typeList = [] - if len(names) != len(typeList): + # 'def foo(oops)' results in typeList==None + if typeList is None or len(names) != len(typeList): # TODO: relax this, use schema=Any for the args that don't have # default values. This would make: # def foo(a, b=int): return None @@ -361,9 +363,18 @@ class RemoteInterfaceConstraint(OpenerConstraint): associated with a remote Referenceable that implements the given RemoteInterface. If 'interface' is None, just assert that it is a RemoteReference at all. + + On the inbound side, this will only accept a suitably-implementing + RemoteReference, or a gift that resolves to such a RemoteReference. On + the outbound side, this will accept either a Referenceable or a + RemoteReference (which might be a your-reference or a their-reference). + + Sending your-references will result in the recipient getting a local + Referenceable, which will not pass the constraint. TODO: think about if + we want this behavior or not. """ - opentypes = [("my-reference",)] - # TODO: accept their-references too + + opentypes = [("my-reference",), ("their-reference",)] name = "RemoteInterfaceConstraint" def __init__(self, interface): @@ -387,7 +398,17 @@ class RemoteInterfaceConstraint(OpenerConstraint): % (obj, self.interface)) else: # this ought to be a Referenceable which implements the desired - # interface + # interface. Or, it might be a RemoteReference which points to + # one. + if ipb.IRemoteReference.providedBy(obj): + # it's a RemoteReference + if not self.interface: + return + iface = obj.tracker.interface + if not iface or iface != self.interface: + raise Violation("'%s' does not provide RemoteInterface %s" + % (obj, self.interface)) + return if not ipb.IReferenceable.providedBy(obj): # TODO: maybe distinguish between OnlyReferenceable and # Referenceable? which is more useful here? diff --git a/src/foolscap/foolscap/slicers/allslicers.py b/src/foolscap/foolscap/slicers/allslicers.py index aa05f177..afb3b470 100644 --- a/src/foolscap/foolscap/slicers/allslicers.py +++ b/src/foolscap/foolscap/slicers/allslicers.py @@ -10,7 +10,7 @@ from foolscap.slicers.unicode import UnicodeSlicer, UnicodeUnslicer from foolscap.slicers.list import ListSlicer, ListUnslicer from foolscap.slicers.tuple import TupleSlicer, TupleUnslicer from foolscap.slicers.set import SetSlicer, SetUnslicer -from foolscap.slicers.set import ImmutableSetSlicer, ImmutableSetUnslicer +from foolscap.slicers.set import FrozenSetSlicer, FrozenSetUnslicer #from foolscap.slicers.set import BuiltinSetSlicer from foolscap.slicers.dict import DictSlicer, DictUnslicer, OrderedDictSlicer from foolscap.slicers.vocab import ReplaceVocabSlicer, ReplaceVocabUnslicer @@ -26,7 +26,7 @@ unused = [ ListSlicer, ListUnslicer, TupleSlicer, TupleUnslicer, SetSlicer, SetUnslicer, - ImmutableSetSlicer, ImmutableSetUnslicer, + FrozenSetSlicer, FrozenSetUnslicer, #from foolscap.slicers.set import BuiltinSetSlicer DictSlicer, DictUnslicer, OrderedDictSlicer, ReplaceVocabSlicer, ReplaceVocabUnslicer, diff --git a/src/foolscap/foolscap/slicers/dict.py b/src/foolscap/foolscap/slicers/dict.py index 6cd30c29..23a69ca3 100644 --- a/src/foolscap/foolscap/slicers/dict.py +++ b/src/foolscap/foolscap/slicers/dict.py @@ -1,7 +1,7 @@ # -*- test-case-name: foolscap.test.test_banana -*- from twisted.python import log -from twisted.internet.defer import Deferred +from twisted.internet.defer import Deferred, DeferredList from foolscap.tokens import Violation, BananaError from foolscap.slicer import BaseSlicer, BaseUnslicer from foolscap.constraint import OpenerConstraint, Any, UnboundedSchema, IConstraint @@ -35,6 +35,7 @@ class DictUnslicer(BaseUnslicer): self.d = {} self.protocol.setObject(count, self.d) self.key = None + self._ready_deferreds = [] def checkToken(self, typebyte, size): if self.maxKeys != None: @@ -72,8 +73,8 @@ class DictUnslicer(BaseUnslicer): self.d[key] = value def receiveChild(self, obj, ready_deferred=None): - assert not isinstance(obj, Deferred) - assert ready_deferred is None + if ready_deferred: + self._ready_deferreds.append(ready_deferred) if self.gettingKey: self.receiveKey(obj) else: @@ -102,7 +103,10 @@ class DictUnslicer(BaseUnslicer): self.d[self.key] = value # placeholder def receiveClose(self): - return self.d, None + ready_deferred = None + if self._ready_deferreds: + ready_deferred = DeferredList(self._ready_deferreds) + return self.d, ready_deferred def describe(self): if self.gettingKey: diff --git a/src/foolscap/foolscap/slicers/list.py b/src/foolscap/foolscap/slicers/list.py index 4e8a0db2..70ad55f8 100644 --- a/src/foolscap/foolscap/slicers/list.py +++ b/src/foolscap/foolscap/slicers/list.py @@ -1,7 +1,7 @@ # -*- test-case-name: foolscap.test.test_banana -*- from twisted.python import log -from twisted.internet.defer import Deferred +from twisted.internet.defer import Deferred, DeferredList from foolscap.tokens import Violation from foolscap.slicer import BaseSlicer, BaseUnslicer from foolscap.constraint import OpenerConstraint, Any, UnboundedSchema, IConstraint @@ -35,8 +35,9 @@ class ListUnslicer(BaseUnslicer): self.list = [] self.count = count if self.debug: - print "%s[%d].start with %s" % (self, self.count, self.list) + log.msg("%s[%d].start with %s" % (self, self.count, self.list)) self.protocol.setObject(count, self.list) + self._ready_deferreds = [] def checkToken(self, typebyte, size): if self.maxLength != None and len(self.list) >= self.maxLength: @@ -65,15 +66,16 @@ class ListUnslicer(BaseUnslicer): def update(self, obj, index): # obj has already passed typechecking if self.debug: - print "%s[%d].update: [%d]=%s" % (self, self.count, index, obj) + log.msg("%s[%d].update: [%d]=%s" % (self, self.count, index, obj)) assert isinstance(index, int) self.list[index] = obj return obj def receiveChild(self, obj, ready_deferred=None): - assert ready_deferred is None + if ready_deferred: + self._ready_deferreds.append(ready_deferred) if self.debug: - print "%s[%d].receiveChild(%s)" % (self, self.count, obj) + log.msg("%s[%d].receiveChild(%s)" % (self, self.count, obj)) # obj could be a primitive type, a Deferred, or a complex type like # those returned from an InstanceUnslicer. However, the individual # object has already been through the schema validation process. The @@ -86,10 +88,12 @@ class ListUnslicer(BaseUnslicer): raise Violation("the list is full") if isinstance(obj, Deferred): if self.debug: - print " adding my update[%d] to %s" % (len(self.list), obj) + log.msg(" adding my update[%d] to %s" % (len(self.list), obj)) obj.addCallback(self.update, len(self.list)) obj.addErrback(self.printErr) - self.list.append("placeholder") + placeholder = "list placeholder for arg[%d], rd=%s" % \ + (len(self.list), ready_deferred) + self.list.append(placeholder) else: self.list.append(obj) @@ -99,7 +103,10 @@ class ListUnslicer(BaseUnslicer): log.err(why) def receiveClose(self): - return self.list, None + ready_deferred = None + if self._ready_deferreds: + ready_deferred = DeferredList(self._ready_deferreds) + return self.list, ready_deferred def describe(self): return "[%d]" % len(self.list) diff --git a/src/foolscap/foolscap/slicers/set.py b/src/foolscap/foolscap/slicers/set.py index 206507f2..f2afa09c 100644 --- a/src/foolscap/foolscap/slicers/set.py +++ b/src/foolscap/foolscap/slicers/set.py @@ -1,7 +1,11 @@ # -*- test-case-name: foolscap.test.test_banana -*- import sets -from foolscap.slicers.list import ListSlicer, ListUnslicer +from twisted.internet import defer +from twisted.python import log +from foolscap.slicers.list import ListSlicer +from foolscap.slicers.tuple import TupleUnslicer +from foolscap.slicer import BaseUnslicer from foolscap.tokens import Violation from foolscap.constraint import OpenerConstraint, UnboundedSchema, Any, \ IConstraint @@ -9,34 +13,41 @@ from foolscap.constraint import OpenerConstraint, UnboundedSchema, Any, \ class SetSlicer(ListSlicer): opentype = ("set",) trackReferences = True - slices = sets.Set + slices = set def sliceBody(self, streamable, banana): for i in self.obj: yield i -class ImmutableSetSlicer(SetSlicer): +class FrozenSetSlicer(SetSlicer): opentype = ("immutable-set",) trackReferences = False + slices = frozenset + +# python2.4 has a builtin 'set' type, which is mutable, and we require +# python2.4 or newer. Code which was written to be compatible with python2.3, +# however, may use the 'sets' module. We will serialize old sets.Set and +# sets.ImmutableSet the same as we serialize new set and frozenset. +# Unfortunately this means that these objects will be deserialized as modern +# 'set' and 'frozenset' objects, which are not entirely compatible. Therefore +# code that is compatible with python2.3 might not work with foolscap. + +class OldSetSlicer(SetSlicer): + slices = sets.Set +class OldImmutableSetSlicer(FrozenSetSlicer): slices = sets.ImmutableSet -have_builtin_set = False -try: - set - # python2.4 has a builtin 'set' type, which is mutable - have_builtin_set = True - class BuiltinSetSlicer(SetSlicer): - slices = set - class BuiltinFrozenSetSlicer(ImmutableSetSlicer): - slices = frozenset -except NameError: - # oh well, I guess we don't have 'set' +class _Placeholder: pass -class SetUnslicer(ListUnslicer): +class SetUnslicer(BaseUnslicer): + # this is a lot like a list, but sufficiently different to make it not + # worth subclassing opentype = ("set",) - def receiveClose(self): - return sets.Set(self.list), None + + debug = False + maxLength = None + itemConstraint = None def setConstraint(self, constraint): if isinstance(constraint, Any): @@ -45,10 +56,101 @@ class SetUnslicer(ListUnslicer): self.maxLength = constraint.maxLength self.itemConstraint = constraint.constraint -class ImmutableSetUnslicer(SetUnslicer): + def start(self, count): + #self.opener = foo # could replace it if we wanted to + self.set = set() + self.count = count + if self.debug: + log.msg("%s[%d].start with %s" % (self, self.count, self.set)) + self.protocol.setObject(count, self.set) + self._ready_deferreds = [] + + def checkToken(self, typebyte, size): + if self.maxLength != None and len(self.set) >= self.maxLength: + # list is full, no more tokens accepted + # this is hit if the max+1 item is a primitive type + raise Violation("the set is full") + if self.itemConstraint: + self.itemConstraint.checkToken(typebyte, size) + + def doOpen(self, opentype): + # decide whether the given object type is acceptable here. Raise a + # Violation exception if not, otherwise give it to our opener (which + # will normally be the RootUnslicer). Apply a constraint to the new + # unslicer. + if self.maxLength != None and len(self.set) >= self.maxLength: + # this is hit if the max+1 item is a non-primitive type + raise Violation("the set is full") + if self.itemConstraint: + self.itemConstraint.checkOpentype(opentype) + unslicer = self.open(opentype) + if unslicer: + if self.itemConstraint: + unslicer.setConstraint(self.itemConstraint) + return unslicer + + def update(self, obj, placeholder): + # obj has already passed typechecking + if self.debug: + log.msg("%s[%d].update: [%s]=%s" % (self, self.count, + placeholder, obj)) + self.set.remove(placeholder) + self.set.add(obj) + return obj + + def receiveChild(self, obj, ready_deferred=None): + if ready_deferred: + self._ready_deferreds.append(ready_deferred) + if self.debug: + log.msg("%s[%d].receiveChild(%s)" % (self, self.count, obj)) + # obj could be a primitive type, a Deferred, or a complex type like + # those returned from an InstanceUnslicer. However, the individual + # object has already been through the schema validation process. The + # only remaining question is whether the larger schema will accept + # it. + if self.maxLength != None and len(self.set) >= self.maxLength: + # this is redundant + # (if it were a non-primitive one, it would be caught in doOpen) + # (if it were a primitive one, it would be caught in checkToken) + raise Violation("the set is full") + if isinstance(obj, defer.Deferred): + if self.debug: + log.msg(" adding my update[%d] to %s" % (len(self.set), obj)) + # note: the placeholder isn't strictly necessary, but it will + # help debugging to see a _Placeholder sitting in the set when it + # shouldn't rather than seeing a set that is smaller than it + # ought to be. If a remote method ever sees a _Placeholder, then + # something inside Foolscap has broken. + placeholder = _Placeholder() + obj.addCallback(self.update, placeholder) + obj.addErrback(self.printErr) + self.set.add(placeholder) + else: + self.set.add(obj) + + def printErr(self, why): + print "ERR!" + print why.getBriefTraceback() + log.err(why) + + def receiveClose(self): + ready_deferred = None + if self._ready_deferreds: + ready_deferred = defer.DeferredList(self._ready_deferreds) + return self.set, ready_deferred + +class FrozenSetUnslicer(TupleUnslicer): opentype = ("immutable-set",) + def receiveClose(self): - return sets.ImmutableSet(self.list), None + obj_or_deferred, ready_deferred = TupleUnslicer.receiveClose(self) + if isinstance(obj_or_deferred, defer.Deferred): + def _convert(the_tuple): + return frozenset(the_tuple) + obj_or_deferred.addCallback(_convert) + else: + obj_or_deferred = frozenset(obj_or_deferred) + return obj_or_deferred, ready_deferred class SetConstraint(OpenerConstraint): @@ -64,12 +166,8 @@ class SetConstraint(OpenerConstraint): opentypes = [("set",), ("immutable-set",)] name = "SetConstraint" - if have_builtin_set: - mutable_set_types = (set, sets.Set) - immutable_set_types = (frozenset, sets.ImmutableSet) - else: - mutable_set_types = (sets.Set,) - immutable_set_types = (sets.ImmutableSet,) + mutable_set_types = (set, sets.Set) + immutable_set_types = (frozenset, sets.ImmutableSet) all_set_types = mutable_set_types + immutable_set_types def __init__(self, constraint, maxLength=30, mutable=None): diff --git a/src/foolscap/foolscap/slicers/tuple.py b/src/foolscap/foolscap/slicers/tuple.py index f7ea5657..6469f822 100644 --- a/src/foolscap/foolscap/slicers/tuple.py +++ b/src/foolscap/foolscap/slicers/tuple.py @@ -1,6 +1,6 @@ # -*- test-case-name: foolscap.test.test_banana -*- -from twisted.internet.defer import Deferred +from twisted.internet.defer import Deferred, DeferredList from foolscap.tokens import Violation from foolscap.slicer import BaseUnslicer from foolscap.slicers.list import ListSlicer @@ -34,6 +34,7 @@ class TupleUnslicer(BaseUnslicer): self.finished = False self.deferred = Deferred() self.protocol.setObject(count, self.deferred) + self._ready_deferreds = [] def checkToken(self, typebyte, size): if self.constraints == None: @@ -64,7 +65,8 @@ class TupleUnslicer(BaseUnslicer): return obj def receiveChild(self, obj, ready_deferred=None): - assert ready_deferred is None + if ready_deferred: + self._ready_deferreds.append(ready_deferred) if isinstance(obj, Deferred): obj.addCallback(self.update, len(self.list)) obj.addErrback(self.explode) @@ -81,20 +83,39 @@ class TupleUnslicer(BaseUnslicer): # not finished yet, we'll fire our Deferred when we are if self.debug: print " not finished yet" - return self.deferred, None + return + # list is now complete. We can finish. + return self.complete() + + def complete(self): + ready_deferred = None + if self._ready_deferreds: + ready_deferred = DeferredList(self._ready_deferreds) + t = tuple(self.list) if self.debug: print " finished! tuple:%s{%s}" % (t, id(t)) self.protocol.setObject(self.count, t) self.deferred.callback(t) - return t, None + return t, ready_deferred def receiveClose(self): if self.debug: print "%s[%d].receiveClose" % (self, self.count) self.finished = 1 - return self.checkComplete() + + if self.num_unreferenceable_children: + # not finished yet, we'll fire our Deferred when we are + if self.debug: + print " not finished yet" + ready_deferred = None + if self._ready_deferreds: + ready_deferred = DeferredList(self._ready_deferreds) + return self.deferred, ready_deferred + + # the list is already complete + return self.complete() def describe(self): return "[%d]" % len(self.list) diff --git a/src/foolscap/foolscap/test/test_banana.py b/src/foolscap/foolscap/test/test_banana.py index 5951cafe..2c95c3f7 100644 --- a/src/foolscap/foolscap/test/test_banana.py +++ b/src/foolscap/foolscap/test/test_banana.py @@ -1597,18 +1597,13 @@ class ThereAndBackAgain(TestBananaMixin, unittest.TestCase): def test_tuple(self): return self.looptest((1,2)) def test_set(self): - d = self.looptest(sets.Set([1,2])) - d.addCallback(lambda res: self.looptest(sets.ImmutableSet([1,2]))) - # verify the python2.4 builtin 'set' type, which is mutable - try: - set - have_set = True - except NameError: - have_set = False - if have_set: - # we serialize builtin 'set' as a regular mutable sets.Set - d.addCallback(lambda res: - self.looptest(set([1,2]), sets.Set([1,2]))) + d = self.looptest(set([1,2])) + d.addCallback(lambda res: self.looptest(frozenset([1,2]))) + # and verify that old sets turn into modern ones, which is + # unfortunate but at least consistent + d.addCallback(lambda res: self.looptest(sets.Set([1,2]), set([1,2]))) + d.addCallback(lambda res: self.looptest(sets.ImmutableSet([1,2]), + frozenset([1,2]))) return d def test_bool(self): @@ -1709,15 +1704,71 @@ class ThereAndBackAgain(TestBananaMixin, unittest.TestCase): self.assertIdentical(z[0]['list'], z[1]) self.assertIdentical(z[0]['list'][0], z) - def testMoreReferences(self): + def test_cycles_1(self): + # a list that contains a tuple that can't be referenced yet a = [] - t = (a,) - t2 = (t,) + t1 = (a,) + t2 = (t1,) a.append(t2) - d = self.loop(t) + d = self.loop(t1) d.addCallback(lambda z: self.assertIdentical(z[0][0][0], z)) return d + def test_cycles_2(self): + # a dict that contains a tuple that can't be referenced yet. + a = {} + t1 = (a,) + t2 = (t1,) + a['foo'] = t2 + d = self.loop(t1) + d.addCallback(lambda z: self.assertIdentical(z[0]['foo'][0], z)) + return d + + def test_cycles_3(self): + # sets seem to be transitively immutable: any mutable contents would + # be unhashable, and sets can only contain hashable objects. + # Therefore sets cannot participate in cycles the way that tuples + # can. + + # a set that contains a tuple that can't be referenced yet. You can't + # actually create this in python, because you can only create a set + # out of hashable objects, and sets aren't hashable, and a tuple that + # contains a set is not hashable. + a = set() + t1 = (a,) + t2 = (t1,) + a.add(t2) + d = self.loop(t1) + d.addCallback(lambda z: self.assertIdentical(list(z[0])[0][0], z)) + + # a list that contains a frozenset that can't be referenced yet + a = [] + t1 = frozenset([a]) + t2 = frozenset([t1]) + a.append(t2) + d = self.loop(t1) + d.addCallback(lambda z: + self.assertIdentical(list(list(z)[0][0])[0], z)) + + # a dict that contains a frozenset that can't be referenced yet. + a = {} + t1 = frozenset([a]) + t2 = frozenset([t1]) + a['foo'] = t2 + d = self.loop(t1) + d.addCallback(lambda z: + self.assertIdentical(list(list(z)[0]['foo'])[0], z)) + + # a set that contains a frozenset that can't be referenced yet. + a = set() + t1 = frozenset([a]) + t2 = frozenset([t1]) + a.add(t2) + d = self.loop(t1) + d.addCallback(lambda z: self.assertIdentical(list(list(list(z)[0])[0])[0], z)) + return d + del test_cycles_3 + class VocabTest1(unittest.TestCase): diff --git a/src/foolscap/foolscap/test/test_call.py b/src/foolscap/foolscap/test/test_call.py index 04dc4abc..79e19beb 100644 --- a/src/foolscap/foolscap/test/test_call.py +++ b/src/foolscap/foolscap/test/test_call.py @@ -9,7 +9,6 @@ if False: log.startLogging(sys.stderr) from twisted.python import failure, log -from twisted.internet import reactor, defer from twisted.trial import unittest from twisted.internet.main import CONNECTION_LOST @@ -36,7 +35,6 @@ class TestCall(TargetMixin, unittest.TestCase): d.addCallback(lambda res: self.failUnlessEqual(target.calls, [(1,2)])) d.addCallback(self._testCall1_1, rr) return d - testCall1.timeout = 3 def _testCall1_1(self, res, rr): # the caller still holds the RemoteReference self.failUnless(self.callingBroker.yourReferenceByCLID.has_key(1)) @@ -46,10 +44,14 @@ class TestCall(TargetMixin, unittest.TestCase): # the targetBroker so *they* can forget about it. del rr # this fires a DecRef gc.collect() # make sure + # we need to give it a moment to deliver the DecRef message and act - # on it - d = defer.Deferred() - reactor.callLater(0.1, d.callback, None) + # on it. Poll until the caller has received it. + def _check(): + if self.callingBroker.yourReferenceByCLID.has_key(1): + return False + return True + d = self.poll(_check) d.addCallback(self._testCall1_2) return d def _testCall1_2(self, res): diff --git a/src/foolscap/foolscap/test/test_gifts.py b/src/foolscap/foolscap/test/test_gifts.py index 3775e657..db6dfd78 100644 --- a/src/foolscap/foolscap/test/test_gifts.py +++ b/src/foolscap/foolscap/test/test_gifts.py @@ -1,10 +1,11 @@ +from zope.interface import implements from twisted.trial import unittest from twisted.internet import defer from twisted.internet.error import ConnectionDone, ConnectionLost -from foolscap import Tub, UnauthenticatedTub +from foolscap import Tub, UnauthenticatedTub, RemoteInterface, Referenceable from foolscap.referenceable import RemoteReference -from foolscap.test.common import HelperTarget +from foolscap.test.common import HelperTarget, RIHelper from foolscap.eventual import flushEventualQueue crypto_available = False @@ -24,6 +25,19 @@ def ignoreConnectionDone(f): f.trap(ConnectionDone, ConnectionLost) return None +class RIConstrainedHelper(RemoteInterface): + def set(obj=RIHelper): return None + + +class ConstrainedHelper(Referenceable): + implements(RIConstrainedHelper) + + def __init__(self, name="unnamed"): + self.name = name + + def remote_set(self, obj): + self.obj = obj + class Gifts(unittest.TestCase): # Here we test the three-party introduction process as depicted in the # classic Granovetter diagram. Alice has a reference to Bob and another @@ -52,9 +66,15 @@ class Gifts(unittest.TestCase): self.bob_url = self.tubB.registerReference(self.bob) self.carol = HelperTarget("carol") self.carol_url = self.tubC.registerReference(self.carol) - self.cindy = HelperTarget("cindy") # cindy is Carol's little sister. She doesn't have a phone, but # Carol might talk about her anyway. + self.cindy = HelperTarget("cindy") + # more sisters. Alice knows them, and she introduces Bob to them. + self.charlene = HelperTarget("charlene") + self.christine = HelperTarget("christine") + self.clarisse = HelperTarget("clarisse") + self.colette = HelperTarget("colette") + self.courtney = HelperTarget("courtney") def createInitialReferences(self): # we must start by giving Alice a reference to both Bob and Carol. @@ -73,6 +93,46 @@ class Gifts(unittest.TestCase): d.addCallback(_aliceGotCarol) return d + def createMoreReferences(self): + # give Alice references to Carol's sisters + dl = [] + + url = self.tubC.registerReference(self.charlene) + d = self.tubA.getReference(url) + def _got_charlene(rref): + self.acharlene = rref + d.addCallback(_got_charlene) + dl.append(d) + + url = self.tubC.registerReference(self.christine) + d = self.tubA.getReference(url) + def _got_christine(rref): + self.achristine = rref + d.addCallback(_got_christine) + dl.append(d) + + url = self.tubC.registerReference(self.clarisse) + d = self.tubA.getReference(url) + def _got_clarisse(rref): + self.aclarisse = rref + d.addCallback(_got_clarisse) + dl.append(d) + + url = self.tubC.registerReference(self.colette) + d = self.tubA.getReference(url) + def _got_colette(rref): + self.acolette = rref + d.addCallback(_got_colette) + dl.append(d) + + url = self.tubC.registerReference(self.courtney) + d = self.tubA.getReference(url) + def _got_courtney(rref): + self.acourtney = rref + d.addCallback(_got_courtney) + dl.append(d) + return defer.DeferredList(dl) + def testGift(self): #defer.setDebugging(True) self.createCharacters() @@ -189,6 +249,76 @@ class Gifts(unittest.TestCase): d.addCallback(_checkBob) return d + def testContainers(self): + self.createCharacters() + self.bob.calls = [] + d = self.createInitialReferences() + d.addCallback(lambda res: self.createMoreReferences()) + def _introduce(res): + # we send several messages to Bob, each of which has a container + # with a gift inside it. This exercises the ready_deferred + # handling inside containers. + dl = [] + cr = self.abob.callRemote + dl.append(cr("append", set([self.acharlene]))) + dl.append(cr("append", frozenset([self.achristine]))) + dl.append(cr("append", [self.aclarisse])) + dl.append(cr("append", obj=(self.acolette,))) + dl.append(cr("append", {'a': self.acourtney})) + # TODO: pass a gift as an attribute of a Copyable + return defer.DeferredList(dl) + d.addCallback(_introduce) + def _checkBob(res): + # this runs after all three messages have been acked by Bob + self.failUnlessEqual(len(self.bob.calls), 5) + + bcharlene = self.bob.calls.pop(0) + self.failUnless(isinstance(bcharlene, set)) + self.failUnlessEqual(len(bcharlene), 1) + self.failUnless(isinstance(list(bcharlene)[0], RemoteReference)) + + bchristine = self.bob.calls.pop(0) + self.failUnless(isinstance(bchristine, frozenset)) + self.failUnlessEqual(len(bchristine), 1) + self.failUnless(isinstance(list(bchristine)[0], RemoteReference)) + + bclarisse = self.bob.calls.pop(0) + self.failUnless(isinstance(bclarisse, list)) + self.failUnlessEqual(len(bclarisse), 1) + self.failUnless(isinstance(bclarisse[0], RemoteReference)) + + bcolette = self.bob.calls.pop(0) + self.failUnless(isinstance(bcolette, tuple)) + self.failUnlessEqual(len(bcolette), 1) + self.failUnless(isinstance(bcolette[0], RemoteReference)) + + bcourtney = self.bob.calls.pop(0) + self.failUnless(isinstance(bcourtney, dict)) + self.failUnlessEqual(len(bcourtney), 1) + self.failUnless(isinstance(bcourtney['a'], RemoteReference)) + + d.addCallback(_checkBob) + return d + + def create_constrained_characters(self): + self.alice = HelperTarget("alice") + self.bob = ConstrainedHelper("bob") + self.bob_url = self.tubB.registerReference(self.bob) + self.carol = HelperTarget("carol") + self.carol_url = self.tubC.registerReference(self.carol) + + def test_constraint(self): + self.create_constrained_characters() + self.bob.calls = [] + d = self.createInitialReferences() + def _introduce(res): + return self.abob.callRemote("set", self.acarol) + d.addCallback(_introduce) + def _checkBob(res): + self.failUnless(isinstance(self.bob.obj, RemoteReference)) + d.addCallback(_checkBob) + return d + # this was used to alice's reference to carol (self.acarol) appeared in # alice's gift table at the right time, to make sure that the # RemoteReference is kept alive while the gift is in transit. The whole diff --git a/src/foolscap/foolscap/test/test_schema.py b/src/foolscap/foolscap/test/test_schema.py index 671b7a73..0c052e78 100644 --- a/src/foolscap/foolscap/test/test_schema.py +++ b/src/foolscap/foolscap/test/test_schema.py @@ -2,7 +2,7 @@ import sets, re from twisted.trial import unittest from foolscap import schema, copyable -from foolscap.tokens import Violation +from foolscap.tokens import Violation, InvalidRemoteInterface from foolscap.constraint import IConstraint from foolscap.remoteinterface import RemoteMethodSchema, \ RemoteInterfaceConstraint, LocalInterfaceConstraint @@ -474,6 +474,13 @@ class Arguments(unittest.TestCase): self.failUnlessRaises(schema.Violation, r.checkResults, 12, False) + def test_bad_arguments(self): + def foo(nodefault): return str + self.failUnlessRaises(InvalidRemoteInterface, + RemoteMethodSchema, method=foo) + def bar(nodefault, a=int): return str + self.failUnlessRaises(InvalidRemoteInterface, + RemoteMethodSchema, method=bar) class Interfaces(unittest.TestCase): @@ -520,9 +527,11 @@ class Interfaces(unittest.TestCase): interfaceName = common.RIHelper.__remote_name__ tracker = RemoteReferenceTracker(parent, clid, url, interfaceName) rr = RemoteReference(tracker) + c1 = RemoteInterfaceConstraint(common.RIHelper) - c2 = RemoteInterfaceConstraint(common.RIMyTarget) self.check_inbound(rr, c1) - self.violates_outbound(rr, c1) + self.check_outbound(rr, c1) # gift + + c2 = RemoteInterfaceConstraint(common.RIMyTarget) self.violates_inbound(rr, c2) self.violates_outbound(rr, c2) diff --git a/src/foolscap/foolscap/test/test_tub.py b/src/foolscap/foolscap/test/test_tub.py index 4b0d31b1..8f36a524 100644 --- a/src/foolscap/foolscap/test/test_tub.py +++ b/src/foolscap/foolscap/test/test_tub.py @@ -2,6 +2,7 @@ import os.path from twisted.trial import unittest +from twisted.internet import defer crypto_available = False try: @@ -10,7 +11,16 @@ try: except ImportError: pass -from foolscap import Tub +from foolscap import Tub, UnauthenticatedTub +from foolscap.referenceable import RemoteReference +from foolscap.eventual import eventually, flushEventualQueue +from foolscap.test.common import HelperTarget, TargetMixin + +# we use authenticated tubs if possible. If crypto is not available, fall +# back to unauthenticated ones +GoodEnoughTub = UnauthenticatedTub +if crypto_available: + GoodEnoughTub = Tub class TestCertFile(unittest.TestCase): def test_generate(self): @@ -38,3 +48,72 @@ class TestCertFile(unittest.TestCase): if not crypto_available: del TestCertFile + +class QueuedStartup(TargetMixin, unittest.TestCase): + # calling getReference and connectTo before the Tub has started should + # put off network activity until the Tub is started. + + def setUp(self): + TargetMixin.setUp(self) + self.tubB = GoodEnoughTub() + self.services = [self.tubB] + for s in self.services: + s.startService() + l = s.listenOn("tcp:0:interface=127.0.0.1") + s.setLocation("127.0.0.1:%d" % l.getPortnum()) + + self.barry = HelperTarget("barry") + self.barry_url = self.tubB.registerReference(self.barry) + + self.bill = HelperTarget("bill") + self.bill_url = self.tubB.registerReference(self.bill) + + self.bob = HelperTarget("bob") + self.bob_url = self.tubB.registerReference(self.bob) + + def tearDown(self): + d = TargetMixin.tearDown(self) + def _more(res): + return defer.DeferredList([s.stopService() for s in self.services]) + d.addCallback(_more) + d.addCallback(flushEventualQueue) + return d + + def test_queued_getref(self): + t1 = GoodEnoughTub() + d1 = t1.getReference(self.barry_url) + d2 = t1.getReference(self.bill_url) + def _check(res): + ((barry_success, barry_rref), + (bill_success, bill_rref)) = res + self.failUnless(barry_success) + self.failUnless(bill_success) + self.failUnless(isinstance(barry_rref, RemoteReference)) + self.failUnless(isinstance(bill_rref, RemoteReference)) + self.failIf(barry_rref == bill_success) + dl = defer.DeferredList([d1, d2]) + dl.addCallback(_check) + self.services.append(t1) + eventually(t1.startService) + return dl + + def test_queued_reconnector(self): + t1 = GoodEnoughTub() + bill_connections = [] + barry_connections = [] + t1.connectTo(self.bill_url, bill_connections.append) + t1.connectTo(self.barry_url, barry_connections.append) + def _check(): + if len(bill_connections) >= 1 and len(barry_connections) >= 1: + return True + return False + d = self.poll(_check) + def _validate(res): + self.failUnless(isinstance(bill_connections[0], RemoteReference)) + self.failUnless(isinstance(barry_connections[0], RemoteReference)) + self.failIf(bill_connections[0] == barry_connections[0]) + d.addCallback(_validate) + self.services.append(t1) + eventually(t1.startService) + return d + diff --git a/src/foolscap/misc/dapper/debian/changelog b/src/foolscap/misc/dapper/debian/changelog index f3331be3..9699a6df 100644 --- a/src/foolscap/misc/dapper/debian/changelog +++ b/src/foolscap/misc/dapper/debian/changelog @@ -1,14 +1,14 @@ -foolscap (0.1.3) unstable; urgency=low +foolscap (0.1.4) unstable; urgency=low * new release - -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + -- Brian Warner Mon, 14 May 2007 22:37:04 -0700 -foolscap (0.1.2+) unstable; urgency=low +foolscap (0.1.3) unstable; urgency=low - * bump revision while between releases + * new release - -- Brian Warner Fri, 13 Apr 2007 00:22:10 -0700 + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 foolscap (0.1.2) unstable; urgency=low @@ -16,79 +16,43 @@ foolscap (0.1.2) unstable; urgency=low -- Brian Warner Wed, 4 Apr 2007 12:32:46 -0700 -foolscap (0.1.1+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Wed, 4 Apr 2007 10:32:22 -0700 - foolscap (0.1.1) unstable; urgency=low * new release -- Brian Warner Tue, 3 Apr 2007 20:48:07 -0700 -foolscap (0.1.0+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 19 Mar 2007 23:11:35 -0700 - foolscap (0.1.0) unstable; urgency=low * new release -- Brian Warner Thu, 15 Mar 2007 16:56:16 -0700 -foolscap (0.0.7+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 22 Jan 2007 12:41:18 -0800 - foolscap (0.0.7) unstable; urgency=low * new release -- Brian Warner Tue, 16 Jan 2007 12:03:00 -0800 -foolscap (0.0.6+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Thu, 4 Jan 2007 18:45:04 -0500 - foolscap (0.0.6) unstable; urgency=low * new release -- Brian Warner Mon, 18 Dec 2006 12:10:51 -0800 -foolscap (0.0.5+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 14 Nov 2006 21:24:17 -0800 - foolscap (0.0.5) unstable; urgency=low * new release -- Brian Warner Sat, 4 Nov 2006 23:20:46 -0800 -foolscap (0.0.4+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 31 Oct 2006 23:38:34 -0800 - foolscap (0.0.4) unstable; urgency=low * new release - -- Brian Warner Thu, 26 Oct 2006 00:46:11 -0700 + -- Brian Warner Thu, 26 Oct 2006 00:46:30 -0700 -foolscap (0.0.3+) unstable; urgency=low +foolscap (0.0.3) unstable; urgency=low * new upstream release, put debian packaging in the tree diff --git a/src/foolscap/misc/dapper/debian/rules b/src/foolscap/misc/dapper/debian/rules index eb2284d2..8ee04543 100644 --- a/src/foolscap/misc/dapper/debian/rules +++ b/src/foolscap/misc/dapper/debian/rules @@ -41,7 +41,7 @@ binary-indep: build install dh_testdir dh_testroot dh_installdocs -i -A NEWS README - dh_installdocs ChangeLog doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-pb.xhtml doc/copyable.xhtml doc/listings doc/specifications + dh_installdocs ChangeLog doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-foolscap.xhtml doc/copyable.xhtml doc/listings doc/specifications dh_installchangelogs -i dh_compress -i -X.py dh_fixperms diff --git a/src/foolscap/misc/edgy/debian/changelog b/src/foolscap/misc/edgy/debian/changelog index 93eb7f3a..9699a6df 100644 --- a/src/foolscap/misc/edgy/debian/changelog +++ b/src/foolscap/misc/edgy/debian/changelog @@ -1,14 +1,14 @@ -foolscap (0.1.3) unstable; urgency=low +foolscap (0.1.4) unstable; urgency=low * new release - -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + -- Brian Warner Mon, 14 May 2007 22:37:04 -0700 -foolscap (0.1.2+) unstable; urgency=low +foolscap (0.1.3) unstable; urgency=low - * bump revision while between releases + * new release - -- Brian Warner Fri, 13 Apr 2007 00:22:10 -0700 + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 foolscap (0.1.2) unstable; urgency=low @@ -16,79 +16,43 @@ foolscap (0.1.2) unstable; urgency=low -- Brian Warner Wed, 4 Apr 2007 12:32:46 -0700 -foolscap (0.1.1+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Wed, 4 Apr 2007 10:32:22 -0700 - foolscap (0.1.1) unstable; urgency=low * new release -- Brian Warner Tue, 3 Apr 2007 20:48:07 -0700 -foolscap (0.1.0+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 19 Mar 2007 23:11:35 -0700 - foolscap (0.1.0) unstable; urgency=low * new release -- Brian Warner Thu, 15 Mar 2007 16:56:16 -0700 -foolscap (0.0.7+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 22 Jan 2007 12:41:18 -0800 - foolscap (0.0.7) unstable; urgency=low * new release -- Brian Warner Tue, 16 Jan 2007 12:03:00 -0800 -foolscap (0.0.6+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Thu, 4 Jan 2007 18:45:04 -0500 - foolscap (0.0.6) unstable; urgency=low * new release -- Brian Warner Mon, 18 Dec 2006 12:10:51 -0800 -foolscap (0.0.5+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 14 Nov 2006 21:24:17 -0800 - foolscap (0.0.5) unstable; urgency=low * new release -- Brian Warner Sat, 4 Nov 2006 23:20:46 -0800 -foolscap (0.0.4+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 31 Oct 2006 23:38:34 -0800 - foolscap (0.0.4) unstable; urgency=low * new release -- Brian Warner Thu, 26 Oct 2006 00:46:30 -0700 -foolscap (0.0.3+) unstable; urgency=low +foolscap (0.0.3) unstable; urgency=low * new upstream release, put debian packaging in the tree diff --git a/src/foolscap/misc/edgy/debian/rules b/src/foolscap/misc/edgy/debian/rules index 3ea18835..252672cc 100644 --- a/src/foolscap/misc/edgy/debian/rules +++ b/src/foolscap/misc/edgy/debian/rules @@ -9,7 +9,7 @@ include /usr/share/cdbs/1/class/python-distutils.mk install/python-foolscap:: - dh_installdocs doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-pb.xhtml doc/copyable.xhtml doc/listings doc/specifications + dh_installdocs doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-foolscap.xhtml doc/copyable.xhtml doc/listings doc/specifications clean:: -rm -rf build diff --git a/src/foolscap/misc/feisty/debian/changelog b/src/foolscap/misc/feisty/debian/changelog index 93eb7f3a..9699a6df 100644 --- a/src/foolscap/misc/feisty/debian/changelog +++ b/src/foolscap/misc/feisty/debian/changelog @@ -1,14 +1,14 @@ -foolscap (0.1.3) unstable; urgency=low +foolscap (0.1.4) unstable; urgency=low * new release - -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + -- Brian Warner Mon, 14 May 2007 22:37:04 -0700 -foolscap (0.1.2+) unstable; urgency=low +foolscap (0.1.3) unstable; urgency=low - * bump revision while between releases + * new release - -- Brian Warner Fri, 13 Apr 2007 00:22:10 -0700 + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 foolscap (0.1.2) unstable; urgency=low @@ -16,79 +16,43 @@ foolscap (0.1.2) unstable; urgency=low -- Brian Warner Wed, 4 Apr 2007 12:32:46 -0700 -foolscap (0.1.1+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Wed, 4 Apr 2007 10:32:22 -0700 - foolscap (0.1.1) unstable; urgency=low * new release -- Brian Warner Tue, 3 Apr 2007 20:48:07 -0700 -foolscap (0.1.0+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 19 Mar 2007 23:11:35 -0700 - foolscap (0.1.0) unstable; urgency=low * new release -- Brian Warner Thu, 15 Mar 2007 16:56:16 -0700 -foolscap (0.0.7+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 22 Jan 2007 12:41:18 -0800 - foolscap (0.0.7) unstable; urgency=low * new release -- Brian Warner Tue, 16 Jan 2007 12:03:00 -0800 -foolscap (0.0.6+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Thu, 4 Jan 2007 18:45:04 -0500 - foolscap (0.0.6) unstable; urgency=low * new release -- Brian Warner Mon, 18 Dec 2006 12:10:51 -0800 -foolscap (0.0.5+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 14 Nov 2006 21:24:17 -0800 - foolscap (0.0.5) unstable; urgency=low * new release -- Brian Warner Sat, 4 Nov 2006 23:20:46 -0800 -foolscap (0.0.4+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 31 Oct 2006 23:38:34 -0800 - foolscap (0.0.4) unstable; urgency=low * new release -- Brian Warner Thu, 26 Oct 2006 00:46:30 -0700 -foolscap (0.0.3+) unstable; urgency=low +foolscap (0.0.3) unstable; urgency=low * new upstream release, put debian packaging in the tree diff --git a/src/foolscap/misc/feisty/debian/rules b/src/foolscap/misc/feisty/debian/rules index 3ea18835..252672cc 100644 --- a/src/foolscap/misc/feisty/debian/rules +++ b/src/foolscap/misc/feisty/debian/rules @@ -9,7 +9,7 @@ include /usr/share/cdbs/1/class/python-distutils.mk install/python-foolscap:: - dh_installdocs doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-pb.xhtml doc/copyable.xhtml doc/listings doc/specifications + dh_installdocs doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-foolscap.xhtml doc/copyable.xhtml doc/listings doc/specifications clean:: -rm -rf build diff --git a/src/foolscap/misc/sarge/debian/changelog b/src/foolscap/misc/sarge/debian/changelog index f3331be3..9699a6df 100644 --- a/src/foolscap/misc/sarge/debian/changelog +++ b/src/foolscap/misc/sarge/debian/changelog @@ -1,14 +1,14 @@ -foolscap (0.1.3) unstable; urgency=low +foolscap (0.1.4) unstable; urgency=low * new release - -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + -- Brian Warner Mon, 14 May 2007 22:37:04 -0700 -foolscap (0.1.2+) unstable; urgency=low +foolscap (0.1.3) unstable; urgency=low - * bump revision while between releases + * new release - -- Brian Warner Fri, 13 Apr 2007 00:22:10 -0700 + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 foolscap (0.1.2) unstable; urgency=low @@ -16,79 +16,43 @@ foolscap (0.1.2) unstable; urgency=low -- Brian Warner Wed, 4 Apr 2007 12:32:46 -0700 -foolscap (0.1.1+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Wed, 4 Apr 2007 10:32:22 -0700 - foolscap (0.1.1) unstable; urgency=low * new release -- Brian Warner Tue, 3 Apr 2007 20:48:07 -0700 -foolscap (0.1.0+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 19 Mar 2007 23:11:35 -0700 - foolscap (0.1.0) unstable; urgency=low * new release -- Brian Warner Thu, 15 Mar 2007 16:56:16 -0700 -foolscap (0.0.7+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 22 Jan 2007 12:41:18 -0800 - foolscap (0.0.7) unstable; urgency=low * new release -- Brian Warner Tue, 16 Jan 2007 12:03:00 -0800 -foolscap (0.0.6+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Thu, 4 Jan 2007 18:45:04 -0500 - foolscap (0.0.6) unstable; urgency=low * new release -- Brian Warner Mon, 18 Dec 2006 12:10:51 -0800 -foolscap (0.0.5+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 14 Nov 2006 21:24:17 -0800 - foolscap (0.0.5) unstable; urgency=low * new release -- Brian Warner Sat, 4 Nov 2006 23:20:46 -0800 -foolscap (0.0.4+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 31 Oct 2006 23:38:34 -0800 - foolscap (0.0.4) unstable; urgency=low * new release - -- Brian Warner Thu, 26 Oct 2006 00:46:11 -0700 + -- Brian Warner Thu, 26 Oct 2006 00:46:30 -0700 -foolscap (0.0.3+) unstable; urgency=low +foolscap (0.0.3) unstable; urgency=low * new upstream release, put debian packaging in the tree diff --git a/src/foolscap/misc/sarge/debian/rules b/src/foolscap/misc/sarge/debian/rules index eb2284d2..8ee04543 100644 --- a/src/foolscap/misc/sarge/debian/rules +++ b/src/foolscap/misc/sarge/debian/rules @@ -41,7 +41,7 @@ binary-indep: build install dh_testdir dh_testroot dh_installdocs -i -A NEWS README - dh_installdocs ChangeLog doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-pb.xhtml doc/copyable.xhtml doc/listings doc/specifications + dh_installdocs ChangeLog doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-foolscap.xhtml doc/copyable.xhtml doc/listings doc/specifications dh_installchangelogs -i dh_compress -i -X.py dh_fixperms diff --git a/src/foolscap/misc/sid/debian/changelog b/src/foolscap/misc/sid/debian/changelog index 93eb7f3a..9699a6df 100644 --- a/src/foolscap/misc/sid/debian/changelog +++ b/src/foolscap/misc/sid/debian/changelog @@ -1,14 +1,14 @@ -foolscap (0.1.3) unstable; urgency=low +foolscap (0.1.4) unstable; urgency=low * new release - -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + -- Brian Warner Mon, 14 May 2007 22:37:04 -0700 -foolscap (0.1.2+) unstable; urgency=low +foolscap (0.1.3) unstable; urgency=low - * bump revision while between releases + * new release - -- Brian Warner Fri, 13 Apr 2007 00:22:10 -0700 + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 foolscap (0.1.2) unstable; urgency=low @@ -16,79 +16,43 @@ foolscap (0.1.2) unstable; urgency=low -- Brian Warner Wed, 4 Apr 2007 12:32:46 -0700 -foolscap (0.1.1+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Wed, 4 Apr 2007 10:32:22 -0700 - foolscap (0.1.1) unstable; urgency=low * new release -- Brian Warner Tue, 3 Apr 2007 20:48:07 -0700 -foolscap (0.1.0+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 19 Mar 2007 23:11:35 -0700 - foolscap (0.1.0) unstable; urgency=low * new release -- Brian Warner Thu, 15 Mar 2007 16:56:16 -0700 -foolscap (0.0.7+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Mon, 22 Jan 2007 12:41:18 -0800 - foolscap (0.0.7) unstable; urgency=low * new release -- Brian Warner Tue, 16 Jan 2007 12:03:00 -0800 -foolscap (0.0.6+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Thu, 4 Jan 2007 18:45:04 -0500 - foolscap (0.0.6) unstable; urgency=low * new release -- Brian Warner Mon, 18 Dec 2006 12:10:51 -0800 -foolscap (0.0.5+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 14 Nov 2006 21:24:17 -0800 - foolscap (0.0.5) unstable; urgency=low * new release -- Brian Warner Sat, 4 Nov 2006 23:20:46 -0800 -foolscap (0.0.4+) unstable; urgency=low - - * bump revision while between releases - - -- Brian Warner Tue, 31 Oct 2006 23:38:34 -0800 - foolscap (0.0.4) unstable; urgency=low * new release -- Brian Warner Thu, 26 Oct 2006 00:46:30 -0700 -foolscap (0.0.3+) unstable; urgency=low +foolscap (0.0.3) unstable; urgency=low * new upstream release, put debian packaging in the tree diff --git a/src/foolscap/misc/sid/debian/rules b/src/foolscap/misc/sid/debian/rules index 3ea18835..252672cc 100644 --- a/src/foolscap/misc/sid/debian/rules +++ b/src/foolscap/misc/sid/debian/rules @@ -9,7 +9,7 @@ include /usr/share/cdbs/1/class/python-distutils.mk install/python-foolscap:: - dh_installdocs doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-pb.xhtml doc/copyable.xhtml doc/listings doc/specifications + dh_installdocs doc/newpb-jobs.txt doc/newpb-todo.txt doc/use-cases.txt doc/using-foolscap.xhtml doc/copyable.xhtml doc/listings doc/specifications clean:: -rm -rf build diff --git a/src/foolscap/misc/testutils/figleaf.py b/src/foolscap/misc/testutils/figleaf.py index a24d4fab..03d8c08d 100644 --- a/src/foolscap/misc/testutils/figleaf.py +++ b/src/foolscap/misc/testutils/figleaf.py @@ -168,7 +168,6 @@ class CodeTracer: Start recording. """ if not self.started: - self.LOG = open("/tmp/flog.out", "w") self.started = True sys.settrace(self.g)