From: Brian Warner Date: Wed, 2 May 2007 22:20:40 +0000 (-0700) Subject: import foolscap-0.1.3 X-Git-Tag: allmydata-tahoe-0.2.0~3 X-Git-Url: https://git.rkrishnan.org/pf/content/en/service/FOOURL?a=commitdiff_plain;h=ee9b0d1de60d50a055710e1257842c58eb72f908;p=tahoe-lafs%2Ftahoe-lafs.git import foolscap-0.1.3 --- diff --git a/src/foolscap/ChangeLog b/src/foolscap/ChangeLog index af1c82eb..06af6258 100644 --- a/src/foolscap/ChangeLog +++ b/src/foolscap/ChangeLog @@ -1,5 +1,15 @@ 2007-05-02 Brian Warner + * foolscap/__init__.py: release Foolscap-0.1.3 + * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same + +2007-05-02 Brian Warner + + * MANIFEST.in: include some recently-added files to the source + tarball + + * NEWS: update for the upcoming release + * foolscap/reconnector.py (Reconnector._failed): simplify log/no-log logic diff --git a/src/foolscap/MANIFEST.in b/src/foolscap/MANIFEST.in index 017bcf38..026d37fa 100644 --- a/src/foolscap/MANIFEST.in +++ b/src/foolscap/MANIFEST.in @@ -1,5 +1,5 @@ include ChangeLog MANIFEST.in NEWS -include doc/*.txt doc/*.xhtml +include doc/*.txt doc/*.xhtml doc/*.css doc/template.tpl include doc/listings/*.py include doc/specifications/*.xhtml include Makefile @@ -8,3 +8,5 @@ include misc/edgy/debian/* include misc/feisty/debian/* include misc/sid/debian/* include misc/sarge/debian/* +include misc/testutils/* +include misc/testutils/twisted/plugins/* diff --git a/src/foolscap/NEWS b/src/foolscap/NEWS index b193ff1e..8ef246fb 100644 --- a/src/foolscap/NEWS +++ b/src/foolscap/NEWS @@ -1,5 +1,202 @@ User visible changes in Foolscap (aka newpb/pb2). -*- outline -*- +* Release 0.1.3 (02 May 2007) + +** Incompatibility Warning + +The 'keepalive' feature described below adds a new pair of banana tokens, +PING and PONG, which introduces a compatibility break between 0.1.2 and 0.1.3 +. Older versions would throw an error upon receipt of a PING token, so the +version-negotiation mechanism is used to prevent banana-v2 (0.1.2) peers from +connecting to banana-v3 (0.1.3+) peers. Our negotiation mechanism would make +it possible to detect the older (v2) peer and refrain from using PINGs, but +that has not been done for this release. + +** Tubs must be running before use + +Tubs are twisted.application.service.Service instances, and as such have a +clear distinction between "running" and "not running" states. Tubs are +started by calling startService(), or by attaching them to a running service, +or by starting the service that they are already attached to. The design rule +in operation here is that Tubs are not allowed to perform network IO until +they are running. + +This rule was not enforced completely in 0.1.2, and calls to +getReference()/connectTo() that occurred before the Tub was started would +proceed normally (initiating a TCP connection, etc). Starting with 0.1.3, +this rule *is* enforced. For now, that means that you must start the Tub +before calling either of these methods, or you'll get an exception. In a +future release, that may be changed to allow these early calls, and queue or +otherwise defer the network IO until the Tub is eventually started. (the +biggest issue is how to warn users who forget to start the Tub, since in the +face of such a bug the getReference will simply never complete). + +** Keepalives + +Tubs now keep track of how long a connection has been idle, and will send a +few bytes (a PING of the other end) if no other traffic has been seen for +roughly 4 to 8 minutes. This serves two purposes. The first is to convince an +intervening NAT box that the connection is still in use, to prevent it from +discarding the connection's table entry, since that would block any further +traffic. The second is to accelerate the detection of such blocked +connections, specifically to reduce the size of a window of buggy behavior in +Foolscap's duplicate-connection detection/suppression code. + +This problem arises when client A (behind a low-end NAT box) connects to +server B, perhaps using connectTo(). The first connection works fine, and is +used for a while. Then, for whatever reason, A and B are silent for a long +time (perhaps as short as 20 minutes, depending upon the NAT box). During +this silence, A's NAT box thinks the connection is no longer in use and drops +the address-translation table entry. Now suppose that A suddenly decides to +talk to B. If the NAT box creates a new entry (with a new outbound port +number), the packets that arrive on B will be rejected, since they do not +match any existing TCP connections. A sees these rejected packets, breaks the +TCP connection, and the Reconnector initiates a new connection. Meanwhile, B +has no idea that anything has gone wrong. When the second connection reaches +B, it thinks this is a duplicate connection from A, and that it already has a +perfectly functional (albeit quiet) connection for that TubID, so it rejects +the connection during the negotiation phase. A sees this rejection and +schedules a new attempt, which ends in the same result. This has the +potential to prevent hosts behind NAT boxes from ever reconnecting to the +other end, at least until the the program at the far end is restarted, or it +happens to try to send some traffic of its own. + +The same problem can occur if a laptop is abruptly shut down, or unplugged +from the network, then moved to a different network. Similar problems have +been seen with virtual machine instances that were suspended and moved to a +different network. + +The longer-term fix for this is a deep change to the way duplicate +connections (and cross-connect race conditions) are handled. The keepalives, +however, mean that both sides are continually checking to see that the +connection is still usable, enabling TCP to break the connection once the +keepalives go unacknowledged for a certain amount of time. The default +keepalive timer is 4 minutes, and due to the way it is implemented this means +that no more than 8 minutes will pass without some traffic being sent. TCP +tends to time out connections after perhaps 15 minutes of unacknowledged +traffic, which means that the window of unconnectability is probably reduced +from infinity down to about 25 minutes. + +The keepalive-sending timer defaults to 4 minutes, and can be changed by +calling tub.setOption("keepaliveTimeout", seconds). + +In addition, an explicit disconnect timer can be enabled, which tells +Foolscap to drop the connection unless traffic has been seen within some +minimum span of time. This timer can be set by calling +tub.setOption("disconnectTimeout", seconds). Obviously it should be set to a +higher value than the keepaliveTimeout. This will close connections faster +than TCP will. Both TCP disconnects and the ones triggered by this +disconnectTimeout run the risk of false negatives, of course, in the face of +unreliable networks. + +** New constraints + +When a tuple appears in a method constraint specification, it now maps to an +actual TupleOf constraint. Previously they mapped to a ChoiceOf constraint. +In practice, TupleOf appears to be much more useful, and thus better +deserving of the shortcut. + +For example, a method defined as follows: + + def get_employee(idnumber=int): + return (str, int, int) # (name, room_number, age) + +can only return a three-element tuple, in which the first element is a string +(specifically it conforms to a default StringConstraint), and the second two +elements are ints (which conform to a default IntegerConstraint, which means +it fits in a 32-bit signed twos-complement value). + +To specify a constraint that can accept alternatives, use ChoiceOf: + + def get_record(key=str): + """Return the record (a string) if it is present, or None if + it is not present.""" + return ChoiceOf(str, None) + +UnicodeConstraint has been added, with minLength=, maxLength=, and regexp= +arguments. + +The previous StringConstraint has been renamed to ByteStringConstraint (for +accuracy), and it is defined to *only* accept string objects (not unicode +objects). 'StringConstraint' itself remains equivalent to +ByteStringConstraint for now, but in the future it may be redefined to be a +constraint that accepts both bytestrings and unicode objects. To accomplish +the bytestring-or-unicode constraint now, you might try +schema.AnyStringConstraint, but it has not been fully tested, and might not +work at all. + +** Bugfixes + +Errors during negotiation were sometimes delivered in the wrong format, +resulting in a "token prefix is limited to 64 bytes" error message. Several +error messages (including that one) have been improved to give developers a +better chance of determining where the actual problem lies. + +RemoteReference.notifyOnDisconnect was buggy when called on a reference that +was already broken: it failed to fire the callback. Now it fires the callback +soon (using an eventual-send). This should remove a race condition from +connectTo+notifyOnDisconnect sequences and allow them to operate reliably. +notifyOnDisconnect() is now tolerant of attempts to remove something twice, +which should make it easier to use safely. + +Remote methods which raise string exceptions should no longer cause Foolscap +to explode. These sorts of exceptions are deprecated, of course, and you +shouldn't use them, but at least they won't break Foolscap. + +The Reconnector class (accessed by tub.connectTo) was not correctly +reconnecting in certain cases (which appeared to be particularly common on +windows). This should be fixed now. + +CopyableSlicer did not work inside containers when streaming was enabled. +Thanks to iacovou-AT-gmail.com for spotting this one. + +** Bugs not fixed + +Some bugs were identified and characterized but *not* fixed in this release + +*** RemoteInterfaces aren't defaulting to fully-qualified classnames + +When defining a RemoteInterface, you can specify its name with +__remote_name__, or you can allow it to use the default name. Unfortunately, +the default name is only the *local* name of the class, not the +fully-qualified name, which means that if you have an RIFoo in two different +.py files, they will wind up with the same name (which will cause an error on +import, since all RemoteInterfaces known to a Foolscap-using program must +have unique names). + +It turns out that it is rather difficult to determine the fully-qualified +name of the RemoteInterface class early enough to be helpful. The workaround +is to always add a __remote_name__ to your RemoteInterface classes. The +recommendation is to use a globally-unique string, like a URI that includes +your organization's DNS name. + +*** Constraints aren't constraining inbound tokens well enough + +Constraints (and the RemoteInterfaces they live inside) serve three purposes. +The primary one is as documentation, describing how remotely-accessible +objects behave. The second purpose is to enforce that documentation, by +inspecting arguments (and return values) before invoking the method, as a +form of precondition checking. The third is to mitigate denial-of-service +attacks, in which an attacker sends so much data (or carefully crafted data) +that the receiving program runs out of memory or stack space. + +It looks like several constraints are not correctly paying attention to the +tokens as they arrive over the wire, such that the third purpose is not being +achieved. Hopefully this will be fixed in a later release. Application code +can be unaware of this change, since the constraints are still being applied +to inbound arguments before they are passed to the method. Continue to use +RemoteInterfaces as usual, just be aware that you are not yet protected +against certain DoS attacks. + +** Use os.urandom instead of falling back to pycrypto + +Once upon a time, when Foolscap was compatible with python2.3 (which lacks +os.urandom), we would try to use PyCrypto's random-number-generation routines +when creating unguessable object identifiers (aka "SwissNumbers"). Now that +we require python2.4 or later, this fallback has been removed, eliminating +the last reference to pycrypto within the Foolscap source tree. + + * Release 0.1.2 (04 Apr 2007) ** Bugfixes diff --git a/src/foolscap/doc/using-pb.xhtml b/src/foolscap/doc/using-pb.xhtml index f7e8c457..97005d01 100644 --- a/src/foolscap/doc/using-pb.xhtml +++ b/src/foolscap/doc/using-pb.xhtml @@ -750,7 +750,7 @@ classname (like package.module.RIFoo), so that two RemoteInterfaces with the same name in different modules can co-exist. In the current release, these two RemoteInterfaces will collide (and provoke an import-time error message complaining about the duplicate name). As a result, -if you have such classes (e.g. foo.RIBar and +if you have such classes (e.g. foo.RIBar and baz.RIBar), you must use __remote_name__ to distinguish them (by naming one of them something other than RIBar to avoid this error. diff --git a/src/foolscap/foolscap/__init__.py b/src/foolscap/foolscap/__init__.py index 27f4006f..e2243f42 100644 --- a/src/foolscap/foolscap/__init__.py +++ b/src/foolscap/foolscap/__init__.py @@ -1,6 +1,6 @@ """Foolscap""" -__version__ = "0.1.2+" +__version__ = "0.1.3" # here are the primary entry points from foolscap.pb import Tub, UnauthenticatedTub, getRemoteURL_TCP diff --git a/src/foolscap/foolscap/pb.py b/src/foolscap/foolscap/pb.py index e8ed1675..9f7cfb9d 100644 --- a/src/foolscap/foolscap/pb.py +++ b/src/foolscap/foolscap/pb.py @@ -598,6 +598,7 @@ class Tub(service.MultiService): b = self._createLoopbackBroker(tubref) # _createLoopbackBroker will call brokerAttached, which will add # it to self.brokers + # TODO: stash this in self.brokers, so we don't create multiples return defer.succeed(b) d = defer.Deferred() diff --git a/src/foolscap/misc/dapper/debian/changelog b/src/foolscap/misc/dapper/debian/changelog index 0a18ccad..f3331be3 100644 --- a/src/foolscap/misc/dapper/debian/changelog +++ b/src/foolscap/misc/dapper/debian/changelog @@ -1,3 +1,9 @@ +foolscap (0.1.3) unstable; urgency=low + + * new release + + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + foolscap (0.1.2+) unstable; urgency=low * bump revision while between releases diff --git a/src/foolscap/misc/edgy/debian/changelog b/src/foolscap/misc/edgy/debian/changelog index acdc7392..93eb7f3a 100644 --- a/src/foolscap/misc/edgy/debian/changelog +++ b/src/foolscap/misc/edgy/debian/changelog @@ -1,3 +1,9 @@ +foolscap (0.1.3) unstable; urgency=low + + * new release + + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + foolscap (0.1.2+) unstable; urgency=low * bump revision while between releases diff --git a/src/foolscap/misc/feisty/debian/changelog b/src/foolscap/misc/feisty/debian/changelog index acdc7392..93eb7f3a 100644 --- a/src/foolscap/misc/feisty/debian/changelog +++ b/src/foolscap/misc/feisty/debian/changelog @@ -1,3 +1,9 @@ +foolscap (0.1.3) unstable; urgency=low + + * new release + + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + foolscap (0.1.2+) unstable; urgency=low * bump revision while between releases diff --git a/src/foolscap/misc/sarge/debian/changelog b/src/foolscap/misc/sarge/debian/changelog index 0a18ccad..f3331be3 100644 --- a/src/foolscap/misc/sarge/debian/changelog +++ b/src/foolscap/misc/sarge/debian/changelog @@ -1,3 +1,9 @@ +foolscap (0.1.3) unstable; urgency=low + + * new release + + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + foolscap (0.1.2+) unstable; urgency=low * bump revision while between releases diff --git a/src/foolscap/misc/sid/debian/changelog b/src/foolscap/misc/sid/debian/changelog index acdc7392..93eb7f3a 100644 --- a/src/foolscap/misc/sid/debian/changelog +++ b/src/foolscap/misc/sid/debian/changelog @@ -1,3 +1,9 @@ +foolscap (0.1.3) unstable; urgency=low + + * new release + + -- Brian Warner Wed, 2 May 2007 14:55:49 -0700 + foolscap (0.1.2+) unstable; urgency=low * bump revision while between releases