+2007-05-02 Brian Warner <warner@lothar.com>
+
+ * foolscap/reconnector.py (Reconnector._failed): simplify
+ log/no-log logic
+
+ * foolscap/slicers/unicode.py (UnicodeConstraint): add a new
+ constraint that only accepts unicode objects. It isn't complete:
+ I've forgotten how the innards of Constraints work, and as a
+ result this one is too permissive: it will probably accept too
+ many tokens over the wire before raising a Violation (although the
+ post-receive just-before-the-method-is-called check should still
+ be enforced, so application code shouldn't notice the issue).
+ * foolscap/test/test_schema.py (ConformTest.testUnicode): test it
+ (CreateTest.testMakeConstraint): check the typemap too
+ * foolscap/test/test_call.py (TestCall.testMegaSchema): test in a call
+ * foolscap/test/common.py: same
+
+ * foolscap/constraint.py (ByteStringConstraint): rename
+ StringConstraint to ByteStringConstraint, to more accurately
+ describe its function. This constraint will *not* accept unicode
+ objects.
+ * foolscap/call.py, foolscap/copyable.py, foolscap/referenceable.py:
+ * foolscap/slicers/vocab.py: same
+
+ * foolscap/schema.py (AnyStringConstraint): add a new constraint
+ to accept either bytestrings or unicode objects. I don't think it
+ actually works yet, particularly when used inside containers.
+ (constraintMap): map 'str' to ByteStringConstraint for now. Maybe
+ someday it should be mapped to AnyStringConstraint, but not today.
+ Map 'unicode' to UnicodeConstraint.
+
+
+ * foolscap/pb.py (Tub.getReference): assert that the Tub is
+ already running, either because someone called Tub.startService(),
+ or because we've been attached (with tub.setServiceParent) to a
+ running service. This requirement appeared with the
+ connector-tracking code, and I hope to relax it at some
+ point (such that any pre-startService getReferences will be queued
+ and serviced when the Tub is finally started), but for this
+ release it is a requirement to start the service before trying to
+ use it.
+ (Tub.connectTo): same
+ * doc/using-pb.xhtml: document it
+ * doc/listings/pb1client.py: update example to match
+ * doc/listings/pb2client.py: update example to match
+ * doc/listings/pb3client.py: update example to match
+
+ * foolscap/pb.py (Tub.connectorFinished): if, for some reason,
+ we're removing the same connector twice, log and ignore rather
+ than explode. I can't find a code path that would allow this, but
+ I *have* seen it occur in practice, and the results aren't pretty.
+ Since the whole connection-tracking thing is really for the
+ benefit of unit tests anyways (who want to know when
+ Tub.stopService is done), I think it's more important to keep
+ application code running.
+
+ * foolscap/negotiate.py (TubConnector.shutdown): clear out
+ self.remainingLocations too, in case it helps to shut things down
+ faster. Add some comments.
+
+ * foolscap/negotiate.py (Negotiation): improve error-message
+ delivery, by keeping track of what state the receiver is in (i.e.
+ whether we should send them an HTTP error block, an rfc822-style
+ error-block, or a banana ERROR token).
+ (Negotiation.switchToBanana): empty self.buffer, to make sure that
+ any extra data is passed entirely to the new Banana protocol and
+ none of it gets passed back to ourselves
+ (Negotiation.dataReceived): same, only recurse if there's something
+ still in self.buffer. In other situtations we recurse here because
+ we might have somehow received data for two separate phases in a
+ single packet.
+
+ * foolscap/banana.py (Banana.sendError): rather than explode when
+ trying to send an overly-long error message, just truncate it.
+
+2007-04-30 Brian Warner <warner@lothar.com>
+
+ * foolscap/broker.py (Broker.notifyOnDisconnect): if the
+ RemoteReference is already dead, notify the callback right away.
+ Previously we would never notify them, which was a problem.
+ (Broker.dontNotifyOnDisconnect): be tolerant of attempts to
+ unregister callbacks that have already fired. I think this makes it
+ easier to write correct code, but on the other hand it loses the
+ assertion feedback if somebody tries to unregister something that
+ was never registered in the first place.
+ * foolscap/test/test_call.py (TestCall.testNotifyOnDisconnect):
+ test this new tolerance
+ (TestCall.testNotifyOnDisconnect_unregister): same
+ (TestCall.testNotifyOnDisconnect_already): test that a handler
+ fires when the reference was already broken
+
+ * foolscap/call.py (InboundDelivery.logFailure): don't use
+ f.getTraceback() on string exceptions: twisted explodes
+ (FailureSlicer.getStateToCopy): same
+ * foolscap/test/test_call.py (TestCall.testFailStringException):
+ skip the test on python2.5, since string exceptions are deprecated
+ anyways and I don't want the warning message to clutter the test
+ logs
+
+ * doc/using-pb.xhtml (RemoteInterfaces): document the fact that
+ the default name is *not* fully-qualified, necessitating the use
+ of __remote_name__ to distinguish between foo.RIBar and baz.RIBar
+ * foolscap/remoteinterface.py: same
+
+ * foolscap/call.py (FailureSlicer.getStateToCopy): handle string
+ exceptions without exploding, annoying as they are.
+ * foolscap/test/test_call.py (TestCall.testFail4): test them
+
2007-04-27 Brian Warner <warner@lothar.com>
+ * foolscap/broker.py (Broker.freeYourReference._ignore_loss):
+ change the way we ignore DeadReferenceError and friends, since
+ f.trap is not suitable for direct use as an errback
+
+ * foolscap/referenceable.py (SturdyRef.__init__): log the repr of
+ the unparseable FURL, rather than just the str, in case there are
+ weird control characters in it
+
+ * foolscap/banana.py (Banana.handleData): rewrite the typebyte
+ scanning loop, to remove the redundant pos<64 check. Also, if we
+ get an overlong prefix, log it so we can figure out what's going
+ wrong.
+ * foolscap/test/test_banana.py: update to match
+
+ * foolscap/negotiate.py (Negotiation.dataReceived): if a
+ non-NegotiationError exception occurs, log it, since it indicates
+ a foolscap coding failure rather than some disagreement with the
+ remote end. Log it with 'log.msg' for now, since some of the unit
+ tests seem to trigger startTLS errors that flunk tests which
+ should normally pass. I suspect some problems with error handling
+ in twisted's TLS implementation, but I'll have to investigate it
+ later. Eventually this will turn into a log.err.
+
* foolscap/pb.py (Tub.keepaliveTimeout): set the default keepalive
timer to 4 minutes. This means that at most 8 minutes will go by
without any traffic at all, which should be a reasonable value to
+++ /dev/null
-Metadata-Version: 1.0
-Name: foolscap
-Version: 0.1.2+
-Summary: Foolscap contains an RPC protocol for Twisted.
-Home-page: http://twistedmatrix.com/trac/wiki/FoolsCap
-Author: Brian Warner
-Author-email: warner@twistedmatrix.com
-License: MIT
-Description: Foolscap (aka newpb) is a new version of Twisted's native RPC protocol, known
- as 'Perspective Broker'. This allows an object in one process to be used by
- code in a distant process. This module provides data marshaling, a remote
- object reference system, and a capability-based security model.
-
-Platform: UNKNOWN
reactor.stop()
tub = Tub()
+tub.startService()
d = tub.getReference("pbu://localhost:12345/math-service")
d.addCallbacks(gotReference, gotError1)
-tub.startService()
reactor.run()
sys.exit(1)
url = sys.argv[1]
tub = Tub()
+tub.startService()
d = tub.getReference(url)
d.addCallbacks(gotReference, gotError1)
-tub.startService()
reactor.run()
url = sys.argv[1]
tub = Tub()
+tub.startService()
d = tub.getReference(url)
d.addCallback(gotRemote)
-tub.startService()
reactor.run()
--- /dev/null
+
+span.footnote {
+ vertical-align: super;
+ font-size: small;
+}
+
+span.footnote:before
+{
+ content: "[Footnote: ";
+}
+
+span.footnote:after
+{
+ content: "]";
+}
+
+div.note:before
+{
+ content: "Note: ";
+}
--- /dev/null
+
+body
+{
+ margin-left: 2em;
+ margin-right: 2em;
+ border: 0px;
+ padding: 0px;
+ font-family: sans-serif;
+ }
+
+.done { color: #005500; background-color: #99ff99 }
+.notdone { color: #550000; background-color: #ff9999;}
+
+pre
+{
+ padding: 1em;
+ font-family: Neep Alt, Courier New, Courier;
+ font-size: 12pt;
+ border: thin black solid;
+}
+
+.boxed
+{
+ padding: 1em;
+ border: thin black solid;
+}
+
+.shell
+{
+ background-color: #ffffdd;
+}
+
+.python
+{
+ background-color: #dddddd;
+}
+
+.htmlsource
+{
+ background-color: #dddddd;
+}
+
+.py-prototype
+{
+ background-color: #ddddff;
+}
+
+
+.python-interpreter
+{
+ background-color: #ddddff;
+}
+
+.doit
+{
+ border: thin blue dashed ;
+ background-color: #0ef
+}
+
+.py-src-comment
+{
+ color: #1111CC
+}
+
+.py-src-keyword
+{
+ color: #3333CC;
+ font-weight: bold;
+}
+
+.py-src-parameter
+{
+ color: #000066;
+ font-weight: bold;
+}
+
+.py-src-identifier
+{
+ color: #CC0000
+}
+
+.py-src-string
+{
+
+ color: #115511
+}
+
+.py-src-endmarker
+{
+ display: block; /* IE hack; prevents following line from being sucked into the py-listing box. */
+}
+
+.py-listing, .html-listing, .listing
+{
+ margin: 1ex;
+ border: thin solid black;
+ background-color: #eee;
+}
+
+.py-listing pre, .html-listing pre, .listing pre
+{
+ margin: 0px;
+ border: none;
+ border-bottom: thin solid black;
+}
+
+.py-listing .python
+{
+ margin-top: 0;
+ margin-bottom: 0;
+ border: none;
+ border-bottom: thin solid black;
+ }
+
+.html-listing .htmlsource
+{
+ margin-top: 0;
+ margin-bottom: 0;
+ border: none;
+ border-bottom: thin solid black;
+ }
+
+.caption
+{
+ text-align: center;
+ padding-top: 0.5em;
+ padding-bottom: 0.5em;
+}
+
+.filename
+{
+ font-style: italic;
+ }
+
+.manhole-output
+{
+ color: blue;
+}
+
+hr
+{
+ display: inline;
+ }
+
+ul
+{
+ padding: 0px;
+ margin: 0px;
+ margin-left: 1em;
+ padding-left: 1em;
+ border-left: 1em;
+ }
+
+li
+{
+ padding: 2px;
+ }
+
+dt
+{
+ font-weight: bold;
+ margin-left: 1ex;
+ }
+
+dd
+{
+ margin-bottom: 1em;
+ }
+
+div.note
+{
+ background-color: #FFFFCC;
+ margin-top: 1ex;
+ margin-left: 5%;
+ margin-right: 5%;
+ padding-top: 1ex;
+ padding-left: 5%;
+ padding-right: 5%;
+ border: thin black solid;
+}
--- /dev/null
+<?xml version="1.0"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+ <head>
+<title>Twisted Documentation: </title>
+<link type="text/css" rel="stylesheet"
+href="stylesheet.css" />
+ </head>
+
+ <body bgcolor="white">
+ <h1 class="title"></h1>
+ <div class="toc"></div>
+ <div class="body">
+
+ </div>
+
+ <p><a href="index.html">Index</a></p>
+ <span class="version">Version: </span>
+ </body>
+</html>
+
can use <code>startService</code> and <code>stopService</code> on the Tub
directly.</p>
+<p>Note that you must start the Tub before calling <code>getReference</code>
+or <code>connectTo</code>, since both of these trigger network activity, and
+Tubs are supposed to be silent until they are started. In a future release
+this requirement may be relaxed, but the principle of "no network activity
+until the Tub is started" will be maintained, probably by queueing the
+<code>getReference</code> calls and handling them after the Tub been
+started.</p>
+
<h3>Making your Tub remotely accessible</h3>
<p>To make any of your <code>Referenceable</code>s available, you must make
from foolscap import Tub
tub = Tub()
+tub.startService()
d = tub.getReference("pb://ABCD@myhost.example.com:12345/math-service")
def gotReference(remote):
print "Got the RemoteReference:", remote
after the <code>RemoteInterface</code> body has been evaluated.</p>
<p>The <code>RemoteInterface</code> <q>class</q> has a name. Normally this is
-the fully-qualified classname<span
+the (short) classname<span
class="footnote"><code>RIFoo.__class__.__name__</code>, if
<code>RemoteInterface</code>s were actually classes, which they're
-not</span>, like <code>package.module.RIFoo</code>. You can override this
+not</span>. You can override this
name by setting a special <code>__remote_name__</code> attribute on the
<code>RemoteInterface</code> (again, in the body). This name is important
because it is externally visible: all <code>RemoteReference</code>s that
<code>RemoteInterface</code>s it implements. This is what enables the
type-checking to be performed on both ends of the wire.</p>
+<p>In the future, this ought to default to the <b>fully-qualified</b>
+classname (like <code>package.module.RIFoo</code>), so that two
+RemoteInterfaces with the same name in different modules can co-exist. In the
+current release, these two RemoteInterfaces will collide (and provoke an
+import-time error message complaining about the duplicate name). As a result,
+if you have such classes (e.g. </code>foo.RIBar</code> and
+<code>baz.RIBar</code>), you <b>must</b> use <code>__remote_name__</code> to
+distinguish them (by naming one of them something other than
+<code>RIBar</code> to avoid this error.
+
+Hopefully this will be improved in a future version, but it looks like a
+difficult change to implement, so the standing recommendation is to use
+<code>__remote_name__</code> on all your RemoteInterfaces, and set it to a
+suitably unique string (like a URI).</p>
+
<p>Here's an example:</p>
<pre class="python">
from foolscap import RemoteInterface, schema
class RIMath(RemoteInterface):
+ __remote_name__ = "RIMath.using-pb.docs.foolscap.twistedmatrix.com"
def add(a=int, b=int):
return int
# declare it with an attribute instead of a function definition
def sendError(self, msg):
if len(msg) > SIZE_LIMIT:
- raise BananaError, \
- "error string is too long to send (%d)" % len(msg)
+ msg = msg[:SIZE_LIMIT-10] + "..."
int2b128(len(msg), self.transport.write)
self.transport.write(ERROR)
self.transport.write(msg)
(repr(buffer),))
self.buffer = buffer
pos = 0
+
for ch in buffer:
if ch >= HIGH_BIT_SET:
break
pos = pos + 1
- # TODO: the 'pos > 64' test should probably move here. If
- # not, a huge chunk will consume more CPU than it needs to.
- # On the other hand, the test would consume extra CPU all
- # the time.
- else:
if pos > 64:
- # drop the connection
+ # drop the connection. We log more of the buffer, but not
+ # all of it, to make it harder for someone to spam our
+ # logs.
raise BananaError("token prefix is limited to 64 bytes: "
- "but got %r" % (buffer[:pos],))
- return # still waiting for header to finish
+ "but got %r" % (buffer[:200],))
+ else:
+ # we've run out of buffer without seeing the high bit, which
+ # means we're still waiting for header to finish
+ return
+ assert pos <= 64
# At this point, the header and type byte have been received.
# The body may or may not be complete.
typebyte = buffer[pos]
- if pos > 64:
- # redundant?
- raise BananaError("token prefix is limited to 64 bytes")
if pos:
header = b1282int(buffer[:pos])
else:
self.yourReferenceByURL = {}
self.myGifts = {}
self.myGiftsByGiftID = {}
- dw, self.disconnectWatchers = self.disconnectWatchers, []
- for (cb,args,kwargs) in dw:
+ for (cb,args,kwargs) in self.disconnectWatchers:
eventually(cb, *args, **kwargs)
+ self.disconnectWatchers = []
banana.Banana.connectionLost(self, why)
if self.tub:
# TODO: remove the conditional. It is only here to accomodate
def notifyOnDisconnect(self, callback, *args, **kwargs):
marker = (callback, args, kwargs)
- self.disconnectWatchers.append(marker)
+ if self.disconnected:
+ eventually(callback, *args, **kwargs)
+ else:
+ self.disconnectWatchers.append(marker)
return marker
def dontNotifyOnDisconnect(self, marker):
- self.disconnectWatchers.remove(marker)
+ if self.disconnected:
+ return
+ # be tolerant of attempts to unregister a callback that has already
+ # fired. I think it is hard to write safe code without this
+ # tolerance.
+
+ # TODO: on the other hand, I'm not sure this is the best policy,
+ # since you lose the feedback that tells you about
+ # unregistering-the-wrong-thing bugs. We need to look at the way that
+ # register/unregister gets used and see if there is a way to retain
+ # the typechecking that results from insisting that you can only
+ # remove something that was stil in the list.
+ if marker in self.disconnectWatchers:
+ self.disconnectWatchers.remove(marker)
# methods to handle RemoteInterfaces
def getRemoteInterfaceByName(self, name):
d = rb.callRemote("decref", clid=tracker.clid, count=count)
# if the connection was lost before we can get an ack, we're
# tearing this down anyway
- d.addErrback(lambda f: f.trap(DeadReferenceError))
- d.addErrback(lambda f: f.trap(error.ConnectionLost))
- d.addErrback(lambda f: f.trap(error.ConnectionDone))
+ def _ignore_loss(f):
+ f.trap(DeadReferenceError,
+ error.ConnectionLost,
+ error.ConnectionDone)
+ return None
+ d.addErrback(_ignore_loss)
# once the ack comes back, or if we know we'll never get one,
# release the tracker
d.addCallback(self.freeYourReferenceTracker, tracker)
-from cStringIO import StringIO
-
from twisted.python import failure, log, reflect
from twisted.internet import defer
from foolscap import copyable, slicer, tokens
from foolscap.eventual import eventually
from foolscap.copyable import AttributeDictConstraint
-from foolscap.constraint import StringConstraint
+from foolscap.constraint import ByteStringConstraint
from foolscap.slicers.list import ListConstraint
from tokens import BananaError, Violation
klass = failure.Failure
def __init__(self):
- attrs = [('type', StringConstraint(200)),
- ('value', StringConstraint(1000)),
- ('traceback', StringConstraint(2000)),
- ('parents', ListConstraint(StringConstraint(200))),
+ attrs = [('type', ByteStringConstraint(200)),
+ ('value', ByteStringConstraint(1000)),
+ ('traceback', ByteStringConstraint(2000)),
+ ('parents', ListConstraint(ByteStringConstraint(200))),
]
AttributeDictConstraint.__init__(self, *attrs)
(self.reqID, self.obj, self.methodname))
log.msg(" args=%s" % (self.allargs.args,))
log.msg(" kwargs=%s" % (self.allargs.kwargs,))
- stack = f.getTraceback()
+ if isinstance(f.type, str):
+ stack = "getTraceback() not available for string exceptions\n"
+ else:
+ stack = f.getTraceback()
# TODO: trim stack to everything below Broker._doCall
stack = "LOCAL: " + stack.replace("\n", "\nLOCAL: ")
log.msg(" the failure was:")
#state['stack'] = []
state = {}
+ # string exceptions show up as obj.value == None and
+ # isinstance(obj.type, str). Normal exceptions show up as obj.value
+ # == text and obj.type == exception class. We need to make sure we
+ # can handle both.
if isinstance(obj.value, failure.Failure):
# TODO: how can this happen? I got rid of failure2Copyable, so
# if this case is possible, something needs to replace it
raise RuntimeError("not implemented yet")
#state['value'] = failure2Copyable(obj.value, banana.unsafeTracebacks)
+ elif isinstance(obj.type, str):
+ state['value'] = str(obj.value)
+ state['type'] = obj.type # a string
else:
state['value'] = str(obj.value) # Exception instance
- state['type'] = reflect.qual(obj.type) # Exception class
+ state['type'] = reflect.qual(obj.type) # Exception class
+
if broker.unsafeTracebacks:
- io = StringIO()
- obj.printTraceback(io)
- state['traceback'] = io.getvalue()
+ if isinstance(obj.type, str):
+ stack = "getTraceback() not available for string exceptions\n"
+ else:
+ stack = obj.getTraceback()
+ state['traceback'] = stack
# TODO: provide something with globals and locals and HTML and
# all that cool stuff
else:
limit = self.taster.get(typebyte, "not in list")
if limit == "not in list":
if self.strictTaster:
- raise BananaError("invalid token type")
+ raise BananaError("invalid token type: %s" %
+ tokenNames[typebyte])
else:
raise Violation("%s token rejected by %s" % \
(tokenNames[typebyte], self.name))
# constraints which describe individual banana tokens
-class StringConstraint(Constraint):
+class ByteStringConstraint(Constraint):
opentypes = [] # redundant, as taster doesn't accept OPEN
- name = "StringConstraint"
+ name = "ByteStringConstraint"
def __init__(self, maxLength=1000, minLength=0, regexp=None):
self.maxLength = maxLength
self.regexp = re.compile(regexp)
self.taster = {STRING: self.maxLength,
VOCAB: None}
+
def checkObject(self, obj, inbound):
- if not isinstance(obj, (str, unicode)):
- raise Violation("not a String")
+ if not isinstance(obj, str):
+ raise Violation("not a bytestring")
if self.maxLength != None and len(obj) > self.maxLength:
raise Violation("string too long (%d > %d)" %
(len(obj), self.maxLength))
import slicer, tokens
from tokens import BananaError, Violation
from foolscap.constraint import OpenerConstraint, IConstraint, \
- StringConstraint, UnboundedSchema, Optional
+ ByteStringConstraint, UnboundedSchema, Optional
Interface = interface.Interface
seen.append(self)
total = self.OPENBYTES("attributedict")
for name, constraint in self.keys.iteritems():
- total += StringConstraint(len(name)).maxSize(seen)
+ total += ByteStringConstraint(len(name)).maxSize(seen)
total += constraint.maxSize(seen[:])
return total
from foolscap import broker, referenceable, vocab
from foolscap.eventual import eventually
-from foolscap.tokens import BananaError, \
- NegotiationError, RemoteNegotiationError
+from foolscap.tokens import SIZE_LIMIT, ERROR, \
+ BananaError, NegotiationError, RemoteNegotiationError
+from foolscap.banana import int2b128
crypto_available = False
try:
raise NegotiationError("I can't handle %s %d" % (name, decision))
# negotiation phases
-PLAINTEXT, ENCRYPTED, DECIDING, ABANDONED = range(4)
+PLAINTEXT, ENCRYPTED, DECIDING, BANANA, ABANDONED = range(5)
# version number history:
tub = None
theirTubID = None
- phase = PLAINTEXT
+ receive_phase = PLAINTEXT # we are expecting this
+ send_phase = PLAINTEXT # the other end is expecting this
encrypted = False
doNegotiation = True
self.switchToBanana({})
def connectionMadeClient(self):
- assert self.phase == PLAINTEXT
+ assert self.receive_phase == PLAINTEXT
# the client needs to send the HTTP-compatible tubid GET,
# along with the TLS upgrade request
self.sendPlaintextClient()
req.append("Connection: Upgrade")
self.transport.write("\r\n".join(req))
self.transport.write("\r\n\r\n")
+ # the next thing the other end expects to see is the encrypted phase
+ self.send_phase = ENCRYPTED
def connectionMadeServer(self):
# the server just waits for the GET message to arrive, but set up the
def dataReceived(self, chunk):
if self.debugNegotiation:
log.msg("dataReceived(isClient=%s,phase=%s,options=%s): '%s'"
- % (self.isClient, self.phase, self.options, chunk))
- if self.phase == ABANDONED:
+ % (self.isClient, self.receive_phase, self.options, chunk))
+ if self.receive_phase == ABANDONED:
return
self.buffer += chunk
if eoh == -1:
return
header, self.buffer = self.buffer[:eoh], self.buffer[eoh+4:]
- if self.phase == PLAINTEXT:
+ if self.receive_phase == PLAINTEXT:
if self.isClient:
self.handlePLAINTEXTClient(header)
else:
self.handlePLAINTEXTServer(header)
- elif self.phase == ENCRYPTED:
+ elif self.receive_phase == ENCRYPTED:
self.handleENCRYPTED(header)
- elif self.phase == DECIDING:
+ elif self.receive_phase == DECIDING:
self.handleDECIDING(header)
else:
assert 0, "should not get here"
- # there might be some leftover data for the next phase
- self.dataReceived("")
+ # there might be some leftover data for the next phase.
+ # self.buffer will be emptied when we switchToBanana, so in that
+ # case we won't call the wrong dataReceived.
+ if self.buffer:
+ self.dataReceived("")
except Exception, e:
why = Failure()
if self.debugNegotiation:
- log.msg("negotation had exception: %s" % why)
+ log.msg("negotiation had exception: %s" % why)
if isinstance(e, RemoteNegotiationError):
pass # they've already hung up
else:
if isinstance(e, NegotiationError):
errmsg = str(e)
else:
+ log.msg("negotiation had internal error:")
+ log.msg(why)
errmsg = "internal server error, see logs"
errmsg = errmsg.replace("\n", " ").replace("\r", " ")
- if self.phase == PLAINTEXT:
+ if self.send_phase == PLAINTEXT:
resp = ("HTTP/1.1 500 Internal Server Error: %s\r\n\r\n"
% errmsg)
self.transport.write(resp)
- elif self.phase in (ENCRYPTED, DECIDING):
+ elif self.send_phase in (ENCRYPTED, DECIDING):
block = {'banana-decision-version': 1,
'error': errmsg,
}
self.sendBlock(block)
+ elif self.send_phase == BANANA:
+ self.sendBananaError(errmsg)
+
self.failureReason = why
self.transport.loseConnection()
return
- # TODO: the error-handling needs some work, try to tell the other end
- # what happened. In certain states we may need to send a header
- # block, in others we may have to send a banana ERROR token.
+ def sendBananaError(self, msg):
+ if len(msg) > SIZE_LIMIT:
+ msg = msg[:SIZE_LIMIT-10] + "..."
+ int2b128(len(msg), self.transport.write)
+ self.transport.write(ERROR)
+ self.transport.write(msg)
+ # now you should drop the connection
def connectionLost(self, reason):
# force connectionMade to happen, so connectionLost can occur
if self.isClient:
l = self.tub.options.get("debug_gatherPhases")
if l is not None:
- l.append(self.phase)
+ l.append(self.receive_phase)
if not self.failureReason:
self.failureReason = reason
self.negotiationFailed()
])
self.transport.write(resp)
self.transport.write("\r\n\r\n")
+ # the next thing they expect is the encrypted block
+ self.send_phase = ENCRYPTED
self.startENCRYPTED(encrypted)
def sendRedirect(self, redirect):
self.startTLS(self.tub.myCertificate)
self.encrypted = encrypted
# TODO: can startTLS trigger dataReceived?
- self.phase = ENCRYPTED
+ self.receive_phase = ENCRYPTED
self.sendHello()
def sendHello(self):
decision, params = None, None
if iAmTheMaster:
- # we get to decide everything
+ # we get to decide everything. The other side is now waiting for
+ # a decision block.
+ self.send_phase = DECIDING
decision = {}
# combine their 'offer' and our own self.negotiationOffer to come
# up with a 'decision' to be sent back to the other end, and the
}
else:
- # otherwise, the other side gets to decide
- pass
+ # otherwise, the other side gets to decide. The next thing they
+ # expect to hear from us is banana.
+ self.send_phase = BANANA
if iAmTheMaster:
self.sendDecision(decision, params)
else:
# I am not the master, I receive the decision
- self.phase = DECIDING
+ self.receive_phase = DECIDING
def evaluateNegotiationVersion2(self, offer):
# version 2 changes the meaning of reqID=0 in a 'call' sequence, to
self.sendDecision, decision, params):
return
self.sendBlock(decision)
+ self.send_phase = BANANA
self.switchToBanana(params)
def handleDECIDING(self, header):
b.setTub(self.tub)
self.transport.protocol = b
b.makeConnection(self.transport)
- b.dataReceived(self.buffer)
+ buf, self.buffer = self.buffer, "" # empty our buffer, just in case
+ b.dataReceived(buf) # and hand it to the new protocol
# if we were created as a client, we'll have a TubConnector. Let them
# know that this connection has succeeded, so they can stop any other
# track down
log.msg("Negotiation.negotiationFailed: %s" % reason)
self.stopNegotiationTimer()
- if self.phase != ABANDONED and self.isClient:
+ if self.receive_phase != ABANDONED and self.isClient:
eventually(self.connector.negotiationFailed, self.factory, reason)
- self.phase = ABANDONED
+ self.receive_phase = ABANDONED
cb = self.options.get("debug_negotiationFailed_cb")
if cb:
# note that this gets called with a NegotiationError, not a
# Failure
eventually(cb, reason)
-# TODO: make sure code that examines self.phase handles ABANDONED
+# TODO: make sure code that examines self.receive_phase handles ABANDONED
class TubConnectorClientFactory(protocol.ClientFactory, object):
# this is for internal use only. Application code should use
self.tub = parent
self.target = tubref
self.remainingLocations = self.target.getLocations()
- # attemptedLocations keeps track of where we've already try to
+ # attemptedLocations keeps track of where we've already tried to
# connect, so we don't try them twice.
self.attemptedLocations = []
def shutdown(self):
self.active = False
+ self.remainingLocations = []
self.stopConnectionTimer()
for c in self.pendingConnections.values():
c.disconnect()
+ # as each disconnect() finishes, it will either trigger our
+ # clientConnectionFailed or our negotiationFailed methods, both of
+ # which will trigger checkForIdle, and the last such message will
+ # invoke self.tub.connectorFinished()
def connectToAll(self):
while self.remainingLocations:
from zope.interface import implements
from twisted.internet import defer, protocol
from twisted.application import service, strports
+from twisted.python import log
from foolscap import ipb, base32, negotiate, broker, observer
from foolscap.referenceable import SturdyRef
assert self.running
self._activeConnectors.append(c)
def connectorFinished(self, c):
+ if c not in self._activeConnectors:
+ # TODO: I've seen this happen, but I can't figure out how it
+ # could possibly happen. Log and ignore rather than exploding
+ # when we try to do .remove, since this whole connector-tracking
+ # thing is mainly for the benefit of the unit tests (applications
+ # which never shut down a Tub aren't going to care), and it is
+ # more important to let application code run normally than to
+ # force an error here.
+ log.msg("Tub.connectorFinished: WEIRD, %s is not in %s"
+ % (c, self._activeConnectors))
+ return
self._activeConnectors.remove(c)
if not self.running and not self._activeConnectors:
self._allConnectorsAreFinished.fire(self)
def getReference(self, sturdyOrURL):
"""Acquire a RemoteReference for the given SturdyRef/URL.
+ The Tub must be running (i.e. Tub.startService()) when this is
+ invoked. Future releases may relax this requirement.
+
@return: a Deferred that fires with the RemoteReference
"""
+
+ assert self.running
+
if isinstance(sturdyOrURL, SturdyRef):
sturdy = sturdyOrURL
else:
connection goes away. At some point after it goes away, the
Reconnector will reconnect.
+ The Tub must be running (i.e. Tub.startService()) when this is
+ invoked. Future releases may relax this requirement.
+
I return a Reconnector object. When you no longer want to maintain
this connection, call the stopConnecting() method on the Reconnector.
I promise to not invoke your callback after you've called
rc.stopConnecting() # later
"""
+ assert self.running
rc = Reconnector(self, sturdyOrURL, cb, *args, **kwargs)
self.reconnectors.append(rc)
return rc
cb(rref, *args, **kwargs)
def _failed(self, f):
- # I'd like to trap NegotiationError and basic TCP connection
- # failures here, but not hide coding errors.
- if self.verbose:
- log.msg("Reconnector._failed: %s" % f)
+ # I'd like to quietly handle "normal" problems (basically TCP
+ # failures and NegotiationErrors that result from the target either
+ # not speaking Foolscap or not hosting the Tub that we want), but not
+ # hide coding errors or version mismatches.
+ log_it = self.verbose
+
# log certain unusual errors, even without self.verbose, to help
# people figure out why their reconnectors aren't connecting, since
# the usual getReference errback chain isn't active. This doesn't
# include ConnectError (which is a parent class of
- # ConnectionRefusedError)
+ # ConnectionRefusedError), so it won't fire if we just had a bad
+ # host/port, since the way we use connection hints will provoke that
+ # all the time.
if f.check(RemoteNegotiationError, NegotiationError):
- if not self.verbose:
- log.msg("Reconnector._failed: %s" % f)
+ log_it = True
+ if log_it:
+ log.msg("Reconnector._failed (furl=%s): %s" % (self._url, f))
if not self._active:
return
self._delay = min(self._delay * self.factor, self.maxDelay)
from foolscap import ipb, slicer, tokens, call
BananaError = tokens.BananaError
Violation = tokens.Violation
-from foolscap.constraint import IConstraint, StringConstraint
+from foolscap.constraint import IConstraint, ByteStringConstraint
from foolscap.remoteinterface import getRemoteInterface, \
getRemoteInterfaceByName, RemoteInterfaceConstraint
from foolscap.schema import constraintMap
clid = None
interfaceName = None
url = None
- inameConstraint = StringConstraint(200) # TODO: only known RI names?
- urlConstraint = StringConstraint(200)
+ inameConstraint = ByteStringConstraint(200) # TODO: only known RI names?
+ urlConstraint = ByteStringConstraint(200)
def checkToken(self, typebyte, size):
if self.state == 0:
state = 0
giftID = None
url = None
- urlConstraint = StringConstraint(200)
+ urlConstraint = ByteStringConstraint(200)
def checkToken(self, typebyte, size):
if self.state == 0:
self.tubID = None
self.location = url[:slash]
else:
- raise ValueError("unknown PB-URL prefix in '%s'" % url)
+ raise ValueError("unknown FURL prefix in %r" % (url,))
def getTubRef(self):
if self.encrypted:
__remote_name__: can be set to a string to specify the globally-unique
name for this interface. This should be a URL in a
namespace you administer. If not set, defaults to the
- fully qualified classname.
+ short classname.
RIFoo.names() returns the list of remote method names.
from foolscap.tokens import Violation, UnknownSchemaType
# make constraints available in a single location
-from foolscap.constraint import Constraint, Any, StringConstraint, \
+from foolscap.constraint import Constraint, Any, ByteStringConstraint, \
IntegerConstraint, NumberConstraint, \
UnboundedSchema, IConstraint, Optional, Shared
+from foolscap.slicers.unicode import UnicodeConstraint
from foolscap.slicers.bool import BooleanConstraint
from foolscap.slicers.dict import DictConstraint
from foolscap.slicers.list import ListConstraint
from foolscap.slicers.none import Nothing
# we don't import RemoteMethodSchema from remoteinterface.py, because
# remoteinterface.py needs to import us (for addToConstraintTypeMap)
-ignored = [Constraint, Any, StringConstraint, IntegerConstraint,
- NumberConstraint, BooleanConstraint, DictConstraint,
- ListConstraint, SetConstraint, TupleConstraint, Nothing,
- Optional, Shared,
+ignored = [Constraint, Any, ByteStringConstraint, UnicodeConstraint,
+ IntegerConstraint, NumberConstraint, BooleanConstraint,
+ DictConstraint, ListConstraint, SetConstraint, TupleConstraint,
+ Nothing, Optional, Shared,
] # hush pyflakes
# convenience shortcuts
SetOf = SetConstraint
+# note: using PolyConstraint (aka ChoiceOf) for inbound tasting is probably
+# not fully vetted. One of the issues would be with something like
+# ListOf(ChoiceOf(TupleOf(stuff), SetOf(stuff))). The ListUnslicer, when
+# handling an inbound Tuple, will do
+# TupleUnslicer.setConstraint(polyconstraint), since that's all it really
+# knows about, and the TupleUnslicer will then try to look inside the
+# polyconstraint for attributes that talk about tuples, and might fail.
+
class PolyConstraint(Constraint):
name = "PolyConstraint"
ChoiceOf = PolyConstraint
+def AnyStringConstraint(*args, **kwargs):
+ return ChoiceOf(ByteStringConstraint(*args, **kwargs),
+ UnicodeConstraint(*args, **kwargs))
+
+# keep the old meaning, for now. Eventually StringConstraint should become an
+# AnyStringConstraint
+StringConstraint = ByteStringConstraint
constraintMap = {
- str: StringConstraint(),
+ str: ByteStringConstraint(),
+ unicode: UnicodeConstraint(),
bool: BooleanConstraint(),
int: IntegerConstraint(),
long: IntegerConstraint(maxBytes=1024),
# -*- test-case-name: foolscap.test.test_banana -*-
+import re
from twisted.internet.defer import Deferred
-from foolscap.constraint import Any, StringConstraint
-from foolscap.tokens import BananaError, STRING
+from foolscap.tokens import BananaError, STRING, VOCAB, Violation
from foolscap.slicer import BaseSlicer, LeafUnslicer
+from foolscap.constraint import OpenerConstraint, Any, UnboundedSchema
class UnicodeSlicer(BaseSlicer):
opentype = ("unicode",)
def setConstraint(self, constraint):
if isinstance(constraint, Any):
return
- assert isinstance(constraint, StringConstraint)
+ assert isinstance(constraint, UnicodeConstraint)
self.constraint = constraint
def checkToken(self, typebyte, size):
- if typebyte != STRING:
+ if typebyte not in (STRING, VOCAB):
raise BananaError("UnicodeUnslicer only accepts strings")
- if self.constraint:
- self.constraint.checkToken(typebyte, size)
+ #if self.constraint:
+ # self.constraint.checkToken(typebyte, size)
def receiveChild(self, obj, ready_deferred=None):
assert not isinstance(obj, Deferred)
return self.string, None
def describe(self):
return "<unicode>"
+
+class UnicodeConstraint(OpenerConstraint):
+ """The object must be a unicode object. The maxLength and minLength
+ parameters restrict the number of characters (code points, *not* bytes)
+ that may be present in the object, which means that the on-wire (UTF-8)
+ representation may take up to 6 times as many bytes as characters.
+ """
+
+ strictTaster = True
+ opentypes = [("unicode",)]
+ name = "UnicodeConstraint"
+
+ def __init__(self, maxLength=1000, minLength=0, regexp=None):
+ self.maxLength = maxLength
+ self.minLength = minLength
+ # allow VOCAB in case the Banana-level tokenizer decides to tokenize
+ # the UTF-8 encoded body of a unicode object, since this is just as
+ # likely as tokenizing regular bytestrings. TODO: this is disabled
+ # because it doesn't currently work.. once I remember how Constraints
+ # work, I'll fix this. The current version is too permissive of
+ # tokens.
+ #self.taster = {STRING: 6*self.maxLength,
+ # VOCAB: None}
+ # regexp can either be a string or a compiled SRE_Match object..
+ # re.compile appears to notice SRE_Match objects and pass them
+ # through unchanged.
+ self.regexp = None
+ if regexp:
+ self.regexp = re.compile(regexp)
+
+ def checkObject(self, obj, inbound):
+ if not isinstance(obj, unicode):
+ raise Violation("not a String")
+ if self.maxLength != None and len(obj) > self.maxLength:
+ raise Violation("string too long (%d > %d)" %
+ (len(obj), self.maxLength))
+ if len(obj) < self.minLength:
+ raise Violation("string too short (%d < %d)" %
+ (len(obj), self.minLength))
+ if self.regexp:
+ if not self.regexp.search(obj):
+ raise Violation("regexp failed to match")
+
+ def maxSize(self, seen=None):
+ if self.maxLength == None:
+ raise UnboundedSchema
+ return self.OPENBYTES("unicode") + self.maxLength * 6
+
+ def maxDepth(self, seen=None):
+ return 1+1
# -*- test-case-name: foolscap.test.test_banana -*-
from twisted.internet.defer import Deferred
-from foolscap.constraint import Any, StringConstraint
+from foolscap.constraint import Any, ByteStringConstraint
from foolscap.tokens import Violation, BananaError, INT, STRING
from foolscap.slicer import BaseSlicer, BaseUnslicer, LeafUnslicer
from foolscap.slicer import BananaUnslicerRegistry
opentype = ('set-vocab',)
unslicerRegistry = BananaUnslicerRegistry
maxKeys = None
- valueConstraint = StringConstraint(100)
+ valueConstraint = ByteStringConstraint(100)
def setConstraint(self, constraint):
if isinstance(constraint, Any):
return
- assert isinstance(constraint, StringConstraint)
+ assert isinstance(constraint, ByteStringConstraint)
self.valueConstraint = constraint
def start(self, count):
unslicerRegistry = BananaUnslicerRegistry
index = None
value = None
- valueConstraint = StringConstraint(100)
+ valueConstraint = ByteStringConstraint(100)
def setConstraint(self, constraint):
if isinstance(constraint, Any):
return
- assert isinstance(constraint, StringConstraint)
+ assert isinstance(constraint, ByteStringConstraint)
self.valueConstraint = constraint
def checkToken(self, typebyte, size):
from foolscap.remoteinterface import getRemoteInterface, RemoteMethodSchema, \
UnconstrainedMethod
from foolscap.schema import Any, SetOf, DictOf, ListOf, TupleOf, \
- NumberConstraint, StringConstraint, IntegerConstraint
+ NumberConstraint, ByteStringConstraint, IntegerConstraint, \
+ UnicodeConstraint
from twisted.python import failure
from twisted.internet.main import CONNECTION_DONE
MegaSchema1 = DictOf(str,
ListOf(TupleOf(SetOf(int, maxLength=10, mutable=True),
str, bool, int, long, float, None,
+ UnicodeConstraint(),
+ ByteStringConstraint(),
Any(), NumberConstraint(),
IntegerConstraint(),
- StringConstraint(maxLength=100,
- minLength=90,
- regexp="\w+"),
- StringConstraint(regexp=Digits),
+ ByteStringConstraint(maxLength=100,
+ minLength=90,
+ regexp="\w+"),
+ ByteStringConstraint(regexp=Digits),
),
maxLength=20),
maxKeys=5)
def getName(): return str
disputed = RemoteMethodSchema(_response=int, a=int)
def fail(): return str # actually raises an exception
+ def failstring(): return str # raises a string exception
class RIMyTarget2(RemoteInterface):
__remote_name__ = "RIMyTargetInterface2"
return 24
def remote_fail(self):
raise ValueError("you asked me to fail")
+ def remote_failstring(self):
+ raise "string exceptions are annoying"
class TargetWithoutInterfaces(Target):
# undeclare the RIMyTarget interface
# would be a string but the header is too long
s = "\x01" * 66 + "\x82" + "stupidly long string"
f = self.shouldDropConnection(s)
- self.failUnlessEqual(f.value.args[0],
- "token prefix is limited to 64 bytes")
+ self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes"))
def testLongHeader2(self):
# bad string while discarding
s = bOPEN("errorful",0) + bINT(1) + s + bINT(2) + bCLOSE(0)
self.banana.mode = "start"
f = self.shouldDropConnection(s)
- self.failUnlessEqual(f.value.args[0],
- "token prefix is limited to 64 bytes")
+ self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes"))
def testCheckToken1(self):
# violation raised in top.openerCheckToken
self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes"))
f = self.shouldDropConnection("\x00" * 65 + "\x82")
self.failUnlessEqual(f.value.where, "<RootUnslicer>")
- self.failUnlessEqual(f.value.args[0],
- "token prefix is limited to 64 bytes")
+ self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes"))
self.check("a", "\x01\x82a")
self.check("b"*130, "\x02\x01\x82" + "b"*130 + "extra")
import gc
import re
import sets
+import sys
if False:
- import sys
from twisted.python import log
log.startLogging(sys.stderr)
-from twisted.python import failure
+from twisted.python import failure, log
from twisted.internet import reactor, defer
from twisted.trial import unittest
from twisted.internet.main import CONNECTION_LOST
self.failUnlessSubstring("TargetWithoutInterfaces", str(f))
self.failUnlessSubstring(" has no attribute 'remote_bogus'", str(f))
+ def testFailStringException(self):
+ # make sure we handle string exceptions correctly
+ if sys.version_info >= (2,5):
+ log.msg("skipping test: string exceptions are deprecated in 2.5")
+ return
+ rr, target = self.setupTarget(TargetWithoutInterfaces())
+ d = rr.callRemote("failstring")
+ self.failIf(target.calls)
+ d.addBoth(self._testFailStringException_1)
+ return d
+ def _testFailStringException_1(self, f):
+ # f should be a CopiedFailure
+ self.failUnless(isinstance(f, failure.Failure),
+ "Hey, we didn't fail: %s" % f)
+ self.failUnless(f.check("string exceptions are annoying"),
+ "wrong exception type: %s" % f)
+
+
def testCall2(self):
# server end uses an interface this time, but not the client end
rr, target = self.setupTarget(Target(), True)
rr, target = self.setupTarget(HelperTarget())
t = (sets.Set([1, 2, 3]),
"str", True, 12, 12L, 19.3, None,
+ u"unicode",
+ "bytestring",
"any", 14.3,
15,
"a"*95,
testFailWrongReturnLocal.timeout = 2
def _testFailWrongReturnLocal_1(self, f):
self.failUnless(f.check(Violation))
- self.failUnlessSubstring("INT token rejected by StringConstraint",
+ self.failUnlessSubstring("INT token rejected by ByteStringConstraint",
str(f))
self.failUnlessSubstring("in inbound method results", str(f))
self.failUnlessSubstring("<RootUnslicer>.Answer(req=1)", str(f))
return d
testDefer.timeout = 2
- def testDisconnect1(self):
+ def testDisconnect_during_call(self):
rr, target = self.setupTarget(HelperTarget())
d = rr.callRemote("hang")
e = RuntimeError("lost connection")
d.addCallbacks(lambda res: self.fail("should have failed"),
lambda why: why.trap(RuntimeError) and None)
return d
- testDisconnect1.timeout = 2
def disconnected(self, *args, **kwargs):
self.lost = 1
self.lost_args = (args, kwargs)
- def testDisconnect2(self):
+ def testNotifyOnDisconnect(self):
rr, target = self.setupTarget(HelperTarget())
self.lost = 0
rr.notifyOnDisconnect(self.disconnected)
def _check(res):
self.failUnless(self.lost)
self.failUnlessEqual(self.lost_args, ((),{}))
+ # it should be safe to unregister now, even though the callback
+ # has already fired, since dontNotifyOnDisconnect is tolerant
+ rr.dontNotifyOnDisconnect(self.disconnected)
d.addCallback(_check)
return d
- def testDisconnect3(self):
+ def testNotifyOnDisconnect_unregister(self):
rr, target = self.setupTarget(HelperTarget())
self.lost = 0
m = rr.notifyOnDisconnect(self.disconnected)
rr.dontNotifyOnDisconnect(m)
+ # dontNotifyOnDisconnect is supposed to be tolerant of duplicate
+ # unregisters, because otherwise it is hard to avoid race conditions.
+ # Validate that we can unregister something multiple times.
+ rr.dontNotifyOnDisconnect(m)
rr.tracker.broker.transport.loseConnection(CONNECTION_LOST)
d = flushEventualQueue()
d.addCallback(lambda res: self.failIf(self.lost))
return d
- def testDisconnect4(self):
+ def testNotifyOnDisconnect_args(self):
rr, target = self.setupTarget(HelperTarget())
self.lost = 0
rr.notifyOnDisconnect(self.disconnected, "arg", foo="kwarg")
d.addCallback(_check)
return d
+ def testNotifyOnDisconnect_already(self):
+ # make sure notifyOnDisconnect works even if the reference was already
+ # broken
+ rr, target = self.setupTarget(HelperTarget())
+ self.lost = 0
+ rr.tracker.broker.transport.loseConnection(CONNECTION_LOST)
+ d = flushEventualQueue()
+ d.addCallback(lambda res: rr.notifyOnDisconnect(self.disconnected))
+ d.addCallback(lambda res: flushEventualQueue())
+ def _check(res):
+ self.failUnless(self.lost, "disconnect handler not run")
+ self.failUnlessEqual(self.lost_args, ((),{}))
+ d.addCallback(_check)
+ return d
+
def testUnsendable(self):
rr, target = self.setupTarget(HelperTarget())
d = rr.callRemote("set", obj=Unsendable())
d = defer.maybeDeferred(s1.stopService)
d.addCallback(self._testPersist_1, s1, s2, t1, public_url, port)
return d
- testPersist.timeout = 10
+ testPersist.timeout = 5
def _testPersist_1(self, res, s1, s2, t1, public_url, port):
self.services.remove(s1)
s3 = Tub(certData=s1.getCertData())
d.addCallback(lambda ref: ref.callRemote('add', a=2, b=2))
d.addCallback(self._testShared_1)
return d
- testShared.timeout = 10
+ testShared.timeout = 5
def _testShared_1(self, res):
t1,t2 = self.targets
self.failUnlessEqual(t1.calls, [(1,1)])
self.conforms(c, -2**512+1)
self.violates(c, -2**512)
- def testString(self):
- c = schema.StringConstraint(10)
+ def testByteString(self):
+ c = schema.ByteStringConstraint(10)
self.assertSize(c, STR10)
self.assertSize(c, STR10) # twice to test seen=[] logic
self.assertDepth(c, 1)
self.violates(c, Dummy())
self.violates(c, None)
- c2 = schema.StringConstraint(15, 10)
+ c2 = schema.ByteStringConstraint(15, 10)
self.violates(c2, "too short")
self.conforms(c2, "long enough")
self.violates(c2, "this is too long")
+ self.violates(c2, u"I am unicode")
- c3 = schema.StringConstraint(regexp="needle")
+ c3 = schema.ByteStringConstraint(regexp="needle")
self.violates(c3, "no present")
self.conforms(c3, "needle in a haystack")
- c4 = schema.StringConstraint(regexp="[abc]+")
+ c4 = schema.ByteStringConstraint(regexp="[abc]+")
self.violates(c4, "spelled entirely without those letters")
self.conforms(c4, "add better cases")
- c5 = schema.StringConstraint(regexp=re.compile("\d+\s\w+"))
+ c5 = schema.ByteStringConstraint(regexp=re.compile("\d+\s\w+"))
self.conforms(c5, ": 123 boo")
self.violates(c5, "more than 1 spaces")
self.violates(c5, "letters first 123")
+ def testString(self):
+ # this test will change once the definition of "StringConstraint"
+ # changes. For now, we assert that StringConstraint is the same as
+ # ByteStringConstraint.
+
+ c = schema.StringConstraint(20)
+ self.conforms(c, "I'm short")
+ self.violates(c, u"I am unicode")
+
+ def testUnicode(self):
+ c = schema.UnicodeConstraint(10)
+ #self.assertSize(c, USTR10)
+ #self.assertSize(c, USTR10) # twice to test seen=[] logic
+ self.assertDepth(c, 2)
+ self.violates(c, "I'm a bytestring")
+ self.conforms(c, u"I'm short")
+ self.violates(c, u"I am too long")
+ self.conforms(c, u"a" * 10)
+ self.violates(c, u"a" * 11)
+ self.violates(c, 123)
+ self.violates(c, Dummy())
+ self.violates(c, None)
+
+ c2 = schema.UnicodeConstraint(15, 10)
+ self.violates(c2, "I'm a bytestring")
+ self.violates(c2, u"too short")
+ self.conforms(c2, u"long enough")
+ self.violates(c2, u"this is too long")
+
+ c3 = schema.UnicodeConstraint(regexp="needle")
+ self.violates(c3, "I'm a bytestring")
+ self.violates(c3, u"no present")
+ self.conforms(c3, u"needle in a haystack")
+ c4 = schema.UnicodeConstraint(regexp="[abc]+")
+ self.violates(c4, "I'm a bytestring")
+ self.violates(c4, u"spelled entirely without those letters")
+ self.conforms(c4, u"add better cases")
+ c5 = schema.UnicodeConstraint(regexp=re.compile("\d+\s\w+"))
+ self.violates(c5, "I'm a bytestring")
+ self.conforms(c5, u": 123 boo")
+ self.violates(c5, u"more than 1 spaces")
+ self.violates(c5, u"letters first 123")
+
def testBool(self):
c = schema.BooleanConstraint()
self.assertSize(c, 147)
self.violates(c, None)
def testPoly(self):
- c = schema.PolyConstraint(schema.StringConstraint(100),
+ c = schema.PolyConstraint(schema.ByteStringConstraint(100),
schema.IntegerConstraint())
self.assertSize(c, 165)
self.assertDepth(c, 1)
def testTuple(self):
- c = schema.TupleConstraint(schema.StringConstraint(10),
- schema.StringConstraint(100),
+ c = schema.TupleConstraint(schema.ByteStringConstraint(10),
+ schema.ByteStringConstraint(100),
schema.IntegerConstraint() )
self.conforms(c, ("hi", "there buddy, you're number", 1))
self.violates(c, "nope")
self.assertDepth(c, 2)
def testNestedTuple(self):
- inner = schema.TupleConstraint(schema.StringConstraint(10),
+ inner = schema.TupleConstraint(schema.ByteStringConstraint(10),
schema.IntegerConstraint())
self.assertSize(inner, 72+75+73)
self.assertDepth(inner, 2)
- outer = schema.TupleConstraint(schema.StringConstraint(100),
+ outer = schema.TupleConstraint(schema.ByteStringConstraint(100),
inner)
self.assertSize(outer, 72+165 + 72+75+73)
self.assertDepth(outer, 3)
self.violates(outer2, ("hi", 1, "flat", 2) )
def testUnbounded(self):
- big = schema.StringConstraint(None)
+ big = schema.ByteStringConstraint(None)
self.assertUnboundedSize(big)
self.assertDepth(big, 1)
self.conforms(big, "blah blah blah blah blah" * 1024)
def testRecursion(self):
# we have to fiddle with PolyConstraint's innards
- value = schema.ChoiceOf(schema.StringConstraint(),
+ value = schema.ChoiceOf(schema.ByteStringConstraint(),
schema.IntegerConstraint(),
# will add 'value' here
)
self.conforms(value, 123)
self.violates(value, [])
- mapping = schema.TupleConstraint(schema.StringConstraint(10),
+ mapping = schema.TupleConstraint(schema.ByteStringConstraint(10),
value)
self.assertSize(mapping, 72+75+1065)
self.assertDepth(mapping, 2)
self.violates(mapping, ("name", l))
def testList(self):
- l = schema.ListOf(schema.StringConstraint(10))
+ l = schema.ListOf(schema.ByteStringConstraint(10))
self.assertSize(l, 71 + 30*75)
self.assertDepth(l, 2)
self.conforms(l, ["one", "two", "three"])
self.violates(l, [0, "numbers", "allowed"])
self.conforms(l, ["short", "sweet"])
- l2 = schema.ListOf(schema.StringConstraint(10), 3)
+ l2 = schema.ListOf(schema.ByteStringConstraint(10), 3)
self.assertSize(l2, 71 + 3*75)
self.assertDepth(l2, 2)
self.conforms(l2, ["the number", "shall be", "three"])
self.violates(l2, ["five", "is", "...", "right", "out"])
- l3 = schema.ListOf(schema.StringConstraint(10), None)
+ l3 = schema.ListOf(schema.ByteStringConstraint(10), None)
self.assertUnboundedSize(l3)
self.assertDepth(l3, 2)
self.conforms(l3, ["long"] * 35)
self.violates(l3, ["number", 1, "rule", "is", 0, "numbers"])
- l4 = schema.ListOf(schema.StringConstraint(10), 3, 3)
+ l4 = schema.ListOf(schema.ByteStringConstraint(10), 3, 3)
self.conforms(l4, ["three", "is", "good"])
self.violates(l4, ["but", "four", "is", "bad"])
self.violates(l4, ["two", "too"])
def testDict(self):
- d = schema.DictOf(schema.StringConstraint(10),
+ d = schema.DictOf(schema.ByteStringConstraint(10),
schema.IntegerConstraint(),
maxKeys=4)
self.failUnlessEqual(c.maxBytes, -1)
c = make(str)
- self.check(c, schema.StringConstraint)
+ self.check(c, schema.ByteStringConstraint)
+ self.failUnlessEqual(c.maxLength, 1000)
+
+ c = make(unicode)
+ self.check(c, schema.UnicodeConstraint)
self.failUnlessEqual(c.maxLength, 1000)
self.check(make(bool), schema.BooleanConstraint)
c = make((int, str))
self.check(c, schema.TupleConstraint)
self.check(c.constraints[0], schema.IntegerConstraint)
- self.check(c.constraints[1], schema.StringConstraint)
+ self.check(c.constraints[1], schema.ByteStringConstraint)
c = make(common.RIHelper)
self.check(c, RemoteInterfaceConstraint)
self.failUnless(isinstance(getkw("c")[1], schema.IntegerConstraint))
self.failUnless(isinstance(r.getResponseConstraint(),
- schema.StringConstraint))
+ schema.ByteStringConstraint))
self.failUnless(isinstance(getkw("c", 1, [])[1],
schema.IntegerConstraint))
--- /dev/null
+#! /usr/bin/env python
+"""
+figleaf is another tool to trace code coverage (yes, in Python ;).
+
+figleaf uses the sys.settrace hook to record which statements are
+executed by the CPython interpreter; this record can then be saved
+into a file, or otherwise communicated back to a reporting script.
+
+figleaf differs from the gold standard of coverage tools
+('coverage.py') in several ways. First and foremost, figleaf uses the
+same criterion for "interesting" lines of code as the sys.settrace
+function, which obviates some of the complexity in coverage.py (but
+does mean that your "loc" count goes down). Second, figleaf does not
+record code executed in the Python standard library, which results in
+a significant speedup. And third, the format in which the coverage
+format is saved is very simple and easy to work with.
+
+You might want to use figleaf if you're recording coverage from
+multiple types of tests and need to aggregate the coverage in
+interesting ways, and/or control when coverage is recorded.
+coverage.py is a better choice for command-line execution, and its
+reporting is a fair bit nicer.
+
+Command line usage: ::
+
+ figleaf.py <python file to execute> <args to python file>
+
+The figleaf output is saved into the file '.figleaf', which is an
+*aggregate* of coverage reports from all figleaf runs from this
+directory. '.figleaf' contains a pickled dictionary of sets; the keys
+are source code filenames, and the sets contain all line numbers
+executed by the Python interpreter. See the docs or command-line
+programs in bin/ for more information.
+
+High level API: ::
+
+ * ``start(ignore_lib=True)`` -- start recording code coverage.
+ * ``stop()`` -- stop recording code coverage.
+ * ``get_trace_obj()`` -- return the (singleton) trace object.
+ * ``get_info()`` -- get the coverage dictionary
+
+Classes & functions worth knowing about, i.e. a lower level API:
+
+ * ``get_lines(fp)`` -- return the set of interesting lines in the fp.
+ * ``combine_coverage(d1, d2)`` -- combine coverage info from two dicts.
+ * ``read_coverage(filename)`` -- load the coverage dictionary
+ * ``write_coverage(filename)`` -- write the coverage out.
+ * ``annotate_coverage(...)`` -- annotate a Python file with its coverage info.
+
+Known problems:
+
+ -- module docstrings are *covered* but not found.
+
+AUTHOR: C. Titus Brown, titus@idyll.org
+
+'figleaf' is Copyright (C) 2006. It will be released under the BSD license.
+"""
+import sys
+import os
+import threading
+from cPickle import dump, load
+
+### import builtin sets if in > 2.4, otherwise use 'sets' module.
+# we require 2.4 or later
+assert set
+
+
+from token import tok_name, NEWLINE, STRING, INDENT, DEDENT, COLON
+import parser, types, symbol
+
+def get_token_name(x):
+ """
+ Utility to help pretty-print AST symbols/Python tokens.
+ """
+ if symbol.sym_name.has_key(x):
+ return symbol.sym_name[x]
+ return tok_name.get(x, '-')
+
+class LineGrabber:
+ """
+ Count 'interesting' lines of Python in source files, where
+ 'interesting' is defined as 'lines that could possibly be
+ executed'.
+
+ @CTB this badly needs to be refactored... once I have automated
+ tests ;)
+ """
+ def __init__(self, fp):
+ """
+ Count lines of code in 'fp'.
+ """
+ self.lines = set()
+
+ self.ast = parser.suite(fp.read())
+ self.tree = parser.ast2tuple(self.ast, True)
+
+ self.find_terminal_nodes(self.tree)
+
+ def find_terminal_nodes(self, tup):
+ """
+ Recursively eat an AST in tuple form, finding the first line
+ number for "interesting" code.
+ """
+ (sym, rest) = tup[0], tup[1:]
+
+ line_nos = set()
+ if type(rest[0]) == types.TupleType: ### node
+
+ for x in rest:
+ token_line_no = self.find_terminal_nodes(x)
+ if token_line_no is not None:
+ line_nos.add(token_line_no)
+
+ if symbol.sym_name[sym] in ('stmt', 'suite', 'lambdef',
+ 'except_clause') and line_nos:
+ # store the line number that this statement started at
+ self.lines.add(min(line_nos))
+ elif symbol.sym_name[sym] in ('if_stmt',):
+ # add all lines under this
+ self.lines.update(line_nos)
+ elif symbol.sym_name[sym] in ('global_stmt',): # IGNORE
+ return
+ else:
+ if line_nos:
+ return min(line_nos)
+
+ else: ### leaf
+ if sym not in (NEWLINE, STRING, INDENT, DEDENT, COLON) and \
+ tup[1] != 'else':
+ return tup[2]
+ return None
+
+ def pretty_print(self, tup=None, indent=0):
+ """
+ Pretty print the AST.
+ """
+ if tup is None:
+ tup = self.tree
+
+ s = tup[1]
+
+ if type(s) == types.TupleType:
+ print ' '*indent, get_token_name(tup[0])
+ for x in tup[1:]:
+ self.pretty_print(x, indent+1)
+ else:
+ print ' '*indent, get_token_name(tup[0]), tup[1:]
+
+def get_lines(fp):
+ """
+ Return the set of interesting lines in the source code read from
+ this file handle.
+ """
+ l = LineGrabber(fp)
+ return l.lines
+
+class CodeTracer:
+ """
+ Basic code coverage tracking, using sys.settrace.
+ """
+ def __init__(self, ignore_prefixes=[]):
+ self.c = {}
+ self.started = False
+ self.ignore_prefixes = ignore_prefixes
+
+ def start(self):
+ """
+ Start recording.
+ """
+ if not self.started:
+ self.LOG = open("/tmp/flog.out", "w")
+ self.started = True
+
+ sys.settrace(self.g)
+ if hasattr(threading, 'settrace'):
+ threading.settrace(self.g)
+
+ def stop(self):
+ if self.started:
+ sys.settrace(None)
+ if hasattr(threading, 'settrace'):
+ threading.settrace(None)
+
+ self.started = False
+
+ def g(self, f, e, a):
+ """
+ global trace function.
+ """
+ if e is 'call':
+ for p in self.ignore_prefixes:
+ if f.f_code.co_filename.startswith(p):
+ return
+
+ return self.t
+
+ def t(self, f, e, a):
+ """
+ local trace function.
+ """
+
+ if e is 'line':
+ self.c[(f.f_code.co_filename, f.f_lineno)] = 1
+ return self.t
+
+ def clear(self):
+ """
+ wipe out coverage info
+ """
+
+ self.c = {}
+
+ def gather_files(self):
+ """
+ Return the dictionary of lines of executed code; the dict
+ contains items (k, v), where 'k' is the filename and 'v'
+ is a set of line numbers.
+ """
+ files = {}
+ for (filename, line) in self.c.keys():
+ d = files.get(filename, set())
+ d.add(line)
+ files[filename] = d
+
+ return files
+
+def combine_coverage(d1, d2):
+ """
+ Given two coverage dictionaries, combine the recorded coverage
+ and return a new dictionary.
+ """
+ keys = set(d1.keys())
+ keys.update(set(d2.keys()))
+
+ new_d = {}
+ for k in keys:
+ v = d1.get(k, set())
+ v2 = d2.get(k, set())
+
+ s = set(v)
+ s.update(v2)
+ new_d[k] = s
+
+ return new_d
+
+def write_coverage(filename, combine=True):
+ """
+ Write the current coverage info out to the given filename. If
+ 'combine' is false, destroy any previously recorded coverage info.
+ """
+ if _t is None:
+ return
+
+ d = _t.gather_files()
+
+ # combine?
+ if combine:
+ old = {}
+ fp = None
+ try:
+ fp = open(filename)
+ except IOError:
+ pass
+
+ if fp:
+ old = load(fp)
+ fp.close()
+ d = combine_coverage(d, old)
+
+ # ok, save.
+ outfp = open(filename, 'w')
+ try:
+ dump(d, outfp)
+ finally:
+ outfp.close()
+
+def read_coverage(filename):
+ """
+ Read a coverage dictionary in from the given file.
+ """
+ fp = open(filename)
+ try:
+ d = load(fp)
+ finally:
+ fp.close()
+
+ return d
+
+def annotate_coverage(in_fp, out_fp, covered, all_lines,
+ mark_possible_lines=False):
+ """
+ A simple example coverage annotator that outputs text.
+ """
+ for i, line in enumerate(in_fp):
+ i = i + 1
+
+ if i in covered:
+ symbol = '>'
+ elif i in all_lines:
+ symbol = '!'
+ else:
+ symbol = ' '
+
+ symbol2 = ''
+ if mark_possible_lines:
+ symbol2 = ' '
+ if i in all_lines:
+ symbol2 = '-'
+
+ out_fp.write('%s%s %s' % (symbol, symbol2, line,))
+
+#######################
+
+#
+# singleton functions/top-level API
+#
+
+_t = None
+
+def start(ignore_python_lib=True, ignore_prefixes=[]):
+ """
+ Start tracing code coverage. If 'ignore_python_lib' is True,
+ ignore all files that live below the same directory as the 'os'
+ module.
+ """
+ global _t
+ if _t is None:
+ ignore_prefixes = ignore_prefixes[:]
+ if ignore_python_lib:
+ ignore_prefixes.append(os.path.realpath(os.path.dirname(os.__file__)))
+ _t = CodeTracer(ignore_prefixes)
+
+ _t.start()
+
+def stop():
+ """
+ Stop tracing code coverage.
+ """
+ global _t
+ if _t is not None:
+ _t.stop()
+
+def get_trace_obj():
+ """
+ Return the (singleton) trace object, if it exists.
+ """
+ return _t
+
+def get_info():
+ """
+ Get the coverage dictionary from the trace object.
+ """
+ if _t:
+ return _t.gather_files()
+
+#############
+
+def display_ast():
+ l = LineGrabber(open(sys.argv[1]))
+ l.pretty_print()
+
+def main():
+ """
+ Execute the given Python file with coverage, making it look like it is
+ __main__.
+ """
+ ignore_pylibs = False
+
+ def print_help():
+ print 'Usage: figleaf [-i] <program-to-profile> <program-options>'
+ print ''
+ print 'Options:'
+ print ' -i Ignore Python standard libraries when calculating coverage'
+
+ args = sys.argv[1:]
+
+ if len(args) < 1:
+ print_help()
+ raise SystemExit()
+ elif len(args) > 2 and args[0] == '-i':
+ ignore_pylibs = True
+
+ ## Make sure to strip off the -i or --ignore-python-libs option if it exists
+ args = args[1:]
+
+ ## Reset system args so that the subsequently exec'd file can read from sys.argv
+ sys.argv = args
+
+ sys.path[0] = os.path.dirname(args[0])
+
+ cwd = os.getcwd()
+
+ start(ignore_pylibs) # START code coverage
+
+ import __main__
+ try:
+ execfile(args[0], __main__.__dict__)
+ finally:
+ stop() # STOP code coverage
+
+ write_coverage(os.path.join(cwd, '.figleaf'))
--- /dev/null
+#! /usr/bin/env python
+import figleaf_htmlizer
+figleaf_htmlizer.main()
--- /dev/null
+#! /usr/bin/env python
+import sys
+import figleaf
+import os
+import re
+
+from optparse import OptionParser
+
+import logging
+logging.basicConfig(level=logging.DEBUG)
+
+logger = logging.getLogger('figleaf.htmlizer')
+
+def read_exclude_patterns(f):
+ if not f:
+ return []
+ exclude_patterns = []
+
+ fp = open(f)
+ for line in fp:
+ line = line.rstrip()
+ if line and not line.startswith('#'):
+ pattern = re.compile(line)
+ exclude_patterns.append(pattern)
+
+ return exclude_patterns
+
+def report_as_html(coverage, directory, exclude_patterns=[], root=None):
+ ### now, output.
+
+ keys = coverage.keys()
+ info_dict = {}
+ for k in keys:
+ skip = False
+ for pattern in exclude_patterns:
+ if pattern.search(k):
+ logger.debug('SKIPPING %s -- matches exclusion pattern' % k)
+ skip = True
+ break
+
+ if skip:
+ continue
+
+ if k.endswith('figleaf.py'):
+ continue
+
+ display_filename = k
+ if root:
+ if not k.startswith(root):
+ continue
+ display_filename = k[len(root):]
+ assert not display_filename.startswith("/")
+ assert display_filename.endswith(".py")
+ display_filename = display_filename[:-3] # trim .py
+ display_filename = display_filename.replace("/", ".")
+
+ if not k.startswith("/"):
+ continue
+
+ try:
+ pyfile = open(k)
+# print 'opened', k
+ except IOError:
+ logger.warning('CANNOT OPEN: %s' % k)
+ continue
+
+ try:
+ lines = figleaf.get_lines(pyfile)
+ except KeyboardInterrupt:
+ raise
+ except Exception, e:
+ pyfile.close()
+ logger.warning('ERROR: %s %s' % (k, str(e)))
+ continue
+
+ # ok, got all the info. now annotate file ==> html.
+
+ covered = coverage[k]
+ n_covered = n_lines = 0
+
+ pyfile = open(k)
+ output = []
+ for i, line in enumerate(pyfile):
+ is_covered = False
+ is_line = False
+
+ i += 1
+
+ if i in covered:
+ is_covered = True
+
+ n_covered += 1
+ n_lines += 1
+ elif i in lines:
+ is_line = True
+
+ n_lines += 1
+
+ color = 'black'
+ if is_covered:
+ color = 'green'
+ elif is_line:
+ color = 'red'
+
+ line = escape_html(line.rstrip())
+ output.append('<font color="%s">%4d. %s</font>' % (color, i, line.rstrip()))
+
+ try:
+ pcnt = n_covered * 100. / n_lines
+ except ZeroDivisionError:
+ pcnt = 100
+ info_dict[k] = (n_lines, n_covered, pcnt, display_filename)
+
+ html_outfile = make_html_filename(display_filename)
+ html_outfp = open(os.path.join(directory, html_outfile), 'w')
+ html_outfp.write('source file: <b>%s</b><br>\n' % (k,))
+ html_outfp.write('file stats: <b>%d lines, %d executed: %.1f%% covered</b>\n' % (n_lines, n_covered, pcnt))
+
+ html_outfp.write('<pre>\n')
+ html_outfp.write("\n".join(output))
+ html_outfp.close()
+
+ ### print a summary, too.
+
+ info_dict_items = info_dict.items()
+
+ def sort_by_pcnt(a, b):
+ a = a[1][2]
+ b = b[1][2]
+
+ return -cmp(a,b)
+ info_dict_items.sort(sort_by_pcnt)
+
+ summary_lines = sum([ v[0] for (k, v) in info_dict_items])
+ summary_cover = sum([ v[1] for (k, v) in info_dict_items])
+
+ summary_pcnt = 100
+ if summary_lines:
+ summary_pcnt = float(summary_cover) * 100. / float(summary_lines)
+
+
+ pcnts = [ float(v[1]) * 100. / float(v[0]) for (k, v) in info_dict_items if v[0] ]
+ pcnt_90 = [ x for x in pcnts if x >= 90 ]
+ pcnt_75 = [ x for x in pcnts if x >= 75 ]
+ pcnt_50 = [ x for x in pcnts if x >= 50 ]
+
+ stats_fp = open('%s/stats.out' % (directory,), 'w')
+ stats_fp.write("total files: %d\n" % len(pcnts))
+ stats_fp.write("total source lines: %d\n" % summary_lines)
+ stats_fp.write("total covered lines: %d\n" % summary_cover)
+ stats_fp.write("total coverage percentage: %.1f\n" % summary_pcnt)
+ stats_fp.close()
+
+ ## index.html
+ index_fp = open('%s/index.html' % (directory,), 'w')
+ # summary info
+ index_fp.write('<title>figleaf code coverage report</title>\n')
+ index_fp.write('<h2>Summary</h2> %d files total: %d files > '
+ '90%%, %d files > 75%%, %d files > 50%%<p>'
+ % (len(pcnts), len(pcnt_90),
+ len(pcnt_75), len(pcnt_50)))
+ # sorted by percentage covered
+ index_fp.write('<h3>Sorted by Coverage Percentage</h3>\n')
+ index_fp.write('<table border=1><tr><th>Filename</th>'
+ '<th># lines</th><th># covered</th>'
+ '<th>% covered</th></tr>\n')
+ index_fp.write('<tr><td><b>totals:</b></td><td><b>%d</b></td>'
+ '<td><b>%d</b></td><td><b>%.1f%%</b></td></tr>'
+ '<tr></tr>\n'
+ % (summary_lines, summary_cover, summary_pcnt,))
+
+ for filename, stuff in info_dict_items:
+ (n_lines, n_covered, percent_covered, display_filename) = stuff
+ html_outfile = make_html_filename(display_filename)
+
+ index_fp.write('<tr><td><a href="./%s">%s</a></td>'
+ '<td>%d</td><td>%d</td><td>%.1f</td></tr>\n'
+ % (html_outfile, display_filename, n_lines,
+ n_covered, percent_covered,))
+
+ index_fp.write('</table>\n')
+
+ # sorted by module name
+ index_fp.write('<h3>Sorted by Module Name (alphabetical)</h3>\n')
+ info_dict_items.sort()
+ index_fp.write('<table border=1><tr><th>Filename</th>'
+ '<th># lines</th><th># covered</th>'
+ '<th>% covered</th></tr>\n')
+
+ for filename, stuff in info_dict_items:
+ (n_lines, n_covered, percent_covered, display_filename) = stuff
+ html_outfile = make_html_filename(display_filename)
+
+ index_fp.write('<tr><td><a href="./%s">%s</a></td>'
+ '<td>%d</td><td>%d</td><td>%.1f</td></tr>\n'
+ % (html_outfile, display_filename, n_lines,
+ n_covered, percent_covered,))
+
+ index_fp.write('</table>\n')
+
+ index_fp.close()
+
+ logger.info('reported on %d file(s) total\n' % len(info_dict))
+ return len(info_dict)
+
+def prepare_reportdir(dirname='html'):
+ try:
+ os.mkdir(dirname)
+ except OSError: # already exists
+ pass
+
+def make_html_filename(orig):
+ return orig + ".html"
+
+def escape_html(s):
+ s = s.replace("&", "&")
+ s = s.replace("<", "<")
+ s = s.replace(">", ">")
+ s = s.replace('"', """)
+ return s
+
+def main():
+ ###
+
+ option_parser = OptionParser()
+
+ option_parser.add_option('-x', '--exclude-patterns', action="store",
+ dest="exclude_patterns_file",
+ help="file containing regexp patterns to exclude")
+
+ option_parser.add_option('-d', '--output-directory', action='store',
+ dest="output_dir",
+ default = "html",
+ help="directory for HTML output")
+ option_parser.add_option('-r', '--root', action="store",
+ dest="root",
+ default=None,
+ help="only pay attention to modules under this directory")
+
+ option_parser.add_option('-q', '--quiet', action='store_true', dest='quiet', help='Suppress all but error messages')
+
+ (options, args) = option_parser.parse_args()
+
+ if options.quiet:
+ logging.disable(logging.DEBUG)
+
+ if options.root:
+ options.root = os.path.abspath(options.root)
+ if options.root[-1] != "/":
+ options.root = options.root + "/"
+
+ ### load
+
+ if not args:
+ args = ['.figleaf']
+
+ coverage = {}
+ for filename in args:
+ logger.debug("loading coverage info from '%s'\n" % (filename,))
+ d = figleaf.read_coverage(filename)
+ coverage = figleaf.combine_coverage(coverage, d)
+
+ if not coverage:
+ logger.warning('EXITING -- no coverage info!\n')
+ sys.exit(-1)
+
+ ### make directory
+ prepare_reportdir(options.output_dir)
+ report_as_html(coverage, options.output_dir,
+ read_exclude_patterns(options.exclude_patterns_file),
+ options.root)
+
--- /dev/null
+
+"""A Trial IReporter plugin that gathers figleaf code-coverage information.
+
+Once this plugin is installed, trial can be invoked with one of two new
+--reporter options:
+
+ trial --reporter=verbose-figleaf ARGS
+ trial --reporter-bwverbose-figleaf ARGS
+
+Once such a test run has finished, there will be a .figleaf file in the
+top-level directory. This file can be turned into a directory of .html files
+(with index.html as the starting point) by running:
+
+ figleaf2html -d OUTPUTDIR [-x EXCLUDEFILE]
+
+Figleaf thinks of everyting in terms of absolute filenames rather than
+modules. The EXCLUDEFILE may be necessary to keep it from providing reports
+on non-Code-Under-Test files that live in unusual locations. In particular,
+if you use extra PYTHONPATH arguments to point at some alternate version of
+an upstream library (like Twisted), or if something like debian's
+python-support puts symlinks to .py files in sys.path but not the .py files
+themselves, figleaf will present coverage information on both of these. The
+EXCLUDEFILE option might help to inhibit these.
+
+Other figleaf problems:
+
+ the annotated code files are written to BASENAME(file).html, which results
+ in collisions between similarly-named source files.
+
+ The line-wise coverage information isn't quite right. Blank lines are
+ counted as unreached code, lambdas aren't quite right, and some multiline
+ comments (docstrings?) aren't quite right.
+
+"""
+
+from twisted.trial.reporter import TreeReporter, VerboseTextReporter
+
+# These plugins are registered via twisted/plugins/figleaf_trial.py . See
+# the notes there for an explanation of how that works.
+
+
+
+# Reporters don't really get told about the suite starting and stopping.
+
+# The Reporter class is imported before the test classes are.
+
+# The test classes are imported before the Reporter is created. To get
+# control earlier than that requires modifying twisted/scripts/trial.py .
+
+# Then Reporter.__init__ is called.
+
+# Then tests run, calling things like write() and addSuccess(). Each test is
+# framed by a startTest/stopTest call.
+
+# Then the results are emitted, calling things like printErrors,
+# printSummary, and wasSuccessful.
+
+# So for code-coverage (not including import), start in __init__ and finish
+# in printSummary. To include import, we have to start in our own import and
+# finish in printSummary.
+
+import figleaf
+figleaf.start()
+
+
+class FigleafReporter(TreeReporter):
+ def __init__(self, *args, **kwargs):
+ TreeReporter.__init__(self, *args, **kwargs)
+
+ def printSummary(self):
+ figleaf.stop()
+ figleaf.write_coverage(".figleaf")
+ print "Figleaf results written to .figleaf"
+ return TreeReporter.printSummary(self)
+
+class FigleafTextReporter(VerboseTextReporter):
+ def __init__(self, *args, **kwargs):
+ VerboseTextReporter.__init__(self, *args, **kwargs)
+
+ def printSummary(self):
+ figleaf.stop()
+ figleaf.write_coverage(".figleaf")
+ print "Figleaf results written to .figleaf"
+ return VerboseTextReporter.printSummary(self)
+
+class not_FigleafReporter(object):
+ # this class, used as a reporter on a fully-passing test suite, doesn't
+ # trigger exceptions. So it is a guide to what methods are invoked on a
+ # Reporter.
+ def __init__(self, *args, **kwargs):
+ print "FIGLEAF HERE"
+ self.r = TreeReporter(*args, **kwargs)
+ self.shouldStop = self.r.shouldStop
+ self.separator = self.r.separator
+ self.testsRun = self.r.testsRun
+ self._starting2 = False
+
+ def write(self, *args):
+ if not self._starting2:
+ self._starting2 = True
+ print "FIRST WRITE"
+ return self.r.write(*args)
+
+ def startTest(self, *args, **kwargs):
+ return self.r.startTest(*args, **kwargs)
+
+ def stopTest(self, *args, **kwargs):
+ return self.r.stopTest(*args, **kwargs)
+
+ def addSuccess(self, *args, **kwargs):
+ return self.r.addSuccess(*args, **kwargs)
+
+ def printErrors(self, *args, **kwargs):
+ return self.r.printErrors(*args, **kwargs)
+
+ def writeln(self, *args, **kwargs):
+ return self.r.writeln(*args, **kwargs)
+
+ def printSummary(self, *args, **kwargs):
+ print "PRINT SUMMARY"
+ return self.r.printSummary(*args, **kwargs)
+
+ def wasSuccessful(self, *args, **kwargs):
+ return self.r.wasSuccessful(*args, **kwargs)
+
--- /dev/null
+
+from zope.interface import implements
+from twisted.trial.itrial import IReporter
+from twisted.plugin import IPlugin
+
+# register a plugin that can create our FigleafReporter. The reporter itself
+# lives in a separate place
+
+# note that this .py file is *not* in a package: there is no __init__.py in
+# our parent directory. This is important, because otherwise ours would fight
+# with Twisted's. When trial looks for plugins, it merely executes all the
+# *.py files it finds in and twisted/plugins/ subdirectories of anything on
+# sys.path . The namespace that results from executing these .py files is
+# examined for instances which provide both IPlugin and the target interface
+# (in this case, trial is looking for IReporter instances). Each such
+# instance tells the application how to create a plugin by naming the module
+# and class that should be instantiated.
+
+# When installing our package via setup.py, arrange for this file to be
+# installed to the system-wide twisted/plugins/ directory.
+
+class _Reporter(object):
+ implements(IPlugin, IReporter)
+
+ def __init__(self, name, module, description, longOpt, shortOpt, klass):
+ self.name = name
+ self.module = module
+ self.description = description
+ self.longOpt = longOpt
+ self.shortOpt = shortOpt
+ self.klass = klass
+
+
+fig = _Reporter("Figleaf Code-Coverage Reporter",
+ "trial_figleaf",
+ description="verbose color output (with figleaf coverage)",
+ longOpt="verbose-figleaf",
+ shortOpt="f",
+ klass="FigleafReporter")
+
+bwfig = _Reporter("Figleaf Code-Coverage Reporter (colorless)",
+ "trial_figleaf",
+ description="Colorless verbose output (with figleaf coverage)",
+ longOpt="bwverbose-figleaf",
+ shortOpt=None,
+ klass="FigleafTextReporter")
+
-#!/usr/bin/env python
+#!/usr/bin/python
import sys
from distutils.core import setup
projects / tahoe-lafs / tahoe-lafs.git / commitdiff