Using Perspective Broker

Basic Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from twisted.spread import pb
from twisted.internet import reactor

class Echoer(pb.Root):
    def remote_echo(self, st):
        print 'echoing:', st
        return st

if __name__ == '__main__':
    reactor.listenTCP(8789, pb.PBServerFactory(Echoer()))
    reactor.run()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from twisted.spread import pb
from twisted.internet import reactor
from twisted.python import util

factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 8789, factory)
d = factory.getRootObject()
d.addCallback(lambda object: object.callRemote("echo", "hello network"))
d.addCallback(lambda echo: 'server echoed: '+echo)
d.addErrback(lambda reason: 'error: '+str(reason.value))
d.addCallback(util.println)
d.addCallback(lambda _: reactor.stop())
reactor.run()

First we look at the server. This defines an Echoer class (derived from pb.Root), with a method called remote_echo(). pb.Root objects (because of their inheritance of pb.Referenceable, described later) can define methods with names of the form remote_*; a client which obtains a remote reference to that pb.Root object will be able to invoke those methods.

The pb.Root-ish object is given to a pb.PBServerFactory(). This is a Factory object like any other: the Protocol objects it creates for new connections know how to speak the PB protocol. The object you give to pb.PBServerFactory() becomes the root object, which simply makes it available for the client to retrieve. The client may only request references to the objects you want to provide it: this helps you implement your security model. Because it is so common to export just a single object (and because a remote_* method on that one can return a reference to any other object you might want to give out), the simplest example is one where the PBServerFactory is given the root object, and the client retrieves it.

The client side uses pb.PBClientFactory to make a connection to a given port. This is a two-step process involving opening a TCP connection to a given host and port and requesting the root object using .getRootObject().

Because .getRootObject() has to wait until a network connection has been made and exchange some data, it may take a while, so it returns a Deferred, to which the gotObject() callback is attached. (See the documentation on Deferring Execution for a complete explanation of Deferreds). If and when the connection succeeds and a reference to the remote root object is obtained, this callback is run. The first argument passed to the callback is a remote reference to the distant root object. (you can give other arguments to the callback too, see the other parameters for .addCallback() and .addCallbacks()).

The callback does:

1
object.callRemote("echo", "hello network")

which causes the server's .remote_echo() method to be invoked. (running .callRemote("boom") would cause .remote_boom() to be run, etc). Again because of the delay involved, callRemote() returns a Deferred. Assuming the remote method was run without causing an exception (including an attempt to invoke an unknown method), the callback attached to that Deferred will be invoked with any objects that were returned by the remote method call.

In this example, the server's Echoer object has a method invoked, exactly as if some code on the server side had done:

1
echoer_object.remote_echo("hello network")

and from the definition of remote_echo() we see that this just returns the same string it was given: hello network.

From the client's point of view, the remote call gets another Deferred object instead of that string. callRemote() always returns a Deferred. This is why PB is described as a system for translucent remote method calls instead of transparent ones: you cannot pretend that the remote object is really local. Trying to do so (as some other RPC mechanisms do, coughCORBAcough) breaks down when faced with the asynchronous nature of the network. Using Deferreds turns out to be a very clean way to deal with the whole thing.

The remote reference object (the one given to getRootObject()'s success callback) is an instance the RemoteReference class. This means you can use it to invoke methods on the remote object that it refers to. Only instances of RemoteReference are eligible for .callRemote(). The RemoteReference object is the one that lives on the remote side (the client, in this case), not the local side (where the actual object is defined).

In our example, the local object is that Echoer() instance, which inherits from pb.Root, which inherits from pb.Referenceable. It is that Referenceable class that makes the object eligible to be available for remote method calls1. If you have an object that is Referenceable, then any client that manages to get a reference to it can invoke any remote_* methods they please.

You don't have to be a pb.Root to be remotely callable, but you do have to be pb.Referenceable. (Objects that inherit from pb.Referenceable but not from pb.Root can be remotely called, but only pb.Root-ish objects can be given to the PBServerFactory.)

Complete Example

Here is an example client and server which uses pb.Referenceable as a root object and as the result of a remotely exposed method. In each context, methods can be invoked on the exposed Referenceable instance. In this example, the initial root object has a method that returns a reference to the second object.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb

class Two(pb.Referenceable):
    def remote_three(self, arg):
        print "Two.three was given", arg

class One(pb.Root):
    def remote_getTwo(self):
        two = Two()
        print "returning a Two called", two
        return two

from twisted.internet import reactor
reactor.listenTCP(8800, pb.PBServerFactory(One()))
reactor.run()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb
from twisted.internet import reactor

def main():
    factory = pb.PBClientFactory()
    reactor.connectTCP("localhost", 8800, factory)
    def1 = factory.getRootObject()
    def1.addCallbacks(got_obj1, err_obj1)
    reactor.run()

def err_obj1(reason):
    print "error getting first object", reason
    reactor.stop()

def got_obj1(obj1):
    print "got first object:", obj1
    print "asking it to getTwo"
    def2 = obj1.callRemote("getTwo")
    def2.addCallbacks(got_obj2)

def got_obj2(obj2):
    print "got second object:", obj2
    print "telling it to do three(12)"
    obj2.callRemote("three", 12)

main()

pb.PBClientFactory.getRootObject will handle all the details of waiting for the creation of a connection. It returns a Deferred, which will have its callback called when the reactor connects to the remote server and pb.PBClientFactory gets the root, and have its errback called when the object-connection fails for any reason, whether it was host lookup failure, connection refusal, or some server-side error.

The root object has a method called remote_getTwo, which returns the Two() instance. On the client end, the callback gets a RemoteReference to that instance. The client can then invoke two's .remote_three() method.

RemoteReference objects have one method which is their purpose for being: callRemote. This method allows you to call a remote method on the object being referred to by the Reference. RemoteReference.callRemote, like pb.PBClientFactory.getRootObject, returns a Deferred. When a response to the method-call being sent arrives, the Deferred's callback or errback will be made, depending on whether an error occurred in processing the method call.

You can use this technique to provide access to arbitrary sets of objects. Just remember that any object that might get passed over the wire must inherit from Referenceable (or one of the other flavors). If you try to pass a non-Referenceable object (say, by returning one from a remote_* method), you'll get an InsecureJelly exception2.

References can come back to you

If your server gives a reference to a client, and then that client gives the reference back to the server, the server will wind up with the same object it gave out originally. The serialization layer watches for returning reference identifiers and turns them into actual objects. You need to stay aware of where the object lives: if it is on your side, you do actual method calls. If it is on the other side, you do .callRemote()3.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb
from twisted.internet import reactor

class Two(pb.Referenceable):
    def remote_print(self, arg):
        print "two.print was given", arg

class One(pb.Root):
    def __init__(self, two):
        #pb.Root.__init__(self)   # pb.Root doesn't implement __init__
        self.two = two
    def remote_getTwo(self):
        print "One.getTwo(), returning my two called", self.two
        return self.two
    def remote_checkTwo(self, newtwo):
        print "One.checkTwo(): comparing my two", self.two
        print "One.checkTwo(): against your two", newtwo
        if self.two == newtwo:
            print "One.checkTwo(): our twos are the same"


two = Two()
root_obj = One(two)
reactor.listenTCP(8800, pb.PBServerFactory(root_obj))
reactor.run()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb
from twisted.internet import reactor

def main():
    foo = Foo()
    factory = pb.PBClientFactory()
    reactor.connectTCP("localhost", 8800, factory)
    factory.getRootObject().addCallback(foo.step1)
    reactor.run()

# keeping globals around is starting to get ugly, so we use a simple class
# instead. Instead of hooking one function to the next, we hook one method
# to the next.

class Foo:
    def __init__(self):
        self.oneRef = None

    def step1(self, obj):
        print "got one object:", obj
        self.oneRef = obj
        print "asking it to getTwo"
        self.oneRef.callRemote("getTwo").addCallback(self.step2)

    def step2(self, two):
        print "got two object:", two
        print "giving it back to one"
        print "one is", self.oneRef
        self.oneRef.callRemote("checkTwo", two)

main()

The server gives a Two() instance to the client, who then returns the reference back to the server. The server compares the two given with the two received and shows that they are the same, and that both are real objects instead of remote references.

A few other techniques are demonstrated in pb2client.py. One is that the callbacks are are added with .addCallback instead of .addCallbacks. As you can tell from the Deferred documentation, .addCallback is a simplified form which only adds a success callback. The other is that to keep track of state from one callback to the next (the remote reference to the main One() object), we create a simple class, store the reference in an instance thereof, and point the callbacks at a sequence of bound methods. This is a convenient way to encapsulate a state machine. Each response kicks off the next method, and any data that needs to be carried from one state to the next can simply be saved as an attribute of the object.

Remember that the client can give you back any remote reference you've given them. Don't base your zillion-dollar stock-trading clearinghouse server on the idea that you trust the client to give you back the right reference. The security model inherent in PB means that they can only give you back a reference that you've given them for the current connection (not one you've given to someone else instead, nor one you gave them last time before the TCP session went down, nor one you haven't yet given to the client), but just like with URLs and HTTP cookies, the particular reference they give you is entirely under their control.

References to client-side objects

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb
from twisted.internet import reactor

class One(pb.Root):
    def remote_takeTwo(self, two):
        print "received a Two called", two
        print "telling it to print(12)"
        two.callRemote("print", 12)

reactor.listenTCP(8800, pb.PBServerFactory(One()))
reactor.run()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb
from twisted.internet import reactor

class Two(pb.Referenceable):
    def remote_print(self, arg):
        print "Two.print() called with", arg

def main():
    two = Two()
    factory = pb.PBClientFactory()
    reactor.connectTCP("localhost", 8800, factory)
    def1 = factory.getRootObject()
    def1.addCallback(got_obj, two) # hands our 'two' to the callback
    reactor.run()

def got_obj(obj, two):
    print "got One:", obj
    print "giving it our two"
    obj.callRemote("takeTwo", two)

main()

In this example, the client gives a reference to its own object to the server. The server then invokes a remote method on the client-side object.

Raising Remote Exceptions

The only special thing you do is to define your Exception subclass by deriving it from pb.Error. When any remotely-invokable method (like remote_* or perspective_*) raises a pb.Error-derived exception, a serialized form of that Exception object will be sent back over the wire4. The other side (which did callRemote) will have the errback callback run with a Failure object that contains a copy of the exception object. This Failure object can be queried to retrieve the error message and a stack traceback.

Failure is a special class, defined in twisted/python/failure.py, created to make it easier to handle asynchronous exceptions. Just as exception handlers can be nested, errback functions can be chained. If one errback can't handle the particular type of failure, it can be passed along to a errback handler further down the chain.

For simple purposes, think of the Failure as just a container for remotely-thrown Exception objects. To extract the string that was put into the exception, use its .getErrorMessage() method. To get the type of the exception (as a string), look at its .type attribute. The stack traceback is available too. The intent is to let the errback function get just as much information about the exception as Python's normal try: clauses do, even though the exception occurred in somebody else's memory space at some unknown time in the past.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb
from twisted.internet import reactor

class MyError(pb.Error):
    """This is an Expected Exception. Something bad happened."""
    pass

class MyError2(Exception):
    """This is an Unexpected Exception. Something really bad happened."""
    pass

class One(pb.Root):
    def remote_broken(self):
        msg = "fall down go boom"
        print "raising a MyError exception with data '%s'" % msg
        raise MyError(msg)
    def remote_broken2(self):
        msg = "hadda owie"
        print "raising a MyError2 exception with data '%s'" % msg
        raise MyError2(msg)

def main():
    reactor.listenTCP(8800, pb.PBServerFactory(One()))
    reactor.run()

if __name__ == '__main__':
    main()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb
from twisted.internet import reactor

def main():
    factory = pb.PBClientFactory()
    reactor.connectTCP("localhost", 8800, factory)
    d = factory.getRootObject()
    d.addCallbacks(got_obj)
    reactor.run()

def got_obj(obj):
    # change "broken" into "broken2" to demonstrate an unhandled exception
    d2 = obj.callRemote("broken")
    d2.addCallback(working)
    d2.addErrback(broken)

def working():
    print "erm, it wasn't *supposed* to work.."

def broken(reason):
    print "got remote Exception"
    # reason should be a Failure (or subclass) holding the MyError exception
    print " .__class__ =", reason.__class__
    print " .getErrorMessage() =", reason.getErrorMessage()
    print " .type =", reason.type
    reactor.stop()

main()

$ ./exc_client.py 
got remote Exception
 .__class__ = twisted.spread.pb.CopiedFailure
 .getErrorMessage() = fall down go boom
 .type = __main__.MyError
Main loop terminated.

Oh, and what happens if you raise some other kind of exception? Something that isn't subclassed from pb.Error? Well, those are called unexpected exceptions, which make Twisted think that something has really gone wrong. These will raise an exception on the server side. This won't break the connection (the exception is trapped, just like most exceptions that occur in response to network traffic), but it will print out an unsightly stack trace on the server's stderr with a message that says Peer Will Receive PB Traceback, just as if the exception had happened outside a remotely-invokable method. (This message will go the current log target, if log.startLogging was used to redirect it). The client will get the same Failure object in either case, but subclassing your exception from pb.Error is the way to tell Twisted that you expect this sort of exception, and that it is ok to just let the client handle it instead of also asking the server to complain. Look at exc_client.py and change it to invoke broken2() instead of broken() to see the change in the server's behavior.

If you don't add an errback function to the Deferred, then a remote exception will still send a Failure object back over, but it will get lodged in the Deferred with nowhere to go. When that Deferred finally goes out of scope, the side that did callRemote will emit a message about an Unhandled error in Deferred, along with an ugly stack trace. It can't raise an exception at that point (after all, the callRemote that triggered the problem is long gone), but it will emit a traceback. So be a good programmer and always add errback handlers, even if they are just calls to log.err.

Try/Except blocks and Failure.trap

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.internet import reactor
from twisted.spread import pb

class MyException(pb.Error):
    pass

class One(pb.Root):
    def remote_fooMethod(self, arg):
        if arg == "panic!":
            raise MyException
        return "response"
    def remote_shutdown(self):
        reactor.stop()

reactor.listenTCP(8800, pb.PBServerFactory(One()))
reactor.run()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
#!/usr/bin/env python

# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.spread import pb, jelly
from twisted.python import log
from twisted.internet import reactor

class MyException(pb.Error): pass
class MyOtherException(pb.Error): pass

class ScaryObject:
    # not safe for serialization
    pass

def worksLike(obj):
    # the callback/errback sequence in class One works just like an
    # asynchronous version of the following:
    try:
        response = obj.callMethod(name, arg)
    except pb.DeadReferenceError:
        print " stale reference: the client disconnected or crashed"
    except jelly.InsecureJelly:
        print " InsecureJelly: you tried to send something unsafe to them"
    except (MyException, MyOtherException):
        print " remote raised a MyException" # or MyOtherException
    except:
        print " something else happened"
    else:
        print " method successful, response:", response

class One:
    def worked(self, response):
        print " method successful, response:", response
    def check_InsecureJelly(self, failure):
        failure.trap(jelly.InsecureJelly)
        print " InsecureJelly: you tried to send something unsafe to them"
        return None
    def check_MyException(self, failure):
        which = failure.trap(MyException, MyOtherException)
        if which == MyException:
            print " remote raised a MyException"
        else:
            print " remote raised a MyOtherException"
        return None
    def catch_everythingElse(self, failure):
        print " something else happened"
        log.err(failure)
        return None

    def doCall(self, explanation, arg):
        print explanation
        try:
            deferred = self.remote.callRemote("fooMethod", arg)
            deferred.addCallback(self.worked)
            deferred.addErrback(self.check_InsecureJelly)
            deferred.addErrback(self.check_MyException)
            deferred.addErrback(self.catch_everythingElse)
        except pb.DeadReferenceError:
            print " stale reference: the client disconnected or crashed"

    def callOne(self):
        self.doCall("callOne: call with safe object", "safe string")
    def callTwo(self):
        self.doCall("callTwo: call with dangerous object", ScaryObject())
    def callThree(self):
        self.doCall("callThree: call that raises remote exception", "panic!")
    def callShutdown(self):
        print "telling them to shut down"
        self.remote.callRemote("shutdown")
    def callFour(self):
        self.doCall("callFour: call on stale reference", "dummy")

    def got_obj(self, obj):
        self.remote = obj
        reactor.callLater(1, self.callOne)
        reactor.callLater(2, self.callTwo)
        reactor.callLater(3, self.callThree)
        reactor.callLater(4, self.callShutdown)
        reactor.callLater(5, self.callFour)
        reactor.callLater(6, reactor.stop)

factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 8800, factory)
deferred = factory.getRootObject()
deferred.addCallback(One().got_obj)
reactor.run()

$ ./trap_client.py 
callOne: call with safe object
 method successful, response: response
callTwo: call with dangerous object
 InsecureJelly: you tried to send something unsafe to them
callThree: call that raises remote exception
 remote raised a MyException
telling them to shut down
callFour: call on stale reference
 stale reference: the client disconnected or crashed

In this example, callTwo tries to send an instance of a locally-defined class through callRemote. The default security model implemented by jelly on the remote end will not allow unknown classes to be unserialized (i.e. taken off the wire as a stream of bytes and turned back into an object: a living, breathing instance of some class): one reason is that it does not know which local class ought to be used to create an instance that corresponds to the remote object5.

The receiving end of the connection gets to decide what to accept and what to reject. It indicates its disapproval by raising a jelly.InsecureJelly exception. Because it occurs at the remote end, the exception is returned to the caller asynchronously, so an errback handler for the associated Deferred is run. That errback receives a Failure which wraps the InsecureJelly.

Remember that trap re-raises exceptions that it wasn't asked to look for. You can only check for one set of exceptions per errback handler: all others must be checked in a subsequent handler. check_MyException shows how multiple kinds of exceptions can be checked in a single errback: give a list of exception types to trap, and it will return the matching member. In this case, the kinds of exceptions we are checking for (MyException and MyOtherException) may be raised by the remote end: they inherit from pb.Error.

The handler can return None to terminate processing of the errback chain (to be precise, it switches to the callback that follows the errback; if there is no callback then processing terminates). It is a good idea to put an errback that will catch everything (no trap tests, no possible chance of raising more exceptions, always returns None) at the end of the chain. Just as with regular try: except: handlers, you need to think carefully about ways in which your errback handlers could themselves raise exceptions. The extra importance in an asynchronous environment is that an exception that falls off the end of the Deferred will not be signalled until that Deferred goes out of scope, and at that point may only cause a log message (which could even be thrown away if log.startLogging is not used to point it at stdout or a log file). In contrast, a synchronous exception that is not handled by any other except: block will very visibly terminate the program immediately with a noisy stack trace.

callFour shows another kind of exception that can occur while using callRemote: pb.DeadReferenceError. This one occurs when the remote end has disconnected or crashed, leaving the local side with a stale reference. This kind of exception happens to be reported right away (XXX: is this guaranteed? probably not), so must be caught in a traditional synchronous try: except pb.DeadReferenceError block.

Yet another kind that can occur is a pb.PBConnectionLost exception. This occurs (asynchronously) if the connection was lost while you were waiting for a callRemote call to complete. When the line goes dead, all pending requests are terminated with this exception. Note that you have no way of knowing whether the request made it to the other end or not, nor how far along in processing it they had managed before the connection was lost. XXX: explain transaction semantics, find a decent reference.

Footnotes

1. There are a few other classes that can bestow this ability, but pb.Referenceable is the easiest to understand; see 'flavors' below for details on the others.
2. This can be overridden, by subclassing one of the Serializable flavors and defining custom serialization code for your class. See Passing Complex Types for details.
3. The binary nature of this local vs. remote scheme works because you cannot give RemoteReferences to a third party. If you could, then your object A could go to B, B could give it to C, C might give it back to you, and you would be hard pressed to tell if the object lived in C's memory space, in B's, or if it was really your own object, tarnished and sullied after being handed down like a really ugly picture that your great aunt owned and which nobody wants but which nobody can bear to throw out. Ok, not really like that, but you get the idea.
4. To be precise, the Failure will be sent if any exception is raised, not just pb.Error-derived ones. But the server will print ugly error messages if you raise ones that aren't derived from pb.Error.

5. The naive approach of simply doing import SomeClass to match a remote caller who claims to have an object of type SomeClass could have nasty consequences for some modules that do significant operations in their __init__ methods (think telnetlib.Telnet(host='localhost', port='chargen'), or even more powerful classes that you have available in your server program). Allowing a remote entity to create arbitrary classes in your namespace is nearly equivalent to allowing them to run arbitrary code.

   The InsecureJelly exception arises because the class being sent over the wire has not been registered with the serialization layer (known as jelly). The easiest way to make it possible to copy entire class instances over the wire is to have them inherit from pb.Copyable, and then to use setUnjellyableForClass(remoteClass, localClass) on the receiving side. See Passing Complex Types for an example.