[Twisted-Python] XML-RPC, LineReceiver and defer

[Jp Calderone]

On Wed, 24 Nov 2004 09:27:55 +0100, Roland Hedberg <roland.hedberg at adm.umu.se> wrote:
>Hi!
> 
> I'm having problems with defer again :-(
> There is something about defer's that I just don't get, obviously.
> 
> Anyway, I'd like to get tips on how to solve this problem:
> 
> I have a module who on one side listens on XML-RPC and on the other 
> uses the LineReceiver protocol to find some information it needs, to 
> service the XML-RFC function.
> 
> If I have understood it correctly xmlrpc.XMLRFC can return a defer and 
> have the XMLRPC layer hold the reply until some event has happend.

  Yep.

> 
> Now, the LineReceiver protocol has one function for sending a lines and
> another function for receiving them.
> 
> So how do I make XMLRPC wait for the right line.
> 
> The XML-RPC server should of course be able to handle serveral requests 
> at the same time, so there might be more than one outstanding reply on 
> the lineReceiver side.
> 
> Knowing which reply that is connected to which sent line is doable by 
> keeping a queue and using IDs ( the protocol on the LineReceiver side, 
> which is no standard protocol, allows for having tags on queries and 
> replies).
> 
> But how do I keep the connection between the XML-RPC request and the 
> line received ?
> 
> I though about having defer checking for the change of a value in a 
> dictonary, but I don't know if that is doable, is it ?

  Hopefully you already have something like this:

    from twisted.protocols import basic

    class SimpleRequestProtocol(basic.LineReciever):
        def connectionMade(self):
            self.outstanding = {}
            self.counter = 0

        def request(self, query):
            d = self.outstanding[self.counter] = defer.Deferred()
            self.sendLine('%d %s' % (self.counter, query))
            self.counter += 1
            return d

        def lineReceived(self, response):
            tag, rest = response.split(None, 1)
            self.outstanding.pop(int(tag)).callback(rest)

  Letting the XML-RPC side just call `request' and get back a Deferred that fires whenever the response arrives.

> 
> And finally I don't want to use callLater if I can avoid it. I'd like 
> the performance to be as high as possible. That is, the XMLRPC reply 
> should be sent as soon as the LineReceiver has gotten the reply.
> I also don't want to setup and tear down a connection on the 
> LineReceiver side for every request that comes in on the XMLRPC side. 
> There is one server on the LineReceiver side that gets all the 
> questions, so I liked to be able to have one connection going.
> 

  So, first, you want to be able to get a connected instance:

    from twisted.internet import reactor, protocol

    def getConnectedProto(host, port):
        f = protocol.ClientCreator(reactor, SimpleRequestProtocol)
        return f.connectTCP(host, port)

  This returns a Deferred that fires with the instance once the connection is made.  You can call it from your XML-RPC code, add a callback to the result, and in the callback invoke .request() with whatever args is appropriate.

  But you also don't want a connection per request.  So you want to cache the connection:

    from twisted.internet import defer

    class ConnCache:
        def __init__(self):
            self._cache = {}

        def getConnectedProto(self, host, port)
            try:
                return defer.succeed(self._cache[host, port])
            except KeyError:
                d = self._makeConn(host, port)
                d.addCallback(self._cacheConn, host, port)
                return d

        def _makeConn(self, host, port):
            f = protocol.ClientCreator(reactor, SimpleRequestProtocol)
            return f.connectTCP(host, port)

        def _cacheConn(self, protoInstance, host, port):
            self._cache[host, port] = protoInstance
            return protoInstance

  So from your XML-RPC server, you can instantiate ConnCache and store the instance.  When you need to do something with your SimpleRequestProtocol, you can just call .getConnectedProto() on it.  If there's a connection, it'll be quick; if there's not, it'll go make one and save it for next time, too.

  Hope this helps,

  Jp