[Twisted-Python] Returning a deferred from buildProtocol t.i.p.Factory

[Lucas Taylor]

On Nov 16, 2013, at 7:09 AM, Tom van Neerijnen wrote:

> Hi all
> 
> I'm building a simple TCP load balancer based on a code snippet from Glyph on SO: http://stackoverflow.com/questions/4096061/general-question-regarding-wether-or-not-use-twisted-in-tcp-proxy-project
> 
> It's served me well but I can't work out how to convert Glyphs round robin retrieval of the server endpoint into an async balancing decision in the buildProtocol method of the Factory. If I return a deferred here it fails with an AttributeError: Deferred instance has no attribute 'makeConnection'.
> 
> Currently I'm working around this by running a separate management loop that periodically updates a dictionary with all the data necessary to make my routing decision so that I can do it without a deferred. This worries me because I may be making my decision on slightly stale data and I'd really like this to be a real time decision as the connection comes in. Does anyone have a clever way of doing this?
> 


Hi Tom,

One possibly unexpected aspect of using @inlineCallbacks is that the decorated function itself returns a Deferred. This is why you see the AttributeError...the machinery calling buildProtocol expects an IProtocol instance (or None), and the function is returning a Deferred.   `defer.returnValue()` is provided to the callback on that Deferred, not as a direct return value from the decorated function.

If you want to make the routing decision when the client connects, then you could push the decision-making process down into the Protocol itself.

Here's a quick mockup overriding connectionMade in a ProxyServer protocol subclass. It calls the factory routing function (which may or may not return a deferred), and connects the proxy once the decision has been made.


from twisted.internet.protocol import Factory
from twisted.protocols.portforward import ProxyServer


class Balancer(Factory):
    protocol = RoutingProxyServer
    routing_func = port_routing_decision_async


class RoutingProxyServer(ProxyServer):

    def connectionMade(self):
        # Don't read anything from the connecting client until we have
        # somewhere to send it to.
        self.transport.pauseProducing()

        client = self.clientProtocolFactory()
        client.setServer(self)
        
        if self.reactor is None:
            from twisted.internet import reactor
            self.reactor = reactor
        
        def connectProxy(host, port):
            self.reactor.connectTCP(host, port, client)

        d = maybeDeferred(self.factory.routing_func)
        d.addCallback(connectProxy)
        d.addErrback(log.err)


Lucas





> An example is below. The hashed out buildProtocol is a synchronous decision which works. Thanks in advance!
> 
> from twisted.internet.protocol import Factory
> from twisted.protocols.portforward import ProxyFactory
> from twisted.internet import reactor, defer
> import random
> 
> from twisted.python import log
> import sys
> log.startLogging(sys.stderr)
> 
> local_ports = set([1024, 1025])
> 
> def port_routing_decision_sync():
>     return random.choice(list(local_ports))
> 
> def port_routing_decision_async():
>     d = defer.Deferred()
>     reactor.callLater(1, d.callback, port_routing_decision_sync())
>     return d
> 
> class Balancer(Factory):
>     # def buildProtocol(self, addr):
>     #     port = port_routing_decision_sync()
>     #     print "connecting to local port {}".format(port)
>     #     return ProxyFactory("127.0.0.1", port).buildProtocol(addr)
> 
>     @defer.inlineCallbacks
>     def buildProtocol(self, addr):
>         port = yield port_routing_decision_async()
>         print "connecting to local port {}".format(port)
>         defer.returnValue(ProxyFactory("127.0.0.1", port).buildProtocol(addr))
> 
> def main():
>     factory = Balancer()
>     reactor.listenTCP(5678, factory)
>     reactor.run()
> 
> if __name__ == "__main__":
>     main()

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20131116/b6f70e6c/attachment-0002.html>