[Twisted-Python] Newbie to Twisted. Converting Asyncore App to Twisted App

[Sean Roark]

I am new to this Twisted Architecture so if my plan makes no sense, please
guide me down a smoother path.

I have a program using Asyncore as the communications interface but I want
to switch to Twisted.

Server1                                Server 2
<-------TCP------|             | <---TCP-------
                     | My App |
----TCP--------->|             |----TCP----->

Basically MyApp takes stuff from Server 1, modifies it in some way, then
sends it to server 2 and vice versa for Server 2 to Server 1. It is a
translator between two ICD's languages spoken by two different systems. All
data in and out is raw.

I have modules written that can parse the raw data into objects, translate
the objects into the other ICD, then repack the objects into raw data.
Currently, My App uses asyncore to manage the socket between the servers as
pictured above. Asyncore sticks the raw data on a Queue for the next object
to retrieve. Each object is a thread that blocks on the Queue until data
arrives. When a object receives data on the Queue, it performs its operation
and sticks the modified data on another queue for the next thread to handle.
Eventually, data ends up on a Queue being monitored by an Asyncore
dispatcher that grabs the data and sends it to a server.

Server1 ----TCP----->Asyncore.recv() --put--> Queue --get-->Parser--put-->
Queue --get--> Translator --put--> Queue --get--> Packer --put---> Queue
--get-->Asyncore.send()

I would like to use Twisted for the following reasons:

1. Add a gui, stand alone or web based, allowing users to view statistics
being generated by modules and to inject messages to one of the servers.
Other services may be necessary in the future
2. We have a problem with asyncore behaving differently on Windows servers
then Linux servers with regards to reconnecting after network drops. It is
greatly beneficial if we can run on either platform.
3. Twistd is a great way to run this program as a daemon on Linux but still
works on Windows
4. My own personal education
5. Twisted seems so cool

Questions that have arisen as i try to code this in Twisted.

1. What is the proper method for gluing my processing modules to Twisted?
I have followed the finger tutorial and it seems I should call them from a
Service class. Finger tutorial part
3<http://twistedmatrix.com/projects/core/documentation/howto/tutorial/style.html>
This leads to question 2

I have toyed with using DeferredQueue's to pass data between the modules. I
would pretty much follow the pattern above, but the DeferredQueue's should
allow my modules to NOT run as threads. Instead they just fire events when
data arrives on their Queue.

2. How do I have My App send data to the servers via events?
In the finger tutorial Part 3 there are getUser and getUsers methods that
are called by the FingerProtocol class via the FingerClientFactory. This all
makes sense for processing data from the network and responding to it, but
suppose I want some other object, say my Packer object, to fire an event
that will make FingerProtocol send some data from Packer to the server? How
do I expose a method to do this if Packer only knows about the Service, but
the Service has no access to methods in FingerProtocol?

I have come up with two solutions to this problem, neither of which is
impressive to me.

First solution, use DeferredQueue's to pass all the data around. This does
work as a prototype, but I have no Service class wrapping it.

class QueueProtocol(Protocol):
    "Handles the network receive and send work for a send and a receive
queue"
    def dataReceived(self, data):
        print "Received Data:",data
        self.factory.putData(data)

    def connectionMade(self):
        print "Conn Made"
        self.sendData()

    def sendData(self):
        print "sendData"
        self.factory.getData().addCallback(self.sendHelp)

    def sendHelp(self,data):
        print "SendHelp",data
        self.transport.write(data+"\r\n")
        self.sendData()

class QueueClientFactory(ClientFactory):
    "Handle the Connection"
    protocol = QueueProtocol
    def __init__(self,recQueue, sendQueue):
        self.recQueue = recQueue #both deferredQueue's
        self.sendQueue = sendQueue

    def putData(self,data):
        self.recQueue.put(data)

    def getData(self):
        "Returns a defered because sendQueue is a DeferredQueue"
        return self.sendQueue.get()

Second solution, that I have not managed to code. I have the feeling I'm way
of the mark here.

Each component (yes I have been reading the Zope docs) in the application
knows about interfaces. All components will implement a receive(data) method
and a process(data) method. Each of these objects can pass data to other
components by adding another components receive(data) method as a callback
to its own process(data) method.

Parse
def receive(data):
self.process(data).addCallback( Translate.receive(data) )

The Parse object receives data from somewhere. The process method takes this
raw data and parses it into an object. When process has completed the
callback method, defined in the Translate object, is called with the new
object as a parameter.

Parse's Factory class will take an instance of Translate in __init__.

Some guidance would make my brain hurt less.

Thanks,

-- 

Sean Roark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20061110/3c545c46/attachment.html>