[Twisted-Python] Need some pointers for writing asynchronous code for Twisted app

[Brian Costlow]

Thanks everyone for your comments, you were all a lot of help.

Andrew, I was aware that the deferred didn't go anywhere, I oversimplified
things in the example, in the original design in my head, the code that
actually does the db calls was waiting on the deferred. Actually, the way
it's written, it never even runs, I left out the first call to
processChunk() necessary to start the iteration. But your comments helped a
lot.

My concern wasn't ElementTrees built in stuff, but 300+ line spaghetti
procedure that massages the element tree data into a dictionary. That needs
to be rewritten anyway.

I messed around this morning, and here's what I have done. I compiled
cElementTree onto the test box. The LineReceiver uses
cElementTree.XMLTreeBuilder.feed() and builds the tree on the fly.

At connectionClose it calls a method that runs the business logic routine in
a thread, using threads.deferToThread, and adding a callback; said callback
executes the db code. That still blocks for now.

So, my current plan is to refactor the tree-->dict code, but call that in a
separate thread. Rewite the dict --> db layer. The only question I have, is
since it's a long series of insert/update calls in which I don't need
results other than, 'it worked' , is, after reading the comments, it seems
like I might just best be served by writing a series of straight dbi calls;
and wrapping that all in one call to thread.callInThread.

The db is MySQL, and feeds a PHP-based reporting app, so I can't use
pgasync.

Thanks again all.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20070310/cd040366/attachment.html>