[Twisted-Python] Using sqlalchemy in twisted.

[luper rouch]

2009/3/12 Peter Cai <newptcai at gmail.com>:
> Hi,
> I've read some mails in this list that recommend use sqlalchemy in separate
> processes.
> But if u use multiprocessing which provides only synchronous API, doesn't
> you have to use deferToThread also?
> Another way would be using "twisted.internet.protocol.ProcessProtocol",
> which use pipes.  But can we pass SA Objects through pipes?
> If you can write a little code to describe how you use multiprocessing,  I
> will be very thankful!

multiprocessing [1] works like the threading module, except that it
runs callables in subprocesses instead of threads.

To pass python objects to your process you can use the primitives
offered by the modules (Queue, Pipe, Connection...). Here is a minimal
example using Queue :

Child process implementation (let's call it my_twisted_module.py):
--------------------------
from twisted.internet import reactor, task

dispatch_task = None
in_queue = None
out_queue = None

def recv():
    obj = in_queue.get_nowait()
    [do whatever you want with obj]

def send(obj):
    out_queue.put_nowait(obj)

def twisted_process(inq, outq):
    global dispatch_task, in_queue, out_queue
    in_queue = inq
    out_queue = outq
    dispatch_task = task.LoopingCall(recv)
    dispatch_task.start(0.1)
    reactor.run()
--------------------------

On the parent process side:
--------------------------
from my_twisted_module import twisted_process

[something similar to the child process implementation for communications]

# Note that out_queue in this side is passed as in_queue to the other
side and vice versa
in_queue = Queue()
out_queue = Queue()
p = multiprocessing.Process(target=twisted_process, args=(out_queue, in_queue))
p.start()
--------------------------

You can pass any python object through the queues as long as it is
picklabe. Passing SA objects is probably not a good idea though as
this whole mess is about isolating Twisted from SA :) In my project I
organized the communications with a dispatcher (louie [2]), and I pass
signals through the queues. This way I have a well defined protocol to
communicate with the sub process(es).

Although I did not do any timing I think the communications between
the processes must introduce some significant overhead, so it is
probably not a good idea to do this if you pass objects all the time.
In my case, the communications are limited to initialization calls,
status keeping, etc...

Luper

[1]http://docs.python.org/library/multiprocessing.html
[2]http://pylouie.org/