[Twisted-Python] Scaling problem

[Glyph Lefkowitz]

Uwe C. Schroeder wrote:

> Is there a way to have twisted do something like apache or postgres and split 
> the incoming connection to several processes ? It's not a web application, 
> I'm solely using pb.

This is possible, although not as easy as I'd like it to be.  In 
principle, on most UNIX-based OSes, it's possible to listen() and 
accept() in one process, pass the returned file descriptor to a 
subprocess and do communication there.  I'd like to make this automatic 
at some point, but won't have the time to do it for at least another few 
months.

The fact that it's not as graceful or automatic as I like doesn't mean 
it's impossible though.  If you want to just have one "control" process 
that shuttles around I/O and N worker processes that do the actual work, 
you can still use spawnProcess and communicate over stdin/stdout to the 
subprocesses about new connections using a simple protocol which 
multiplexes multiple socket connections down to the one pipe connection 
and back.

Also, if database latency is your main problem, or you can move a little 
bit of performance-sensitive code into C and relinquish the GIL while 
they're happening, you can just use callInThread to run your 
transactions in threads instead of having the reactor waiting on them.

Is this helpful?