[Twisted-Python] Re: Twisted and the Posh Module

[Ed Suominen]

On Monday 14 March 2005 10:13 am, 
twisted-python-request at twistedmatrix.com wrote:
> Has anyone tried playing with Twisted and the posh module? I need to
> do some CPU-intensive stuff inside the reactor, which unfortunately
> holds on to the GIL. I was thinking of just making a deferToPosh()
> type of method in the spirit of deferToThread(), but forking on each
> operation would be pretty expensive. Which brings up the question of
> having process pools.
>
> Has anyone done anything like this? I don't really need any
> interaction between the worker processes and Twisted, so in theory I
> could spawn off some worker processes early and just communicate
> queues of tasks to complete, independent of Twisted and the reactor.

I think having some sort of process pool mangagement in Twisted is a 
great idea, especially with multi-core CPUs emerging on the scene. I 
have access to a dual-core Pentium Prescott CPU and it would be great 
to have that available to keep both cores humming on a certain 
CPU-intensive project I'm considering.

However, I'm not sure the best way to go about it would be with the posh 
module, pypar (see http://datamining.anu.edu.au/~ole/pypar/), or just 
using Perspective Broker as an underlying message-passing mechanism 
with UNIX sockets and/or TCP. One thought might be to have a single 
master process start up and act as a PB server and process pool 
manager. Subsidiary processes could then make authenticated PB 
connections to the server to "volunteer" for work in the process pool. 

Note that pypar lets you easily find out how many CPUs you have under 
kernel control, with pypar.size(). Thus, the main process could start 
the process pool by spawning a subsidiary "volunteer" process for each 
CPU core present.

---
Ed Suominen
Registered Patent Agent
Open-Source Software Author (yes, both...)
Web Site: http://www.eepatents.com