[Twisted-Python] Re: Twisted and the Posh Module

[Ken Kinder]

Ed Suominen wrote:

>I think having some sort of process pool mangagement in Twisted is a 
>great idea, especially with multi-core CPUs emerging on the scene. I 
>have access to a dual-core Pentium Prescott CPU and it would be great 
>to have that available to keep both cores humming on a certain 
>CPU-intensive project I'm considering.
>
>However, I'm not sure the best way to go about it would be with the posh 
>module, pypar (see http://datamining.anu.edu.au/~ole/pypar/), or just 
>using Perspective Broker as an underlying message-passing mechanism 
>with UNIX sockets and/or TCP. One thought might be to have a single 
>master process start up and act as a PB server and process pool 
>manager. Subsidiary processes could then make authenticated PB 
>connections to the server to "volunteer" for work in the process pool. 
>
>Note that pypar lets you easily find out how many CPUs you have under 
>kernel control, with pypar.size(). Thus, the main process could start 
>the process pool by spawning a subsidiary "volunteer" process for each 
>CPU core present.
>  
>
My goal is somewhat similar in that I have multiple CPU's that aren't 
being used very much, but the other problem is that certain libraries I 
use (PIL) grab the GIL for long-running operations like image 
modifications. I can thread them off, but it still liberally acquires 
the GIL and makes the server unresponsive for the duration of the operation.

I too had looked at using Perspective Broker to communicate with 
separate "worker" processes, and the only reason I'm not excited with 
that option is that the bandwidth between the master and worker 
processes involves a lot of large binary strings. Shared memory seemed 
more efficient than PB for transferring those strings.

So for that reason, finding out how many CPU's I have isn't that 
important, because I'll still want more worker processes than I have CPU's.