[Twisted-Python] hosting cpu intensive algorithms with twisted. Is it worth it?

[James Y Knight]

On Aug 4, 2004, at 11:12 AM, Kelly Dyer wrote:
> Hi,
>
> I have a number of cpu-heavy signal processing routines that I want to
> make remotely available.
>
> Is it worth the energy to use the twisted framework in order to build
> an application like this?  I have a cluster available and was hoping
> to use its thread-migration to even out the processing and provide
> scalability.
>
> If you feel this is a do-able application, any design suggestions for
> an excited twisted neophyte would be welcome!

Are you going to be writing your routines in python? If so, you cannot 
use threads, as python only allows one thread at a time to run python 
bytecode, thus making any CPUs above one useless. Also, what clustering 
software are you using? The ones I've seen used in the past only allow 
migration of unthreaded programs, as shared memory consistency across a 
network is hard (and/or slow).

What you can do, though, (and something I would generally recommend, 
although I don't know the specifics of your problem), is to spawn off 
multiple *processes* to handle the long running routines, and use 
Twisted to coordinate the network and process IO.

If you are not writing your routines in python and your cluster 
software actually does support migration of threads, you can probably 
get away with running threads in twisted.

James