[Twisted-Python] hosting cpu intensive algorithms withtwisted. Is it worth it?

[Ed Suominen]

Now that's an interesting idea...

twisted.cluster:

Cluster:
"""
Maintains multiple process nodes of a LAN cluster

Vaporware donated to the public domain by Ed Suominen

"""
	def __init__(self, listenPort)
		"""
		Listens on listenPort for requests by individual nodes to join the cluster
		"""

	def generator(self, callable, sequence, **kw):
		"""
		Yields result of callable(sequenceItem, **kw) over items of sequence,
		which are distributed among cluster nodes for processing. Each run of
		callable() cannot assume anything about the order of the item it is given
		in the sequence.
		"""


> From: James Y Knight <foom at fuhm.net>
> Subject: Re: [Twisted-Python] hosting cpu intensive algorithms with
> 	twisted.	Is it worth it?
> To: Twisted general discussion <twisted-python at twistedmatrix.com>
> Message-ID: <87AAEEC1-E706-11D8-8D12-000A95A50FB2 at fuhm.net>
> Content-Type: text/plain; charset=US-ASCII; format=flowed
>
> On Aug 4, 2004, at 11:12 AM, Kelly Dyer wrote:
> > Hi,
> >
> > I have a number of cpu-heavy signal processing routines that I want to
> > make remotely available.
> >
> > Is it worth the energy to use the twisted framework in order to build
> > an application like this?  I have a cluster available and was hoping
> > to use its thread-migration to even out the processing and provide
> > scalability.
> >
> > If you feel this is a do-able application, any design suggestions for
> > an excited twisted neophyte would be welcome!
>
> Are you going to be writing your routines in python? If so, you cannot
> use threads, as python only allows one thread at a time to run python
> bytecode, thus making any CPUs above one useless. Also, what clustering
> software are you using? The ones I've seen used in the past only allow
> migration of unthreaded programs, as shared memory consistency across a
> network is hard (and/or slow).
>
> What you can do, though, (and something I would generally recommend,
> although I don't know the specifics of your problem), is to spawn off
> multiple *processes* to handle the long running routines, and use
> Twisted to coordinate the network and process IO.
>
> If you are not writing your routines in python and your cluster
> software actually does support migration of threads, you can probably
> get away with running threads in twisted.
>
> James