[Twisted-Python] Ideas on limiting/throttling spawnProcess

[Valentino Volonghi]

On 28 Oct 2011, at 15:21, Daryl Herzmann wrote:

> Hi,
> 
> I have an application that ingests data and does a reactor.spawnProcess()
> for each chunk of data (a product) to pass it as STDIN to a compiled binary
> and then I harvest the STDOUT. It has been working well, but I have an issue
> when my data rates get too high and the machine gets overloaded with spawned
> processes (I think) and starts running out of file descriptors (yes I can
> raise it :) causing all sorts of pain.  I'm wondering about a mechanism to
> throttle the number of spawned processes going at one time?  It'd be nice to
> only have 10 of these spawned processes going at any one time.  Thanks for
> your ideas :)


Another alternative is to use turtl which was written for a similar purpose.

http://pypi.python.org/pypi/turtl
https://bitbucket.org/adroll/turtl/
https://launchpad.net/turtl
http://twistedmatrix.com/pipermail/twisted-python/2011-August/024415.html

-- 
Valentino Volonghi
http://www.adroll.com