[Twisted-Python] Control the number of parallel system calls

[Andy Gayton]

Daniel de la Cuesta wrote:
> Hi,
> 
> I am developing a video conversion server. To process the video 
> conversion I use the method "getProcessOutput" to call "ffmpeg".
> 
> The user upload the video using a HTTP POST and each video must be 
> converted to 8 or 10 output formats.
> 
> Currently, all the conversions for each video are processed in parallel, 
> what I want to do is to control the number of parallel system calls to 
> "ffmpeg". For example I only want 2 processes at the same time and when 
> one of them finishes the next process start.
> 
> How can I do that?
> 
> Is there any function in Twisted that implements that?

Hey Daniel,

There's a semaphore convenience in twisted.internet.defer - which can be 
used to control the number of things which have concurrent access to .. 
something.

You could make that something your function which launches a system 
call, with something like:

     s = defer.DeferredSemaphore(2)
     dlist = [
         s.run(encodeToFormat, video, format)
             for format in formats]
     d = defer.gatherResults(dlist)
     return d

cheers,
Andy.