[Twisted-Python] Throttling outgoing client requests

[Andrew Bennetts]

On Mon, Sep 12, 2005 at 09:30:51AM -0700, David E. Konerding wrote:
> Hi,
> 
> I am writing a client usign Twisted that makes a lot of XMLRPC requests 
> simultaneously (using twisted.web.xmlrpc.Proxy).  There are a bunch of 
> them, all to the same site,
> and when I run on MacOSX, I start getting bind errors-- I think OS X 
> must have more restricted outgoing network connections that the linux
> box I normally work on.
> 
> My goal is to throttle the number of outgoing requests.  I could do this 
> by submitting just one request at a time, and having a deferred callback
> registered such that when the request completes (successfully or not) 
> the next request is submitting.  But this sort of code flow is pretty
> contorted.  My other thought would be to just schedule all the calls 
> with a callLater and a randomized delay for each call.  Again, not so clean.
> 
> Is there a 'clean' way to throttle back the number of ongoing client 
> requests-- perhaps through the xmlrpc.Proxy itself, or through some pattern?

Use twisted.internet.defer.DeferredSemaphore (added in Twisted 2.0).

Rough sketch of how to use it:

    sem = DeferredSemaphore(10) # maximum of 10 jobs at once
    for job in joblist:
        d = sem.acquire()
        d.addCallback(job.run)  # do the work
        d.addErrback(log.err)   # handle any errors by logging them
        d.addBoth(lambda x: sem.release())  # trigger the next job

-Andrew.