[Twisted-Python] Deferred Groups?
Mike C. Fletcher
mcfletch at rogers.com
Sat Jan 21 08:55:01 EST 2006
Duncan McGreggor wrote:
> I have a question about an approach I used... I'm worried that I've
> over-worked it and have over-looked a more elegant and standard solution.
>
> I have the need to fire off network connections in groups. Deferreds
> added to a DeferredList don't fit the bill (because there's no control
> over all the deferreds in the list). As an example, if you wanted to
> make a whole batch of concurrent connections, but didn't want to incur
> the overhead of firing off more than 20 simultaneous connections,
> you'd split your destination hosts up into groups of 20. As a group
> was completed, a callback could fire off the next group, etc.
This functionality (or, something very close) is in the deferred module
already, it's just non-obvious how you use it:
sem = defer.DeferredSemaphore(count)
dl = [sem.run(callable, item, *args, **named) for item in iterable]
return defer.DeferredList(dl, consumeErrors=True)
which, rather than "batching" (which means that you have 3, then 2, then
1 in play, then 3, then 2, then 1), keeps "count" items in play at any
time (as long as there are available items), replacing each item that
completes with one pending item. I wrap that bit of code with a little
function called parallel in my code so I don't have to remember the
trick every time.
HTH,
Mike
--
________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://www.vrplumber.com
http://blog.vrplumber.com
More information about the Twisted-Python
mailing list