[Twisted-Python] Reentrant reactor iteration

[Jean-Paul Calderone]

On Fri, 27 Feb 2009 15:26:43 +0100, Marcel Keller <mkeller at cs.au.dk> wrote:
>Hi,
>
>I am working on the VIFF project (viff.dk) which uses Twisted. I found out 
>that our code is sometimes inefficient because we are generating many 
>deferreds (maybe about 10000) in a callback. While doing that, no network 
>communication is performed. Therefore, I investigated the possibility of 
>adding a function to the reactor which is called every iteration and from 
>which the iteration could be called safely. Then, we could generate all 
>deferreds in that function and activate the reactor from to time. See the 
>attached patch for details.

So you're doing a ton of work all at once now and you want to split up that
ton of work into smaller pieces and do it a little at a time?

If that's the case, then you don't need to modify the reactor, you just
need to split up the work your code is going.  There are a lot of techniques
for doing this.  coiterate and inlineCallbacks are two solutions which are
closest to "cookie cutter" (ie, you have the least flexibility in deciding
how to use them).

>Some of our code runs twice as fast when using that hack. Are there any 
>chances something similar might be included in Twisted? Or does anyone have 
>a better solution for the described problem?

Give coiterate or inlineCallbacks a try.  If you need help applying either
of these to your problem, please ask. :)  I can't make any specific
suggestions now because I can only guess at how you're using the reactor
modification you attached.

You have a very long, steep, uphill battle to convince me that adding support
for re-entrant iteration is a good idea.

>Furthermore, I have a question, which is probably related. The documentation 
>of IReactorCore says about the iterate() function:
>
>"The reactor must have been started (via the run() method) prior to any 
>invocations of this method. It must also be stopped manually after the last 
>call to this method (via the stop() method). This method is not re-entrant: 
>you must not call it recursively; in particular, you must not call it while 
>the reactor is running."
>
>This looks to me as if the reactor needs to be running and not running at 
>the same time so that iterate() can be called. Is there an error in my 
>reasoning?

It's very subtle and intentionally written to sound impossible to use (maybe
that is a stupid idea, but that was my intent when I wrote it ;).  What
you're missing is that there is a way to make reactor.run() return without
stopping the reactor.

Jean-Paul