[Twisted-Python] LoopingCall at a non-idling reactor

[Phil Mayers]

On Sun, Jul 19, 2009 at 09:01:34AM +0100, Ilya Etingof wrote:
>
>[ skipped ]
>
>> The above is something which you're more or less not allowed to do in a
>> Twisted application.  That simulated data processing is blocking the
>> main event loop.  Nothing else happens until it finishes - that includes
>> handling more UDP packets and it includes running timed events like the
>> ones LoopingCall sets up.
>
>But why main loop does not fire timed events when it has a chance to do 
>that? Note that my data processor takes 0.2 sec on each run, while my 
>timed event period is 1 sec.
>
>In other words, I'd understand this behavior if my data processor
>would block main loop for a few periods of timed event, but this
>is not the case.

Because Twisted receives as many datagrams as possible before going back 
round the select loop (I think). So, 10 calls to datagramReceived are 
done (taking 10*0.2 seconds) before twisted gets a chance to schedule 
any other things.

As JP has said, you're not allowed to block the main loop like that, but 
I've seen similar problems with even relatively quick datagramReceived 
handlers (~0.1msec) under very high load (1000pps) resulting in a form 
of starvation. This isn't generally a problem with TCP calls due to flow 
control.

The solution is:

  def datagramReceived(...):
    reactor.callLater(0, dorealwork, ...)

  def dorealwork(self, ...):
    ...

This will schedule the "real" work as soon as possible in the next 
reactor loop, but also "fairly" in line with other calls.


You still shouldn't block the reactor for a "long" time (the exact value 
is dependent on your app)