[Twisted-Python] LoopingCall at a non-idling reactor

[Jean-Paul Calderone]

On Sun, 19 Jul 2009 01:04:25 +0400 (MSD), Ilya Etingof <ilya at glas.net> wrote:
>
>After doing some more research, I've realised that my previous
>conclusion and complaint was indeed inaccurate. Sorry for that!
>
>Now it looks to me that if my datagram receiver function takes some
>time for data processing, rapidly coming datagrams tend to stack up and
>my timer function is not getting called at time.

Ah, it looks like Terry probably guessed the cause of your difficulties
correctly.

>
>Here's example code:
>
>from time import time, sleep
>from twisted.internet import reactor, task
>from twisted.internet.protocol import DatagramProtocol
>
>def timerCbFun():
>   print 'timer called', time()
>
>loopingCall = task.LoopingCall(timerCbFun)
>loopingCall.start(1, False)
>
>class MyProtocol(DatagramProtocol):
>   def datagramReceived(self, datagram, address):
>     print 'datagramReceived', time()
>     sleep(0.2) # simulate data processing

The above is something which you're more or less not allowed to do in a
Twisted application.  That simulated data processing is blocking the
main event loop.  Nothing else happens until it finishes - that includes
handling more UDP packets and it includes running timed events like the
ones LoopingCall sets up.

You need to find another way to do your data processing if this behavior
isn't acceptable.  You can use Twisted's thread pool, if the data processing
is thread safe, or you could try Ampoule to distribute the processing to a
process pool instead of a thread pool, or if the data processing involves a
blocking operation which can be be handled asynchronously somehow, there
might be another Twisted API which can help you out.

Also, aside from that, LoopingCall is designed to handle missed intervals
gracefully.  It will skip calls which cannot be made because the process
was too busy doing something else.  For a 1 second looping call, once you
fix your data processing code, this probably won't happen very often, but
it may happen.  There's some work in progress to make it possible to at
least learn when a call has been skipped, but short of a real time platform
(which Twisted is certainly not), you can't get a 100% guarantee.

Jean-Paul