[Twisted-Python] LoopingCall at a non-idling reactor

[Phil Mayers]

On Sun, Jul 19, 2009 at 02:58:40PM +0100, Jean-Paul Calderone wrote:
>
>It will eventually stop reading datagrams and go do something else.  The
>exact way it decides when to stop is completely arbitrary and I don't
>think anyone has ever demonstrated that it's clever or appropriate in the
>general case (it stops after reading 256k of datagrams).

That's interesting. I didn't know that. Has it always done this, or is 
that a recent thing?

As I say, I used to experience starvation of the reactor doing SNMP 
walks of hundreds of devices (in addition to UDP socket buffer 
overflows) which is why I adopted my solution (see below)

>
>reactor.callLater(0, f) isn't magic.  I probably wouldn't have suggested
>it as a solution here.  All it means (more or less) is "do this work after
>all current i/o events have been dispatched."  You still might end up with
>more than one second of processing bunched up together, and so you'll still
>miss a LoopingCall iteration.

For what it's worth, I actually do something a lot more sophisticated. 
My UDP IO, the protocol parsing, re-transmission logic and so forth 
actually happen in a "farm" of N sub-processes (which lets me make use 
of the multi-core CPU) and I talk to the parent process over TCP.

Viewing the reactor as a device that emits a stream of events (timer 
ticks, socket reads, write requests etc.) the problem can be considered 
a QoS issue.  It's hard to see how the reactor could "know" how to do 
the right thing in all cases, absent a hinting mechanism from the 
application:

queue = reactor.getDefaultQueue()
subq = queue.newChild()
subq.setMode('round-robin')
subq.listenUDP(CONTROL, importantProto(), weight=10)
subq.listenUDP(DATA, lessimportantProto(), weight=1)

...or something ;o) This isn't a serious suggestion obviously!