[Twisted-Python] Advice on doing thousands of simultaneous UDP queries with Twisted...

[Mike C. Fletcher]

Stephen Thorne wrote:

>Have a look at htb.py, you should be able to rate limit the udp
>throughput rather easily.
>
>For me doing web stuff it was (in its entirety) like this:
>  
>
<bandwidth throttler clipped>

>Hope that helps.
>  
>
If for no other reason than that it convinced me there wasn't a simple 
and elegant solution readily available to be dropped in for a UDP system 
:) .  In the final analysis, what was happening is that individual 
queries were timing out *before the queries were even sent* because it 
was taking so incredibly long to format and send all of the queries.

The manually written system was "pull" based.  It marked the 
time-of-sending for timeout purposes deep in the select loop.  The 
Twisted version is setting the timeout via a callLater when the request 
is *submitted*.  With 2 or 3 thousand queries, the 20 seconds of timeout 
was finishing before any messages were processed from the incoming port, 
then the timeouts would all get called, which kept eating up processing 
time, preventing any messages from getting read, so the port's internal 
buffer would fill up and more timeouts occurred, so basically nothing 
would get through.

Queueing up the outputs before sending didn't help, btw.  Nor would 
simple bandwidth throttling without modifying the timeout characteristics.

My solution, which is rather low tech, is to do an effective "yield" 
after each message is sent, to allow for processing incoming responses.  
This is done with this little method:

    def smallBatch( self, oids, tables, index=0, iterDelay=.01 ):
        if index < len(self.proxies):
            proxy = self.proxies[index]
            self.singleProxy(
                proxy,oids,tables
            )
            reactor.callLater( iterDelay, self.smallBatch, oids, tables, 
index+1 )
        else:
            dl = defer.DeferredList( self.partialDefers )
            dl.addCallback( self.returnFinal )

(which also has a side-effect of introducing a bandwidth throttling), 
but most importantly, it delays setting the timeout on any given query 
until when the query is actually sent (or fairly close to then).  Put 
another way, it gives the system a chance to send the message as soon 
after submitting it as possible.

Have fun all, and thanks, Stephen,
Mike

>Stephen Thorne
>
>On Mon, 2004-02-09 at 20:51, Mike C. Fletcher wrote:
>  
>
...

>>only a few dozen dropped messages.  However, with the Twisted equivalent 
>>(UDPTransport with my simple protocol object), I was seeing huge drop 
>>rates, so, gathering that Twisted isn't queueing up the UDP requests, I 
>>wrote a (byzantine) query-throttling mechanism with Twisted defers.
>>    
>>
...

 
_______________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://members.rogers.com/mcfletch/