[Twisted-Python] Advice on doing thousands of simultaneous UDP queries with Twisted...

[Mike C. Fletcher]

I've written a UDP-based protocol adaptor (TwistedSNMP) where one of the 
key requirements is the ability to scan thousands of SNMP Agents 
"simultaneously" (i.e. the requests should be asynchronously sent and 
retired as the Agents respond).

http://members.rogers.com/mcfletch/programming/index.htm#TwistedSNMP

Writing a simple asynchronous loop myself (poll on a simple socket, send 
message from queue when writable, read one into other queue when 
readable) allowed for doing a few thousand queries simultaneously), with 
only a few dozen dropped messages.  However, with the Twisted equivalent 
(UDPTransport with my simple protocol object), I was seeing huge drop 
rates, so, gathering that Twisted isn't queueing up the UDP requests, I 
wrote a (byzantine) query-throttling mechanism with Twisted defers.

Problem is, it's a byzantine, fragile (and *slow*) solution to what 
would *seem* to be one of the most common requirements in networked 
development.  Worse yet, because I am seeing such high drop rates I wind 
up having to batch in very small groups, serially (instead of 
parallel-ly), so the primary purpose of the system (fast querying of 
thousands of agents) is lost.  (Instead of taking 1 or 2 minutes to 
query 800 or so Agents it will take on the order of 10 minutes.)

So, the question: is there a simple way to turn on buffered mode in UDP 
transports so that they can deal with queueing up a few thousand 
messages to send, sending them, then having a few thousand computers 
send a reply (within a few seconds of one another)?  Is Twisted just not 
Queue-reliant via use of some mechanism I haven't discovered yet?  Even 
if I do find a decent queueing mechanism, I'm still left with the 
problem that timeouts and the like are going to wind up being measured 
from queueing-time, rather than sending time... not an issue if 
everything gets sent in a half-second or so, but a real problem if it 
takes 8 or 9 seconds just to send the original messages out.

Looking at the udp.Port class, I'm not seeing anything providing a 
queue, seems as though there's a non-blocking write or read, but nothing 
to handle overflows of sends or receives AFAICT, though it looks as 
though a protocol could do some queueing on incoming in its 
datagramReceived... just not sure how that would work.

Thoughts appreciated,
Mike

_______________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://members.rogers.com/mcfletch/