[Twisted-Python] Advice on doing thousands of simultaneous UDP queries with Twisted...

[Stephen Thorne]

Have a look at htb.py, you should be able to rate limit the udp
throughput rather easily.

For me doing web stuff it was (in its entirety) like this:


def RateBucket(parent=None):
    bucket = htb.Bucket(parent)
    bucket.rate = 4000
    bucket.maxburst = 12000
    return bucket

bucketFilter = htb.FilterByHost()
bucketFilter.bucketFactory = RateBucket

application = service.Application("simple")
site_factory = server.Site( Directory() )
internet.TCPServer( 
    SERVER_PORT, 
    site_factory
).setServiceParent(application)
site_factory.protocol = htb.ShapedProtocolFactory(
    site_factory.protocol, 
    bucketFilter
)

Hope that helps.

Stephen Thorne

On Mon, 2004-02-09 at 20:51, Mike C. Fletcher wrote:
> I've written a UDP-based protocol adaptor (TwistedSNMP) where one of the 
> key requirements is the ability to scan thousands of SNMP Agents 
> "simultaneously" (i.e. the requests should be asynchronously sent and 
> retired as the Agents respond).
> 
> http://members.rogers.com/mcfletch/programming/index.htm#TwistedSNMP
> 
> Writing a simple asynchronous loop myself (poll on a simple socket, send 
> message from queue when writable, read one into other queue when 
> readable) allowed for doing a few thousand queries simultaneously), with 
> only a few dozen dropped messages.  However, with the Twisted equivalent 
> (UDPTransport with my simple protocol object), I was seeing huge drop 
> rates, so, gathering that Twisted isn't queueing up the UDP requests, I 
> wrote a (byzantine) query-throttling mechanism with Twisted defers.
> 
> Problem is, it's a byzantine, fragile (and *slow*) solution to what 
> would *seem* to be one of the most common requirements in networked 
> development.  Worse yet, because I am seeing such high drop rates I wind 
> up having to batch in very small groups, serially (instead of 
> parallel-ly), so the primary purpose of the system (fast querying of 
> thousands of agents) is lost.  (Instead of taking 1 or 2 minutes to 
> query 800 or so Agents it will take on the order of 10 minutes.)
> 
> So, the question: is there a simple way to turn on buffered mode in UDP 
> transports so that they can deal with queueing up a few thousand 
> messages to send, sending them, then having a few thousand computers 
> send a reply (within a few seconds of one another)?  Is Twisted just not 
> Queue-reliant via use of some mechanism I haven't discovered yet?  Even 
> if I do find a decent queueing mechanism, I'm still left with the 
> problem that timeouts and the like are going to wind up being measured 
> from queueing-time, rather than sending time... not an issue if 
> everything gets sent in a half-second or so, but a real problem if it 
> takes 8 or 9 seconds just to send the original messages out.
> 
> Looking at the udp.Port class, I'm not seeing anything providing a 
> queue, seems as though there's a non-blocking write or read, but nothing 
> to handle overflows of sends or receives AFAICT, though it looks as 
> though a protocol could do some queueing on incoming in its 
> datagramReceived... just not sure how that would work.
> 
> Thoughts appreciated,
> Mike
> 
> _______________________________________
>   Mike C. Fletcher
>   Designer, VR Plumber, Coder
>   http://members.rogers.com/mcfletch/
> 
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python