[Twisted-Python] Twisted and SNMP (was: Advice on doing thousands of simultaneous UDP queries with Twisted...)

[Thomas Favier]

Mike C. Fletcher wrote:
> If for no other reason than that it convinced me there wasn't a simple 
> and elegant solution readily available to be dropped in for a UDP system 
> :) .  In the final analysis, what was happening is that individual 
> queries were timing out *before the queries were even sent* because it 
> was taking so incredibly long to format and send all of the queries.

You bring my point with SNMP : I've tried to build a SNMP poller based 
on twisted and pySNMP. Main trouble i've had is the decoding time of the 
pure python ASN parser. It spent about 1 full second decoding (packets 
are about 7ko, as i get all values and tables with a GET-BULK, there is 
room for improvement by splitting requests, but that's not significant 
for that matter). So i switched to a threaded implementation using 
yapsnmp (and ugly deferToThreads). Decoding times are about 0.02 seconds 
for the same requests. Load on the server was more than halved.

Seems that you are hit by the same kind of problems...

I've dug the source of UCD-SNMP but was not able to see a simple hack to 
integrate the UCD packet decoding and the twisted loop, especially with 
the parts dealing with SNMPv3 (UCD- and NET-snmp almost forces you to 
use their select wrapper...) Moreover, the last yapsnmp release is based 
on v4.3 so a little out of date. As my solution works for my (smaller) 
needs, i currently have no plan to improve my solution, but if anyone is 
interested in bringing a full high performance ASN coder-decoder to 
twisted, i'd be willing to help.


-- 
Thomas FAVIER
.accelance msp
Tel. +33 4 26 29 12 22
http://www.accelance.fr