[Twisted-Python] Twisted and SNMP

[Mike C. Fletcher]

Thomas Favier wrote:

> Mike C. Fletcher wrote:
>
>> If for no other reason than that it convinced me there wasn't a 
>> simple and elegant solution readily available to be dropped in for a 
>> UDP system :) .  In the final analysis, what was happening is that 
>> individual queries were timing out *before the queries were even 
>> sent* because it was taking so incredibly long to format and send all 
>> of the queries.
>
>
> You bring my point with SNMP : I've tried to build a SNMP poller based 
> on twisted and pySNMP. Main trouble i've had is the decoding time of 
> the pure python ASN parser. It spent about 1 full second decoding 
> (packets are about 7ko, as i get all values and tables with a 
> GET-BULK, there is room for improvement by splitting requests, but 
> that's not significant for that matter). So i switched to a threaded 
> implementation using yapsnmp (and ugly deferToThreads). Decoding times 
> are about 0.02 seconds for the same requests. Load on the server was 
> more than halved.
>
> Seems that you are hit by the same kind of problems...

...

> but if anyone is interested in bringing a full high performance ASN 
> coder-decoder to twisted, i'd be willing to help.

Ilya (creator of PySNMP) just released a new version of PySNMP 
yesterday.  It's noticeably faster feeling than the 3.3.x version.  I 
don't have time to do timings with it, but I'd guess it's probably 
taking 3/4 of the time of the previous version.  Still likely not 0.02 
seconds on getbulk requests, but noticeably faster.

Profiling the ASN parser and writing a small accelerator module would 
likely be quite doable.  From what I've seen there's not a lot of 
complex code in there.  It's just hidden deep in the package structure.  
Nice thing about that approach is that it's possible to port to new 
platforms and only worry about the C module being present if you need 
the speed there.

My primary interest here isn't really in speed, we query 8000 agents at 
a time, and only need to query once every few *hours*.  The flexibility 
& robustness of the code is my primary interest.  Too much C gets me 
worried :) , particularly with networking code, where buffer overflows 
and the like are a pain to avoid.

Have fun,
Mike

_______________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://members.rogers.com/mcfletch/