[Twisted-Python] Twisted receiving buffers swamped?

[Glyph Lefkowitz]

> On Jan 2, 2015, at 4:06 AM, Tobias Oberstein <tobias.oberstein at tavendo.de> wrote:
> 
> Fact is: somehow memory runs away.
> 
> How do I track down _where_ exactly the mem is spent? Probably that leads to the "why" then ..

The first place to look - since sometimes looking in a specific place makes memory profilers easier to use - would be twisted.internet.tcp.Server._tempDataBuffer.

The fact that it's extremely slow when you turn on consumer/producer logic in this way makes sense to me.  twisted.internet.abstract.FileDescriptor.bufferSize is hard-coded to 65,536 bytes; every time the write side outpaces the read side by that buffer size, it will result in a call to (in your case) twisted.internet.kqreactor.KQueueReactor._updateRegistration, which makes the kcontrol syscall.

One thing you might try is to run with the environment variable PYPYLOG=jit-summary:- set.  This will give you a bunch of statistics about what the JIT did at the end of the run, on standard out; compare the fast (and run out of memory) to the slow (and work right) run to see what the differences are.

-g
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20150102/10773958/attachment-0002.html>