[Twisted-Python] Twisted receiving buffers swamped?

[Tobias Oberstein]

> >Now, my suspicion is that Twisted is reading off the TCP stack from the
> >kernel and buffering in userspace faster than the echo server is
> >pushing out stuff to the TCP stack into the kernel. Hence, no TCP
> >backpressure results, netperf happily sends more and more, and the
> >memory of the Twisted process runs away.
> 
> What you said here about "buffering in userspace" is ambiguous.  It's not
> clear if you meant data is being buffered in userspace on the read side
> before your protocol gets a chance to handle it  

Yes. That's what I meant. Buffering in userspace inside Twisted, and before data is handled by the app in dataReceived.

> .. or if you meant that data
> being written to the transport by the protocol is being buffered in userspace.

Nope, I didn't meant that. That's the sending side.

> 
> The former doesn't happen.  There are no no userspace read buffers in
> Twisted between the transport and the protocol.  Bytes are read from the

Ok.

> socket and then passed to dataReceived.
> 
> The latter would be addressed by using producer/consumer APIs as Glyph
> suggested.

Mmh.

Fact is: somehow memory runs away.

How do I track down _where_ exactly the mem is spent? Probably that leads to the "why" then ..

/Tobias