[Twisted-Python] Twisted receiving buffers swamped?

[exarkun at twistedmatrix.com]

On 1 Jan, 10:21 am, tobias.oberstein at tavendo.de wrote:
>Hi,
>
>I am doing network performance tests using netperf on a trivial Twisted 
>TCP echo server (code at the end).
>
>One of the tests that netperf offers is throughput, and I am running 
>into an issue with this.
? [snip]
>
>Now, my suspicion is that Twisted is reading off the TCP stack from the 
>kernel and buffering in userspace faster than the echo server is 
>pushing out stuff to the TCP stack into the kernel. Hence, no TCP 
>backpressure results, netperf happily sends more and more, and the 
>memory of the Twisted process runs away.

What you said here about "buffering in userspace" is ambiguous.  It's 
not clear if you meant data is being buffered in userspace on the read 
side before your protocol gets a chance to handle it or if you meant 
that data being written to the transport by the protocol is being 
buffered in userspace.

The former doesn't happen.  There are no no userspace read buffers in 
Twisted between the transport and the protocol.  Bytes are read from the 
socket and then passed to dataReceived.

The latter would be addressed by using producer/consumer APIs as Glyph 
suggested.

Jean-Paul