[Twisted-Python] Twisted receiving buffers swamped?

[Tobias Oberstein]

Hi,

I am doing network performance tests using netperf on a trivial Twisted 
TCP echo server (code at the end).

One of the tests that netperf offers is throughput, and I am running 
into an issue with this.

When running the test (loopback) for 10 seconds on the test box I get a 
throughput of 11.5 Gb/s (which is not bad):

[oberstet at brummer1 ~]$ netperf -N -H 127.0.0.1 -t TCP_STREAM -l 10 -- -P 
9000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 9000 AF_INET to 127.0.0.1 () 
port 9000 AF_INET 
         : no control : histogram : interval : dirty data : demo
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

      0  32768  32768    10.02    11517.89

[Sidenote: when running against the netperf server, the box does 46Gb/s 
on that test. More results here: 
https://github.com/crossbario/crossbar/wiki/Stream-Testee#netperf]

However, when I run the test for 60 s (changing to "-l 60" in above will 
do), the test server is killed by the OS due to out-of-memory.

This screenshot

http://picpaste.com/pics/Clipboard03-DllXR7QE.1420107551.png

shows that the server at the time immediately before of killing 
allocated >30GB RAM.

In fact, memory also runs away with 10 sec test .. it's just that the 
machine has enough RAM to cope with that. So it's a "general" issue.

I tested with:

* CPython 2.7.9 and PyPy 2.4
* select, poll and kqueue reactors

all on FreeBSD 10.1. Same behavior for all combinations.

===

Now, my suspicion is that Twisted is reading off the TCP stack from the 
kernel and buffering in userspace faster than the echo server is pushing 
out stuff to the TCP stack into the kernel. Hence, no TCP backpressure 
results, netperf happily sends more and more, and the memory of the 
Twisted process runs away.

I am aware of 
http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html, but 
that seems to cover the sending side only.

What's the cause? What can I do?

How do I prevent Twisted to read off sockets from kernel as the 
userspace buffer grows?

E.g. can I set a limit on the userspace buffer, so Twisted won't read 
out the sockets until the app has consumed more of the already buffered 
stuff?

Any hints appreciated,

Cheers,
/Tobias



TCP Echo Server used ====>

from twisted.internet import kqreactor
kqreactor.install()
#from twisted.internet import selectreactor
#selectreactor.install()
#from twisted.internet import pollreactor
#pollreactor.install()

from twisted.internet import protocol, reactor, endpoints

class Echo(protocol.Protocol):
     def dataReceived(self, data):
         self.transport.write(data)

class EchoFactory(protocol.Factory):
     def buildProtocol(self, addr):
         return Echo()

endpoints.serverFromString(reactor, "tcp:9000").listen(EchoFactory())

print "running on ", reactor.__class__
reactor.run()