[Twisted-Python] Twisted receiving buffers swamped?

[Tobias Oberstein]

>>I would need to somehow justify/book these investments. I have "ideas" about that, but step by step: what do you think about above?

>It sounds like http://speed.twistedmatrix.com but far more ambitious :).  Are you familiar with that site, and the benchmarks repository that powers it?  It's nowhere near as comprehensive as what you'd like, but it is a good >place to start.

I have stumbled across this before, but haven't looked into much. I am looking right now ..

On the one hand: it has a _lot_ of testee scenarios, not just TCP echo, but e.g. also TLS, which I am definitely interested in.

I was only proposing to have a trivial "TCP echo", essentially the couple of lines from the Twisted homepage - not very ambitious;)

However, as far as I can see, speed.twistedmatrix.com runs all tests over loopback with both the load generator and the testee in Twisted - even in 1 process.

http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/view/head:/tcp_throughput.py
=> uses producer/consumer

https://github.com/twisted/twisted/blob/trunk/twisted/protocols/wire.py#L17
=> does not use producer/consumer (like you suggested https://github.com/oberstet/scratchbox/blob/master/python/asyncio/tcp_echo_server_tx.py#L16)

For various reasons, I don't think this would have been able to demonstrate the issue I ran into .. swamping.

You need a very fast sender (netperf) to even be able to swamp the receiver. Then I am not interested (much) in loopback.

It is these performance test "setup/system things" were my proposal is indeed far more ambitious.

And for me this is crucial: I want hard numbers which are _representative_ of what happens in production (at least close to). 

>I'm particularly curious about the specific performance case we were talking about though; wire speed with flow-control is an important use-case for Twisted and it sounds to me like we may be falling very far short.

Yes, I understand.

Note that we can't get to 10GbE wirespeed in a throughput test with Twisted even leaving "swamping" aside. And I verified that the hardware/OS is not the bottleneck (I can saturate the link at 9.94Gb/s using netperf-to-netserver easily).

/Tobias