[Twisted-Python] TCP Proxy scalability issue

[glyph at divmod.com]

On 1 May, 04:31 pm, saurav.mohapatra at dimdim.com wrote:
>Glyph,
>
>Thanks for the response.

No problem.  Thanks for your clearly described inquiry :).
>We're trying to proxy multiple VNC processes for a screen sharing
>application. So though CPU load is low, the data being pushed out is
>pretty high. We have around 16Kb of data being pushed out per second 
>per
>client for a normal case. The test server is running on EC2, but we see
>the degradation in the production deployment (on physical machines
>running on a cogent fiber connection). I was thinking it might have
>something to do with amount of data being written and number of
>simultenous writes issued. Something to do with transport.write()

When you do a transport.write, twisted puts the data into a buffer on 
your transport and marks it as ready to write some data.  When you next 
get into epoll, epoll should wake up immediately, showing that lots of 
FDs are ready for writing, and Twisted will then immediately begin 
attempting to write all of the buffered data to those connections.

All of those sockets are in non-blocking mode and so send will 
immediately return regardless of how much data can actually be put onto 
the wire.

Therefore, in principle, if Twisted is not 100% busy on the CPU, then 
nothing *should* be causing it to deliver your data slower than it's 
coming in.  Unless, of course, there's a bug, which is always possible. 
But this is a pretty simple bit of code and inspecting it in the course 
of this email I don't see many places for such a bug to hide; so in this 
case I'd probably start the bug hunt in the application.  Or at least, 
by writing the simplest possible benchmark for Twisted itself, so that 
we can see the "raw" performance regardless of any application logic.
>I can make a benchmarking application available to you, please mail me
>off the list at saurav.mohapatra at dimdim.com

I have done so.
>Regards
>Saurav