[Twisted-Python] Re: [Twisted-commits] r10475 - Use writeSequence instead of silly for-loop.

[James Y Knight]

On Apr 5, 2004, at 10:36 AM, Andrew Bennetts wrote:

> Author: spiv
> Date: Mon Apr  5 08:36:43 2004
> New Revision: 10475
>
> Modified:
>    trunk/twisted/protocols/http.py
> Log:
> Use writeSequence instead of silly for-loop.

You might not have noticed that writeSequence is implemented like so:
   self.write("".join(iovec))
thus negating the performance enhancement recently added by not copying 
large data into a new string.

This is of course quite silly, and I suspect a nice performance 
enhancement could be achieved by making FileDescriptor.dataBuffer be a 
list of strings, rather than a single string. That would both cause 
multiple writes in a row to not allocate new strings, and allow 
writeSequence to be implemented sanely.

Doesn't look very hard really; looks like you'd only need to touch 
write/writeSequence/doWrite. This'd also have the advantage of making 
it easier to put a Sendfile-struct in there as well. So pahan's 
volunteered to fix this!

Unfortunately, python doesn't expose the writev syscall, which means 
that you depend on Nagling to not coalesce the TCP packets. There *is* 
a TCP option which you can use to help fix too many small packets: 
TCP_CORK on linux and TCP_NOPUSH on FreeBSD. This keeps partial packets 
from being sent at all until you close the connection or turn off the 
option (except on older freebsd, where it's broken and you need to turn 
off the option and call write again to wake it up).

I suspect it's worthwhile to set TCP_CORK/TCP_NOPUSH while the 
python-side buffer has any data left in it, or else the kernel may be 
deciding to send out small packets when it should be waiting for more 
data from an upcoming write call and wasting time/bandwidth. Python has 
the TCP_CORK constant, even, yay! FreeBSD will  just have to suffer, I 
suppose, since python doesn't know about TCP_NOPUSH.

James