[Twisted-Python] Sending a long string/buffer without copying it

[Jean-Paul Calderone]

On Thu, 28 Sep 2006 00:11:41 -0600, Brian Granger <ellisonbg.net at gmail.com> wrote:
>HI,
>
>In one of my Twisted based applications, I need to send large string
>and buffers.  They can be 100's of MB's long (they come from large
>numpy arrays).  I would like to be able to send them *without making
>any copies* in the process.
>
>This seems to be dificult with the way that certain parts of Twisted
>are written:
>
>in protocols.basic many of the sendString/sendLine method having
>things that make a copy of the string or line to be send:
>
>    def sendLine(self, line):
>        """Sends a line to the other end of the connection.
>        """
>        return self.transport.write(line + self.delimiter)
>
>If line is 100MB, this just made a second 100MB string.  To make
>things worse, in my case a server needs to send this line to many
>clients that are connected.  The line gets copied for each client!  If
>I have 10 clients, I have nearly a GB worth of extra memory allocated
>for this temporary copy.
>
>This problem is easy solve at the protocol level: you just do separate
>writes for the delimiter and the line.  Or if you are using a length
>prefixed protocol, write the length bytes and the string separately.
>
>BUT....
>
>Even if I do that, it appears that Twisted is making copies elsewhere
>- like in FileDescriptor.doWrite.  So, how can I send something
>without making a copy?  I don't mind making copies of slices, just not
>the whole thing.

Don't pass the entire thing to a single call to transport.write() (or
LineReceiver.sendLine).  Instead, write a producer.

Jean-Paul