[Twisted-Python] Strange behavior when transferring large strings

[Jean-Paul Calderone]

On Wed, 23 Apr 2008 10:41:29 +0200, Gabriel Rossetti <mailing_lists at evotex.ch> wrote:
> [snip]
>Ok, so apparently it (Twisted, Python, the OS?) can buffer and send up to 
>16k, after that it splits the message up, thus this would explain the 
>lockup, since the app expects a whole XML message and can't process the 
>second part correctly. So if I understand it correctly, even when I use the 
>P/C paradigm, it waits until the buffer is full before sending the message, 
>so when I get the message on the other side, I only get part of it, the 
>other half is a new message/data arrival, and thus like I said above, the 
>XML is not correct.
>
>Does anyone have an idea of how to solve this?

This can't be solved.  From the way you describe it, the software on the
receiving end is simply broken.  TCP provides no guarantees about how much
data will be delivered to the recipient at a time, regardless of how much
is sent at a time.  Every participant along the delivery path between the
sender and the recipient is allowed to break packets into smaller pieces
or coalesce packets into larger pieces.  The recipient *must* be able to
handle incomplete messages by waiting for more bytes.  It must also be able
to handle packets which contain bytes from more than one message.

There are a number of ways to address this.  Almost all of them involve
changing the software running on the peer you're sending messages to and
the protocol the two programs are using to talk to each other.  For
example, you can send a length prefix before each message allowing the
recipient to buffer up the correct number of bytes before trying to deal
with them.

Jean-Paul