[Twisted-Python] Strange behavior when transferring large strings

[Gabriel Rossetti]

Jean-Paul Calderone wrote:
> On Wed, 23 Apr 2008 10:41:29 +0200, Gabriel Rossetti 
> <mailing_lists at evotex.ch> wrote:
>> [snip]
>> Ok, so apparently it (Twisted, Python, the OS?) can buffer and send 
>> up to 16k, after that it splits the message up, thus this would 
>> explain the lockup, since the app expects a whole XML message and 
>> can't process the second part correctly. So if I understand it 
>> correctly, even when I use the P/C paradigm, it waits until the 
>> buffer is full before sending the message, so when I get the message 
>> on the other side, I only get part of it, the other half is a new 
>> message/data arrival, and thus like I said above, the XML is not 
>> correct.
>>
>> Does anyone have an idea of how to solve this?
>
> This can't be solved.  From the way you describe it, the software on the
> receiving end is simply broken.  TCP provides no guarantees about how 
> much
> data will be delivered to the recipient at a time, regardless of how much
> is sent at a time.  Every participant along the delivery path between the
> sender and the recipient is allowed to break packets into smaller pieces
> or coalesce packets into larger pieces.  The recipient *must* be able to
> handle incomplete messages by waiting for more bytes.  It must also be 
> able
> to handle packets which contain bytes from more than one message.
>
> There are a number of ways to address this.  Almost all of them involve
> changing the software running on the peer you're sending messages to and
> the protocol the two programs are using to talk to each other.  For
> example, you can send a length prefix before each message allowing the
> recipient to buffer up the correct number of bytes before trying to deal
> with them.
>
> Jean-Paul
>
Thank you Jean-Paul, I'll do something like that then and fix the peer, 
I didn't realize that and thought I could get the message as a whole 
somehow. At lease I know my P/C is working :-)

Gabriel