[Twisted-Python] dataReveived() buffer best practice?

[Glyph]

On Oct 7, 2011, at 6:49 PM, Phil Mayers wrote:

> On 10/07/2011 02:11 PM, Glyph wrote:
> 
>>> It is quite unusual (though not unheard of) for something to
>>> re-segment the TCP stream. IP level fragmentation might occur, but
>>> it's relatively uncommon in todays IP networks, and is anyway
>>> irrelevant to TCP - the TCP stack will only see a reassembled IP
>>> packet.
>> 
>> I am skeptical; my recollection (the last time I worked at this layer
>> of the network) is that this happens all the time over the public
>> internet between diverse endpoints.
> 
> Are we talking about TCP re-segmentation, or IP fragmentation?

Pretty sure I'm talking about TCP re-segmentation.

> If you mean TCP re-segmentation, then I have no data. The only places 
> I've personally seen it are in TCP terminating load-balancers (F5, 
> etc.), NAT64 devices, and as a side effect in higher-layer proxies 
> (Squid, etc.).

Hrm.  Now that I think of it, the packets I was examining were probably coming from environments that were heavily managed by "smart" networking hardware, although I could only see the edges of the network.  So it was probably a combination of high-level transparent proxies and wacky load-balancing and caching hardware.

> If you mean IP fragmentation, I claim that fragmentation by routers in 
> the internet is rare. Whether it's 1% or 0.001% I couldn't say, but the 
> vast majority of the time, routers do not end up fragmenting. For one 
> thing, an awful lot of OSes send TCP segments with "don't fragment" set, 
> to trigger ICMP error messages and path MTU discovery. For another, a 
> lot of low MTU networks employ TCP MSS clamping to prevent this ever 
> happening.

That's interesting, and good to know :).

> For what it's worth, this "layer of the network" is my day job, but I 
> don't claim oracle status.
> 
>> But the sizes are also not important.  The point is that TCP is about
>> streams, not packets, and you have to deal with arbitrary chunking if
>> you want your code to work right.  What layer of the network this
>> happens at is not important to your code :).
> 
> Agreed.

Since this is your day job - is there a formally-correct name for this phenomenon?  I usually refer to it as "packet fragmentation" even though I know that's not really it - as you say, IP fragmentation doesn't really happen. It's not like it's really TCP re-segmenting either, because one send() is not necessarily one segment.

Since it's such an important concept that every Twisted developer needs to know, I'd love to have a good word for it :).