[Twisted-Python] deferred TCP writes

[glyph at divmod.com]

On Tue, 18 Apr 2006 10:44:22 -0700, Brian Granger <bgranger at scu.edu> wrote:
>I echo the question: why doesn't write return a deferred?  Surely
>there are times you want to trigger events on the completion of a
>write.

Some folks have already answered with the primary reason, which is to say, efficiency.  While Twisted definitely trades performance for flexibility in a number of places where reasonable, 'write()' is one of those things that just absolutely has to be as fast as possible, because many things which are highly  performance-critical call it.  Creating Deferreds could be 100x faster and it would still be too slow for this case.

There is anothe reason, though: returning a deferred from "write()" implies to the user that when the deferred fires, the bytes have been "written", e.g. the other end has received them.  In cases where .write() blocks, most naive programmers -- and yes, I say this having once been a "naive programmer" myself -- assume that the other end has not only received the bytes, it has processed them.  This is wrong, and can (and often does) have unfortunate consequences.

It sounds like you want control over excessive in-memory buffering, which is a pretty reasonable thing to want, and the IConsumer/IProducer interfaces you have already been directed to should provide that to you quite satisfactorily.

The reality is that when a 'write' completes and returns to userland, your bytes are somewhere between your kernel, your router, your cable modem, your ISP's cable modem, your ISP's internal routers, the backbone (...) your destination's kernel, and your destination's application.  You have no idea where they are, really, and neither does Twisted.  It just hides one more distinction from you: Twisted's buffer to your OS's buffer.