[Twisted-Python] Async-pep (again)

[Tim Allen]

On Thu, Jul 14, 2011 at 10:05:00AM +0200, Laurens Van Houtven wrote:
> On Thu, Jul 14, 2011 at 8:48 AM, Tim Allen <screwtape at froup.com> wrote:
> > The idea of Protocols implementing Transports is vaguely gestured at as
> > a Useful Thing, but not much detail is given. I think it would be useful
> > for the final PEP to address that topic more rigorously - partially
> > because it's good to have a firm basis on which to model SOCKS and SSH
> > libraries, but mostly because figuring out how SSL should interact with
> > TCP is going to give people headaches. Twisted, so far as I can see,
> > just sort of punts and says "Yeah, SSL is just another transport like
> > TCP", but then you have to make the SSL transport support all the same
> > options that the TCP transport supports (socket options? IPv6?), but
> > then what if you want to run SSL over a serial port or a SOCKS
> > connection... AAAAAAAAAAAAA!
> >
> > In practice, it might be simpler because "SSL" means "whatever subset of
> > TCP functionality we can coax OpenSSL into providing" rather than
> > a fully stackable protocol-providing-a-transport.
> 
> Cool. Can I shove those 2 paragraphs into a ticket or will the copyright
> monster haunt me?

Go right ahead! I guess most of these things should be tickets, but
I don't have a GitHub account and I'm not particularly looking to
register on more websites at the moment.

> > The thing with Consumers and Producers seems... very abstract. If I'm
> > sitting down to retrieve email via POP3 (to pick a random protocol),
> > 'transports' and 'protocols' are tools that nestle very comfortably in
> > my mental model of the task in front of me; "consumers" and "producers"
> > are not. Are they concepts that should be handled by transport
> > implementors? Protocol implementors? Protocol users? Should they be
> > mapped onto XON/XOFF or RTS/CTS by serial transports?
> 
> Yes, Consumers and Producers are about flow control, and most Transports
> probably are producers.

Having looked at the issues list after sending that message, I see this
is basically issue 13, "Why are producers/consumers important, how are
they different from protocols/transports?"

If your PEP includes producers and consumers (and I note that the
current example code doesn't, it just has a "FlowControl" class), you'll
want to have an example Protocol that uses producers and consumers in
some useful, illustrative fashion.

> > At least in Twisted, transports and protocols do not exist in a vacuum;
> > they have to be hooked up via the reactor. Will this PEP define
> > a (skeletal) API to be implemented by potential reactors, or is that
> > going to left entirely unspecified, like WSGI?
> 
> Entirely unspecified, because different implementations have to do pretty
> different things.

I guess the selection of available Transports is up to the hosting
event-loop, too - it might be worth noting that in the section on
"Transports". Unless, of course, the Transport in question is
implemented by another Protocol, in which case I guess it's anybody's
guess how you might hook your Protocol up.

I almost think that, for pedagogy's sake, there should be an additional
Encapsulator or Framer abstract class, that inherits from Protocol, but
adds a .connectProtocol() method that takes another Protocol instance,
and hooks itself up as that Protocol's transport. Sure, anyone who
understands what's going on should be able to figure out what's going
on, but I think an extra class would make it blindingly obvious, and
I like APIs that save me from having to think too hard.

While there's still people listening to my half-formed opinions:

- Issue 7 seems to have settled on removing support for half_closing
  transports. I seem to recall somebody mentioning support for
  half-close as being one of those weird-corner cases that nobody thinks
  they need until they're trying to figure out why their SSH sessions
  always die with "broken pipe" errors. While it probably would
  complicate the documentation to include it, I'd hope that many
  frameworks that implement this PEP would want to include support for
  half-closing transports, and it'd be nice if there was a standard API
  for it instead of everybody adding their own methods with their own
  semantics. Perhaps there could be an HalfClosableTransport(Transport)
  ABC, that's optional in the same way that, say, DB-API's
  "connection.rollback()" method is defined but optional.

- For issue 6 ("Scatter/gather IO API doesn't make sense yet"), I can't
  see much of a use for readv/scatter, because I imagine the benefits
  come from having a bunch of pre-allocated buffers lying around, and
  even if the Python VM had such buffers, they probably wouldn't be
  visible or useful to running Python code. On the other hand, I can
  easily imagine Python code having a bunch of independently-generated
  buffers that need to be written in a particular order (framing data
  and framed data, for example), and being able to avoid ''.join() could
  be a big win. Again, perhaps this could be an optional extension
  provided by hosting frameworks that want to implement it - although it
  should be pretty easy to emulate on top of the regular .write()
  method.

- You might also want to create an optional Transport method to wrap the
  sendfile(2) and/or splice(2) functions. Without OS support, it'd just
  be a convenient way to tell the host framework to do the grunt-work of
  shunting bytes around; with OS support it should be a good deal more
  efficient than doing the same operations manually.
  
Tim.