[Twisted-Python] Freevo 2.0, Kamaelia, pyevent, Eventnet/LGT: what's going on?

[Andrew Bennetts]

On Fri, Oct 21, 2005 at 01:10:03AM +0200, Antoine Pitrou wrote:
[...]
> 
> There is a terminology problem, since I don't understand why something
> named "Protocol" should be destroyed at every connection. Or it should
> be, quite logically, renamed "Connection". Also this doesn't appear in
> the API docs (again).

I think high-level concepts like this are much better explained through
proper documents than docstrings (which is the source of our API docs).
Docstrings are generally like man pages; they give a basic reference for
something, if you're lucky an example even, but for anything more than that,
like understanding the design, you really want a more comprehensive and
coherent document than a collection of docstrings.  API docs are mainly
useful to people that already understand what's going on.

That's not to say our documentation in this area is perfect.  You're right
that the design of protocols and factories isn't really discussed
explicitly; either you find it by working through the tutorial, or perhaps
from
http://twistedmatrix.com/projects/core/documentation/howto/servers.html.
Improvements welcome! ;)

As to why it's "Protocol" rather than "Connection"... that would be
differently confusing :)

The "connection" to me sounds like what the transport already is -- the
thing responsible for taking bytes (or datagrams) from here and sending them
to there, and that you can disconnect, and so on.  In Twisted, the
"Protocol" is the thing responsible for interpreting events from the
underlying connection (and in most cases, doesn't care if the underlying
transport is TCP, SSL, UNIX domain sockets, or whatever -- how the bytes are
transported isn't its responsibility).

> For me, a "protocol" is something like HTTP or SMTP. It is "eternal",
> I'm not expecting it to be created/destroyed at every connection (that
> would be like calling an int a float, or whatever).

Well, the logic of how to interpret the byte stream into higher-level events
is "eternal": your Protocol *class* is that.  The actual interpretation
(e.g.  "ok, I've just seen '\n' so this is a lineReceived event... ok, I've
just seen a blank line and I was in the 'expecting HTTP headers' state so
now I'm in the 'expecting request body state', but this is a GET request so
there's no body, so this is a 'requestReceived' event") is almost always
stateful though, and state means objects, i.e. instances.  

So: you need an interpreter per connection (because interpreting a byte
stream is stateful), and interpreters ought to be able to work with any byte
stream, because most of the time there's no or little difference.  So the
interpreter and byte-stream have different objects: a Protocol and a
Transport.

> There is no documentation on the magic that really happens when
> something is triggered (for example when a TCP session is established).
> Which methods are called, which objects are created, in which order,
> etc. Unexplained magic is annoying in software development.

There's not much magic, but there's also not much documentation.  Of course,
the end result is of inadequate documentation isn't very distinguishable
from magic :(

Depending on what levels of abstraction you care about, see the interfaces
for IReactorTCP, IConnector, IProtocolFactory, IProtocol and ITransportTCP,
and Factory itself:
    http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.html
    http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Factory.html

Or ask specific questions on this list!

I hope I've helped,

-Andrew.