[Twisted-Python] Refactoring Documentation

[Tom Davis]

I've been using (and threatening to work on) Twisted for a few years now. It
seems like every time I get back into it, I need to dig up old code or
Google queries just to get started. Yesterday, Jean-Paul introduced me to
the trial-tutorial
branch<http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443-2>
which
has shed some light on the basics of using trial and testing Twisted
client/server applications in general. Until he mentioned it in IRC, I was
stuck looking at the actual tests for protocols and deciding which parts of
that were generically useful to me. I agreed to finish up that documentation
so that it could finally (four years later) be added to trunk (and more
importantly, twistedmatrix.com). But after thinking about it, I believe the
problem runs much deeper than just the lack of a branch merge.

Reading code to find answers isn't rocket science; I've been programming
long enough to be comfortable doing it. But I probably have to resort to
reading Twisted's code about 8 billion percent more often than any other
codebase. And reading code is a hurdle. Reading through Twisted's
semi-random, 45-point FAQ is a hurdle—and recommending it as a starting
point is unhelpful at best. The core documentation isn't awesome either,
given that it has a tendency to be overly cryptic and link to API
documentation that is often incomplete or generally unhelpful.

As one very basic example, see:
http://twistedmatrix.com/documents/current/core/howto/servers.html. Let's
just review a few things wrong with this page:

   - It's tutorial page #1, but basically tells me I need to read
   howto/plugins.html first if I am writing an "application" (whatever that
   is), as opposed to a "TCP, SSL, and Unix socket server". And it's the wrong
   place for UDP.
   - It attempts to introduce Protocols and Factories—two of Twisted's most
   important concepts—and does neither particularly well. I know that Protocols
   (usually) inherit from t.i.p.Protocol and may be instantiated for a variety
   of reasons and aren't (usually) persistent. I also know that Factories
   instantiate Protocols and give a reference to themselves so protocols can
   "access and modify the persistent configuration". I am told I need to
   implement some interface (or something) to actually listen on a host/port. I
   think.
   - At one point "TCP4ServerEndpoint" is instantiated (but never imported);
   its explanation is left to a digression into the endpoints API, which has
   its own issues. Suffice it to say the document doesn't give me sufficient
   reason to actually use the endpoints API.
   - Later on, we just use reactor.listenTCP()—which our previous digression
   (if bothered to click through and read) claims is not preferable.

By the end of the *servers* tutorial (and after reading some linked
documentation), here's all I *really* know:

   1. Factories create protocols somehow
   2. Protocols have methods like connectionMade, connectionLost, and
   dataReceived to handle events
   3. There are other protocols with other methods. One that I know of,
   anyway.
   4. I may need to write a state machine (???)
   5. I should use an Endpoint or maybe a Service or reactor (but probably
   not!)
   6. I should also use Application for serious business

Moving forward, howto/clients.html duplicates a lot of these things and
fills in some gaps in knowledge while creating more holes. Meanwhile, I
still never wanted to create a QOTD or Echo server.

I think the point has been made. My *real* point, though, is that I love
Twisted. And I'm constantly wishing it was more accessible to newcomers.
Twisted is Python's oldest and most mature event-based networking engine and
despite its decade of existence it remains largely confusing and obscure to
the majority of Python programmers who come upon it. It contains concepts
and standards that are alien to the average Python programmer, but they make
a lot of sense and have a lot more consistency and predictability than the
documentation conveys.

I want to fix that, among other things. And as luck may have it, I like
writing documentation. And I know at least enough Twisted to get the
high-level stuff in order and improve the documentation to the point that
people will keep reading long enough to make sense of the "idea of Twisted"
and be able to implement some basic things and expand upon them later.

There are some things I *do* want to accomplish early on:

   - Make the docs accessible (a lot is hidden and hard to find)
   - Make them more concise and useful to somebody who doesn't want to know
   the 50 different ways to skin every cat (including the ones you should never
   use)
   - Make them introduce and explain Twisted in a way that somebody as dumb
   as me can understand it. This means talking about protocols, factories,
   deferreds, etc. in a way that doesn't require thousands of words of circular
   explanations, digressions, and duplications.
   - Document the different Twisted projects as what they are: really useful
   libraries and/or clients/servers built on top of Twisted that just so
   happens to also ship with the core. One of them is likely to do what the
   prospective user wants, too.

There are also some things I *don't* want to accomplish, at least initially:

   - Explain asynchronous programming in depth
   - Get into the level of detail that the Krondo series does (but I plan to
   borrow from it!)

There are things I'll *need help with*:

   - The final word on what is/isn't to be used in what context; practical,
   real-world explanations of why I would use A instead of B (not just "it's
   more flexible").
   - Better documentation from project creators/maintainers/lovers on their
   project. These projects make a lot of hard stuff trivial and many do not
   convey that very well. I'll do my best to reorganize and improve the
   existing documentation, but most of the projects cater to a specific niche
   that requires its own core knowledge.


Most of this was discussed with Jean-Paul and Glyph at tonight's Python
meet-up in Cambridge. Some work has already begun on the new docs here:

https://github.com/tdavis/twisted-docs

I hope the above came off as optimistic and constructive as I meant it: it's
not my intention to insult the documentation or those who wrote it. I just
think with a dose of hindsight and a lot of effort, the docs can be awesome.
And they would improve the entire Twisted ecosystem along with them.


Cheers,

Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110120/b8a47686/attachment.html>