[Twisted-Python] Refactoring Documentation

[Kevin Horn]

On Thu, Jan 20, 2011 at 12:10 AM, Tom Davis <tom at recursivedream.com> wrote:

> I've been using (and threatening to work on) Twisted for a few years now.
> It seems like every time I get back into it, I need to dig up old code or
> Google queries just to get started. Yesterday, Jean-Paul introduced me to
> the trial-tutorial branch<http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443-2> which
> has shed some light on the basics of using trial and testing Twisted
> client/server applications in general. Until he mentioned it in IRC, I was
> stuck looking at the actual tests for protocols and deciding which parts of
> that were generically useful to me. I agreed to finish up that documentation
> so that it could finally (four years later) be added to trunk (and more
> importantly, twistedmatrix.com). But after thinking about it, I believe
> the problem runs much deeper than just the lack of a branch merge.
>
> Reading code to find answers isn't rocket science; I've been programming
> long enough to be comfortable doing it. But I probably have to resort to
> reading Twisted's code about 8 billion percent more often than any other
> codebase. And reading code is a hurdle. Reading through Twisted's
> semi-random, 45-point FAQ is a hurdle—and recommending it as a starting
> point is unhelpful at best. The core documentation isn't awesome either,
> given that it has a tendency to be overly cryptic and link to API
> documentation that is often incomplete or generally unhelpful.
>
> As one very basic example, see:
> http://twistedmatrix.com/documents/current/core/howto/servers.html. Let's
> just review a few things wrong with this page:
>
>    - It's tutorial page #1, but basically tells me I need to read
>    howto/plugins.html first if I am writing an "application" (whatever that
>    is), as opposed to a "TCP, SSL, and Unix socket server". And it's the wrong
>    place for UDP.
>    - It attempts to introduce Protocols and Factories—two of Twisted's
>    most important concepts—and does neither particularly well. I know that
>    Protocols (usually) inherit from t.i.p.Protocol and may be instantiated for
>    a variety of reasons and aren't (usually) persistent. I also know that
>    Factories instantiate Protocols and give a reference to themselves so
>    protocols can "access and modify the persistent configuration". I am told I
>    need to implement some interface (or something) to actually listen on a
>    host/port. I think.
>    - At one point "TCP4ServerEndpoint" is instantiated (but never
>    imported); its explanation is left to a digression into the endpoints API,
>    which has its own issues. Suffice it to say the document doesn't give me
>    sufficient reason to actually use the endpoints API.
>    - Later on, we just use reactor.listenTCP()—which our previous
>    digression (if bothered to click through and read) claims is not preferable.
>
> By the end of the *servers* tutorial (and after reading some linked
> documentation), here's all I *really* know:
>
>    1. Factories create protocols somehow
>    2. Protocols have methods like connectionMade, connectionLost, and
>    dataReceived to handle events
>    3. There are other protocols with other methods. One that I know of,
>    anyway.
>    4. I may need to write a state machine (???)
>    5. I should use an Endpoint or maybe a Service or reactor (but probably
>    not!)
>    6. I should also use Application for serious business
>
> Moving forward, howto/clients.html duplicates a lot of these things and
> fills in some gaps in knowledge while creating more holes. Meanwhile, I
> still never wanted to create a QOTD or Echo server.
>
> I think the point has been made. My *real* point, though, is that I love
> Twisted. And I'm constantly wishing it was more accessible to newcomers.
> Twisted is Python's oldest and most mature event-based networking engine and
> despite its decade of existence it remains largely confusing and obscure to
> the majority of Python programmers who come upon it. It contains concepts
> and standards that are alien to the average Python programmer, but they make
> a lot of sense and have a lot more consistency and predictability than the
> documentation conveys.
>
> I want to fix that, among other things. And as luck may have it, I like
> writing documentation. And I know at least enough Twisted to get the
> high-level stuff in order and improve the documentation to the point that
> people will keep reading long enough to make sense of the "idea of Twisted"
> and be able to implement some basic things and expand upon them later.
>
> There are some things I *do* want to accomplish early on:
>
>    - Make the docs accessible (a lot is hidden and hard to find)
>    - Make them more concise and useful to somebody who doesn't want to
>    know the 50 different ways to skin every cat (including the ones you should
>    never use)
>    - Make them introduce and explain Twisted in a way that somebody as
>    dumb as me can understand it. This means talking about protocols, factories,
>    deferreds, etc. in a way that doesn't require thousands of words of circular
>    explanations, digressions, and duplications.
>    - Document the different Twisted projects as what they are: really
>    useful libraries and/or clients/servers built on top of Twisted that just so
>    happens to also ship with the core. One of them is likely to do what the
>    prospective user wants, too.
>
> There are also some things I *don't* want to accomplish, at least
> initially:
>
>    - Explain asynchronous programming in depth
>    - Get into the level of detail that the Krondo series does (but I plan
>    to borrow from it!)
>
> There are things I'll *need help with*:
>
>    - The final word on what is/isn't to be used in what context;
>    practical, real-world explanations of why I would use A instead of B (not
>    just "it's more flexible").
>    - Better documentation from project creators/maintainers/lovers on
>    their project. These projects make a lot of hard stuff trivial and many do
>    not convey that very well. I'll do my best to reorganize and improve the
>    existing documentation, but most of the projects cater to a specific niche
>    that requires its own core knowledge.
>
>
> Most of this was discussed with Jean-Paul and Glyph at tonight's Python
> meet-up in Cambridge. Some work has already begun on the new docs here:
>
> https://github.com/tdavis/twisted-docs
>
> I hope the above came off as optimistic and constructive as I meant it:
> it's not my intention to insult the documentation or those who wrote it. I
> just think with a dose of hindsight and a lot of effort, the docs can be
> awesome. And they would improve the entire Twisted ecosystem along with
> them.
>
>
> Cheers,
>
> Tom
>
>
Tom,

I very much like this idea, and in fact I have some ideas for this, but I've
been putting off doing anything about it until the (neraly done! really!)
conversion of the current documentation to Sphinx is complete (see ticket
#4500).  I got off track with that the last part of last year, but one of my
New Year's Resolutions is to complete that project.

How can we combine these efforts, or at least keep from working at cross
purposes?

I see from your link above that you are building your own Sphinx project.
Perhaps you would be better off working from the results of the Lore2sphinx
conversion?  Are you modifying existing docs or working from scratch?

Let's get together on this!

Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110120/7f65e0d4/attachment.html>