[Twisted-Python] Refactoring Documentation

Thu Jan 20 00:55:48 MST 2011

Let me begin by saying that I'm _very_ excited about this prospect and I'm looking forward to the output of this project.  So, some of what I'm about to say may sound like harsh criticism but please do not let it discourage you.  I agree with pretty much everything you're saying about problems with the current documentation.  They need to be fixed.  I'm offering criticism at this early stage in the hopes that it will be useful and feed into your project, not in the hopes that it will stop you.

On Jan 20, 2011, at 1:10 AM, Tom Davis wrote:

> I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the trial-tutorial branch which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly, twistedmatrix.com). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge.

Yet, the problem _starts_ with a branch merge.

There are many outstanding documentation branches which are substantial improvements, which need to be edited and merged - the trial tutorial among them.  It would be great if your efforts could start with getting those landed, turning the crank on the process to get our users better documentation in the interim, as you survey the existing documentation.
> Factories create protocols somehow
> Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events
> There are other protocols with other methods. One that I know of, anyway.
> I may need to write a state machine (???)
> I should use an Endpoint or maybe a Service or reactor (but probably not!)
> I should also use Application for serious business
> Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server.
> 
> I think the point has been made.

This kind of tear-down is pretty easy: I'd really like to hear your idea of its inverse, though.  What should a newcomer who reads this document know by the end of it?  A massive pile of improved documentation would of course be useful, but a good start would be a clear statement of requirements and audience.  (As well as an enumeration of different audiences that different documents might serve.)

> My real point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine

(minor nitpick: I really like "event-based" or "event-driven", as you've said here: why does <http://docs.recursivedream.com/twisted/> say "asynchronous"? I find that especially in documentation it's a lot easier to explain "event-driven", because you can enumerate what the events are, instead of explaining the etymology of "synchronicity"...)

> I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later.
> 
> There are some things I do want to accomplish early on:
> Make the docs accessible (a lot is hidden and hard to find)
> Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use)
> Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications.
> Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too.
These all sound like excellent goals.

> There are also some things I don't want to accomplish, at least initially:
> Explain asynchronous programming in depth
> Get into the level of detail that the Krondo series does (but I plan to borrow from it!)
> There are things I'll need help with:
> The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible").
You will probably have to press us core developers on this one, and you may spark some debates.  These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.
> Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge.
The biggest problem with this is that you will find that a very small group of people have created the vast majority of this stuff and don't have time to maintain it all any more :).  We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).

> Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here:
> 
> https://github.com/tdavis/twisted-docs

There have been several abortive efforts to do something grand to re-invent all of the Twisted documentation in the universe, or a complete overhaul of the website, including several false starts that I've made, and most recently the (somewhat arduous, arguably "mostly complete" (fingers crossed on that one)) attempt to do a sphinx migration.

By the way, it seemed tonight that you weren't that familiar with the existing Sphinx efforts, so you may want to have a look at this and see if it is an improvement on our existing documentation index:

	http://twistedsphinx.funsize.net/contents.html

(I was going to point you at the output of the buildbot, but it looks like something in trunk has broken the conversion process.  Anyone know what this is about? <http://buildbot.twistedmatrix.com/builds/sphinx-html-15724/contents.html>)

I'm really glad to see that Kevin beat me to the punch on posting his reply though, hopefully we'll see this moving along soon :).

Before you continue too much further in that github repository though, please, consider this:

~/Projects/Twisted/trunk/doc$ find conch core mail names pair web words -name '*.xhtml' -or -name '*.py' | xargs wc -l
    .
    .
    .
   28836 total

That's 28 thousand lines of documentation and sample code.  It might not all be the best documentation, but some of it is quite good.  Jean-Paul's "twisted web in 60 seconds" tutorial is in there, and that's definitely worth reading.  Would you volunteer to rewrite 28 thousand lines of code all at once?  Or submit a 28 thousand line patch for review?  As problematic as the current situation is, there is a definite potential for some baby vs. bathwater confusion in improving it.

I am open to the idea that documentation needs different review practices from code, and there are things about documentation (like narrative flow) which are impossible to automatically verify and may require significant changes at multiple layers simultaneously.  These features may not have an obvious parallel in code - which can typically be evolved incrementally, despite the occasional frustrated lament that everything needs to be rewritten.

But, parts of the documentation are code.  Another major problem with the current documentation is that it is out of date.  We know when code goes out of date because it starts raising exceptions and emitting warnings from our test suite, but we only find out about documentation examples are going out of date when we do a release and some poor soul encounters an error when trying to follow along.  I'd really like any effort that undertakes to substantially change the structure of the documentation to make a concerted effort to hit the ground running with test-driven examples which can be automatically verified, so that we have some idea of the impact of code changes on the docs.

Also, while you're energized now, please consider what happens to our documentation efforts if you run out of time or out of motivation halfway through this process.  Incremental improvements may not necessarily provide the best outcome, but they _do_ mean that users get to see some results from these volunteer efforts even if the original volunteer gets busy with other aspects of life and can't complete their project.  Many of your bullet-points above are well suited to individual efforts (some of which are actually issues with the web site, rather than the documentation per se).

I'd like to suggest that, at the very least, you create a branch of Twisted (perhaps on Launchpad?) and integrate your work into the doc/ hierarchy of the current tree.  That way, at least, it would be easier for someone else to pick up where you've left off and fold things into the right place.  This would also provide a way to more easily integrate the doc test suite into the main test suite on our existing build and release infrastructure.

> I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them.

Indeed, as the author of many of these original documents, I did not feel insulted in either my person or my work :).  Hopefully you didn't feel that way either after reading this reply!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110120/90887aa9/attachment.html>