[Twisted-Python] Refactoring Documentation

Thu Jan 20 21:20:38 MST 2011

Wow! Okay, so, not being really familiar with the list etiquette combined
with the fact that the topic has digressed a bit, I hope you'll forgive the
non-inline reply here. I want to hit the main points of what everybody said
/ asked, but let it be known that I read all of it!

How can we combine these efforts, or at least keep from working at cross
> purposes?
>
> I see from your link above that you are building your own Sphinx project.
> Perhaps you would be better off working from the results of the Lore2sphinx
> conversion?  Are you modifying existing docs or working from scratch?
>
> Let's get together on this!

Hey Kevin! Glyph and Jean-Paul got me up to speed (somewhat) on the whole
lore2sphinx thing last night—though they were rather unclear as to how close
it is to completion. (which you addressed later in this thread)

I certainly want to combine our efforts as much as possible and I'm not too
worried about working at cross purposes; your main task seems to be
converting the existing documentation while mine is reorganizing, editing,
and standardizing it. I'm sure our paths will start to cross when the lore
conversion is complete and you have a huge Sphinx project ready to be
properly organized and catalogued. My hope is we can then begin the process
of merging the two sides of this coin.

My current plan (which should become a bit more clear throughout this
massive reply) is to start from scratch *but* modify existing documentation
wherever possible. If a piece of documentation fits in a "hole" in the
structure and I can get by with a *mv* and some editing, that's what I'd
like to do—it seems the most pragmatic approach.

So here's what I'm kind of thinking as far as combining efforts:
> 1) I'll continue with the "Project Documentation" conversion, while Tom
> works on the other bits.  Should be fairly easy to combine them, I'm
> thinking.
> 2) Let's leave the "Project Documentation" pretty much as-is for the
> moment, until the Sphinx convo is done.
> 3) I wonder if at least some of the "task-based docs" shouldn't be put into
> the project sections, and then just linked to from the task-based section?

This. Point of fact, (3) is already implemented; that example task simply
links to *projects/web/tasks/serve*. This strikes me as the most logical
layout as the tasks are all specific to a particular project (the only iffy
one being Core which is sort of a few projects plus the foundation)—we just
want to highlight some of them.

On to Glyph's reply...

There are many outstanding documentation branches which are substantial
> improvements, which need to be edited and merged - the trial tutorial among
> them.  It would be great if your efforts could start with getting those
> landed, turning the crank on the process to get our users better
> documentation in the interim, as you survey the existing documentation.

I definitely agree that resolving the low-hanging fruit first is a good
idea. For finishing "docs branch X" to make sense, my personal belief is
that X should:

   - Still be relevant in terms of best practices and simply what's
   available
   - If project documentation, have outstanding issues that a passing
   familiarity with the project [right now] will be sufficient to close them (I
   could spend time learning one project, or the same time improving *all* the
   documentation)
   - Adhere to whatever documentation standards we agree upon, if much is to
   left to do.

I guess my overall opinion here is that, yes, if relatively minor edits can
bring a branch close enough to completion that we can get it out there to
help newcomers *now*, let's do that. If a branch is more of a draft and
requires a good deal of fleshing out (or is simply stale), it's probably
worth nailing down the structure and previously mentioned docs standards
before I just create more work for myself (or others) down the road.

Finally, my biggest hurdle right now is not knowing how to *find* said
branches. I don't see "documentation" as a category in Trac and common
keyword searches didn't show up much for me. I'm sure this is an easy
question to answer, though.

What *should* a newcomer who reads this document know by the end of it?

I'm not sure because I can't see how a practical guide to creating something
so generic really fits in the grand scheme of things. I think if you want to
create a TCP server from scratch you must first create the Universe! In this
case, that means learning how Twisted addresses the *concept* of a server
before ever bothering to write one so generic. My general beef is that many
documents seem to make an attempt to appeal to everybody and in doing so
don't sufficiently help anybody. Maybe I can justify that claim better with
examples of "better" (at least more targeted) documents.

A massive pile of improved documentation would of course be useful, but a
> good start would be a clear statement of requirements and audience.  (As
> well as an enumeration of*different* audiences that different documents
> might serve.)

I completely agree. The updates I made last night (hopefully) address the
issue of different audiences and needs, at least as a beginning stage. What
Jason said about the Python docs is a great parallel: Tutorials, [Project]
Reference, [Core Concepts] Reference. All of these are important and cater
to a different need. I think we're quite on the same page here, though.

(minor nitpick: I really like "event-based" or "event-driven", as you've
> said here: why does <http://docs.recursivedream.com/twisted/> say
> "asynchronous"?

It's inconsistent and I completely agree (and the consensus seems to be)
that event-* is superior.

You will probably have to press us core developers on this one, and you may
> spark some debates. These tend to sputter out with no clear resolution as
> everyone is frustrated that nobody's solution (not even their own) is ideal,
> but you would be doing us all a great service if you really forced us to
> develop a consensus about certain things (like "what's the best way to build
> a twisted command-line program", for example) and agree to agree on the
> current documented "best practice" for those things.

Debates are great! And I fully intend to bitch about the lack of "best ways"
to do certain things—having one clear way to accomplish something is, after
all, a core tenant of Python ;) -- more than that, it's really annoying to
everyone involved (readers, writers, and developers) when there are many
different ways to do something and none of them have very clear use cases or
strengths / weaknesses. Definitely a TODO.

The biggest problem with this is that you will find that a very small group
> of people have created the vast majority of this stuff and don't have time
> to maintain it all any more :).  We certainly don't have a separate
> dedicated maintainer for each project (although I really wish we could get
> to that point).

Kevin touched on this already, but I really think if we make maintaining and
growing a Project something that is both honorable *and* accessible, more
people will want to do it. If you like Python and DNS or SMTP or whatever
the hell else, what has the potential to be a more awesome implementation
than its Twisted one? Let's help people find out for themselves that the
answer is *nothing*. Then they'll want it to be *their* project.

As problematic as the current situation is, there is a definite potential
> for some baby vs. bathwater confusion in improving it... Also, while
> you're energized now, please consider what happens to our documentation
> efforts if you run out of time or out of motivation halfway through this
> process.  Incremental improvements may not necessarily provide the best
> outcome, but they _do_ mean that users get to see some results from these
> volunteer efforts even if the original volunteer gets busy with other
> aspects of life and can't complete their project.

I have no intention of rewriting everything, even if it sounds absurdly
entertaining to make myself learn and write practical examples of every
Twisted project. I am (just) slightly more realistic than that. Having the
Sphinx conversion complete will be a huge help, if for no other reason than
the documentation will be easy to port, edit, and find in the first place.

On the subject of loss of motivation or time, sure. It happens to all of us.
And it's not my intention to keep the repository on GitHub completely
separate and wait until it's perfect to get it to users. Once all the
documentation is under Sphinx, there will be no reason for that. The
existing documentation can be sliced and diced into the newly-minted
structure and then combed over piece by piece. Some of it will likely need
to be thrown out wholesale, but I imagine much of it will live on with minor
edits (and probably new code) at new URLs.

As for Launchpad, etc.—I want to do whatever makes you guys (a) most
comfortable and (b) most likely to contribute. I chose to start the GitHub
project because I know Git well and it allowed me to spend all my time
writing and none of it figuring out something foreign. I don't understand
the link between Twisted Trac and Twisted on Launchpad, so an explanation of
that would be awesome. Provided that, though, how complicated could Bazaar
possibly be? We can leave Twisted's migration to GitHub as task #2 on my
list ;)

I'd really like any effort that undertakes to substantially change the
> structure of the documentation to make a concerted effort to hit the ground
> running with test-driven examples which can be automatically verified, so
> that we have some idea of the impact of code changes on the docs.

Most definitely. Documentation requires the same maintenance that code does
and can benefit from the same automated verification; and it's trivial to
write entire programs and show code samples using *literalinclude* and the *
:lines:* option. Failing code can be fixed and grep'd for realignment in the
documentation.

Indeed, as the author of many of these original documents, I did not feel
> insulted in either my person or my work :).  Hopefully you didn't feel that
> way either after reading this reply!

Not at all!

* * *

I  believe I covered the main thoughts / concerns / queries related to my
original post. My apologies if I missed anybody! And it's great to finally
be working with you :)

Cheers,

Tom

On Thu, Jan 20, 2011 at 8:20 PM, Tim Allen <screwtape at froup.com> wrote:

> On Thu, Jan 20, 2011 at 06:34:17PM -0600, Kevin Horn wrote:
> > On Thu, Jan 20, 2011 at 5:57 PM, Tim Allen <screwtape at froup.com> wrote:
> > > You mean these DTDs?
> > >
> > >    twisted/lore/xhtml1-strict.dtd
> > >    twisted/lore/xhtml1-transitional.dtd
> > >
> > > They reference the xhtml-*.ent entity definitions which are also in the
> > > same directory. It would be neat if lore2sphinx could be taught to use
> > > the DTDs packaged with lore instead of having to download them from the
> > > Internet every time.
> >
> > Huh. Never even knew that was there.  It probably could, and the reason
> it
> > downloads from the internet was because that's the default way of doing
> it
> > in lxml.  I've since figured out how to override that behavior (which is
> how
> > the caching works) so maybe that wouldn't even be hard.  The
> easiest/fastest
> > fix for the moment though would probably be to pre-populate the cache as
> I
> > mentioned before, since IIRC, this would just involve adding the file to
> my
> > hg repo.  I'll have to look into it though, it may be just as easy to do
> it
> > the other way, though I don't want to depend necessarily on having
> Twisted's
> > code available (remember, this is supposed to work on the various divmod
> > projects, and anything else that uses Lore, too).
>
> Well, you wouldn't necessarily be depending on Twisted, just depending
> on Lore - and I imagine any not-Twisted project whose documentation
> depends on Lore has already made peace with the idea of depending on
> Lore. :)
>
> If it's easier to just copy these well-known DTDs into the lore2sphinx
> repository, I guess that's a good plan too - it's not like the W3C is
> going to suddenly issue updated XHTML DTDs.
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110120/e33c4d68/attachment.html>