[Twisted-Python] Proposal (almost) to switch from Lore to Sphinx

[Glyph Lefkowitz]

On Nov 11, 2009, at 7:37 PM, Kevin Horn wrote:

> On Wed, Nov 11, 2009 at 5:56 PM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> On Nov 11, 2009, at 5:52 PM, Kevin Horn wrote:
> 
> > Please let me know what you think.
> 
> Initial reaction: awesome!
> 
> Well, that's encouraging. :)

Please!  Be encouraged!  And therefore maintain our documentation and documentation toolchain! :)

> There's lot's of other work being done in the Sphinx testing arena, though, notably: ...

It's great to see that the testing vibe is out there!

> Can sphinx do that out of the box?
> 
> Yes.  The "literalinclude" directive pulls in an external file and sticks it in a literal block (the ReST equivalent of a <pre> tag).  So you could use whatever testing framework you wanted and just reference those files.

Excellent.  In that case (modulo several weeks of work on your part), I'm sold.

> Hmmm...I hadn't considered the Divmod stuff, but you're right. 

I'm glad you think so.  I do tend to be, especially about Twisted ;-).

> I think the automated conversion plan is the way to go...up to a point.  As I laid out in the proposal, I think at some point, the Sphinx docs (and the automated conversion process) will become "good enough", and then edits should be done manually to the ReST/Sphinx source files, rather than the Lore files.  Otherwise you just end up with another tool to maintain forever.

Well, sure.  The goal of this process is to delete all the lore files and check in a bunch of sphinx files which replace them.  This should be done in a branch, and reviewed like usual; the automated tool is just to make the process easier and more reliable.

You have two options regarding the Divmod projects; either you can make the tool good enough that you just run it, or you can commit to putting the work into the minor edits to correct the tool's deficiencies on the Divmod lore docs _before_ you can remove / desupport twisted.lore.

Another thing to note in the migration plan: despite <http://twistedmatrix.com/trac/wiki/CompatibilityPolicy>, no deprecation period should be necessary for lore, it can just be removed.  It has no users that I'm aware of, and if anyone does still use it they can manually copy their 'twisted/lore' directory and 'lore' script into new installations until they have a chance to run your script.  It's not designed to be used as a library anyway; it's a bit of an accident that it's even present in the twisted/* tree.

One thing to note in the mig	ration plan is the closing of outstanding branches with lore changes in them.  This would be a good topic for a sprint.

> And in my opinion, Twisted needs to get away from having it's own collection of self-maintained tools as much as possible. :)

Not just your opinion :).

> I have some basic code for the conversion process, but it's in a VERY preliminary state, and needs a fair bit of work.  There's still several tags it doesn't handle at all (including tables, which _might_ be a real beast), and it needs to be refactored some.  I mostly wrote it as a proof-of-concept for myself to prove (to myself) that automated conversion was feasible.  That said, it _will_ generate a buildable Sphinx project (though it's currently VERY ugly, and generates warning in the hundreds).
> 
> If there's interest, I can put up what I can generate to this point, though it's awfully early days yet.

Release early, release often.  Most efforts like this end up moribund: you sound serious right now, but never underestimate the extent to which life will get in the way.  If you put up what you've got somewhere public ASAP, (A) some people will probably show up to egg you on, (B) it's possible for them to contribute (and those contributions will be a motivation on days when this seems like a ridiculous amount of effort just to make Twisted's docs look nicer), and (C) they can pick up the slack if you need to disappear, either temporarily or permanently.

*don't* try to get your migration tools to the state where they can get through Twisted code review, etc.  Hopefully we can throw these away quickly.

> Also:
> Were my claims/assumption about Lore pretty much correct?  Did I miss anything?

Your claim about highlighting only python is correct.  See twisted.lore.tree.munge (is it any wonder that we need to replace this thing), specifically the bit where it calls fontifyPython and there is no definition of fontifyAnythingElse.

The "docs errors throw build warnings" thing is an extremely confusing header, although I understand what you're saying.  I'd re-word the whole thing in terms of test integration.  But the important point has nothing to do with doctest, it's that sphinx has various bits of testing support, including "leave the examples outside the documentation so they're just regular python and don't need to be parsed".

There's a "*" for "requires plugin" - it's worth mentioning that Sphinx has a usable, and more importantly *actually used by some people*, extension API, whereas lore's extension mechanisms are... ahem.  Let's not talk about lore's extension mechanism.

You do need to address linking *from* sphinx documentation *to* API documentation, and ideally, vice versa.  Lore can do the former, via 'class="API"', although pydoctor still has no way to do the latter (see <http://twistedmatrix.com/trac/ticket/2801>).

I don't know of any other projects (besides the ones on divmod.org) which currently use lore.  The only other one I'm aware of that ever did is <https://launchpad.net/thinkcspy>, and they switched to using sphinx input a while ago.  Maybe you could bug Jeff Elkner and see if he did any automation which might help?

You're correct that lore cannot include partial external files.

In your description of 'phase 0', you say "once it has been verified": be specific here.  Nobody is going to want to be the one who actually pulls a trigger on a change this big, but it needs to get done.  Propose a specific mechanism for the verification.  Lay out a timeline which culminates in the ReST files being committed to trunk.  It's too early for specific dates, but you can say "first we will do X, then we will do Y".

In general though, your representation of Lore seems accurate.

Thanks again for taking this on.  Good luck!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20091112/8d1a1f94/attachment.html>