[Twisted-Python] Proposal (almost) to switch from Lore to Sphinx

[Kevin Horn]

On Wed, Nov 11, 2009 at 11:22 PM, Glyph Lefkowitz
<glyph at twistedmatrix.com>wrote:

Excellent.  In that case (modulo several weeks of work on your part), I'm
> sold.
>

Hooray!


> Hmmm...I hadn't considered the Divmod stuff, but you're right.
>
>
> I'm glad you think so.  I do tend to be, especially about Twisted ;-).
>
> I think the automated conversion plan is the way to go...up to a point.  As
> I laid out in the proposal, I think at some point, the Sphinx docs (and the
> automated conversion process) will become "good enough", and then edits
> should be done manually to the ReST/Sphinx source files, rather than the
> Lore files.  Otherwise you just end up with another tool to maintain
> forever.
>
>
> Well, sure.  The goal of this process is to delete all the lore files and
> check in a bunch of sphinx files which replace them.  This should be done in
> a branch, and reviewed like usual; the automated tool is just to make the
> process easier and more reliable.
>
> You have two options regarding the Divmod projects; either you can make the
> tool good enough that you just run it, or you can commit to putting the work
> into the minor edits to correct the tool's deficiencies on the Divmod lore
> docs _before_ you can remove / desupport twisted.lore.
>

I think the second option is probably what will end up happening, though it
may not be necessary to do too much of this.  So far, I've only got 3 files
in the Twisted Lore docs that wil definitely need reworking before they can
be processed.  And they have some really strange markup.  So maybe there
won't be too much of this in the Divmod stuff. (crossing fingers!)


> Another thing to note in the migration plan: despite <
> http://twistedmatrix.com/trac/wiki/CompatibilityPolicy>, no deprecation
> period should be necessary for lore, it can just be removed.  It has no
> users that I'm aware of, and if anyone does still use it they can manually
> copy their 'twisted/lore' directory and 'lore' script into new installations
> until they have a chance to run your script.  It's not designed to be used
> as a library anyway; it's a bit of an accident that it's even present in the
> twisted/* tree.
>
> One thing to note in the migration plan is the closing of outstanding
> branches with lore changes in them.  This would be a good topic for a
> sprint.
>

Yikes.  Another thing I hadn't considered.  Depending on how many branches
there, this could get nasty.  I'll have to do some thinking on this.


> And in my opinion, Twisted needs to get away from having it's own
> collection of self-maintained tools as much as possible. :)
>
> Not just your opinion :).
>

Glad to hear it :).

I have some basic code for the conversion process, but it's in a VERY
preliminary state, and needs a fair bit of work.  There's still several tags
it doesn't handle at all (including tables, which _might_ be a real beast),
and it needs to be refactored some.  I mostly wrote it as a proof-of-concept
for myself to prove (to myself) that automated conversion was feasible.
 That said, it _will_ generate a buildable Sphinx project (though it's
currently VERY ugly, and generates warning in the hundreds).

If there's interest, I can put up what I can generate to this point, though
it's awfully early days yet.

Release early, release often.  Most efforts like this end up moribund: you
> sound serious right now, but never underestimate the extent to which life
> will get in the way.  If you put up what you've got somewhere public ASAP,
> (A) some people will probably show up to egg you on, (B) it's possible for
> them to contribute (and those contributions will be a motivation on days
> when this seems like a ridiculous amount of effort just to make Twisted's
> docs look nicer), and (C) they can pick up the slack if you need to
> disappear, either temporarily or permanently.
>

I'll try to put up what I've got later today.



> *don't* try to get your migration tools to the state where they can get
> through Twisted code review, etc.  Hopefully we can throw these away
> quickly.
>

Roger that.

Also:
> Were my claims/assumption about Lore pretty much correct?  Did I miss
> anything?
>
> Your claim about highlighting only python is correct.  See
> twisted.lore.tree.munge (is it any wonder that we need to replace this
> thing), specifically the bit where it calls fontifyPython and there is no
> definition of fontifyAnythingElse.
>

Noted.



> The "docs errors throw build warnings" thing is an extremely confusing
> header, although I understand what you're saying.  I'd re-word the whole
> thing in terms of test integration.  But the important point has nothing to
> do with doctest, it's that sphinx has various bits of testing support,
> including "leave the examples outside the documentation so they're just
> regular python and don't need to be parsed".
>

I've just removed this whole section, as it _was_ confusing, and also mostly
redundant.



> There's a "*" for "requires plugin" - it's worth mentioning that Sphinx has
> a usable, and more importantly *actually used by some people*, extension
> API, whereas lore's extension mechanisms are... ahem.  Let's not talk about
> lore's extension mechanism.
>

I've added something about this.



> You do need to address linking *from* sphinx documentation *to* API
> documentation, and ideally, vice versa.  Lore can do the former, via
> 'class="API"', although pydoctor still has no way to do the latter (see <
> http://twistedmatrix.com/trac/ticket/2801>).
>

This should be doable.  Right now I'm thinking of doing something similar to
an existing extension, which creates links to generated Epydoc API docs.
Shouldn't be too hard to do something similar for pyDoctor.  I'll get
something in the proposal.



> I don't know of any other projects (besides the ones on divmod.org) which
> currently use lore.  The only other one I'm aware of that ever did is <
> https://launchpad.net/thinkcspy>, and they switched to using sphinx input
> a while ago.  Maybe you could bug Jeff Elkner and see if he did any
> automation which might help?
>
>
> In your description of 'phase 0', you say "once it has been verified": be
> specific here.  Nobody is going to want to be the one who actually pulls a
> trigger on a change this big, but it needs to get done.  Propose a specific
> mechanism for the verification.  Lay out a timeline which culminates in the
> ReST files being committed to trunk.  It's too early for specific dates, but
> you can say "first we will do X, then we will do Y".
>

I'll try to get this updated a bit later.



> In general though, your representation of Lore seems accurate.
>
> Thanks again for taking this on.  Good luck!
>

Thanks.  I have a feeling I'll need it :)

Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20091112/44ed29ee/attachment.html>