[Twisted-Python] Lore to Sphinx Conversion Progress Report 6

[Glyph Lefkowitz]

On Jul 10, 2010, at 4:17 PM, Kevin Horn wrote:

>  Some extra thoughts:
>    - The ReviewingDocumentation wiki page has a section called "Editing
>      man pages" that describes how to turn the nicely-formatted
>      manpages into Lore input files. Would it be possible to do that as
>      part of the lore2sphinx run, have the manpages included in the
>      Sphinx documentation, and from then on generate the manpages from
>      the .rst files instead of the other way around?
> 
> Sphinx does have a man page builder now, but I don't think it existed when I was writing lore2sphinx, so I haven't really considered this.
> 
> So you're suggesting convert the man pages to Lore format -> use lore2sphinx to convert the resulting Lore docs to rst -> build as part of the Sphinx process, yes?

I think there may be some confusion here.  However, the current situation is extremely confused, so I will try to shed some light...

First, the goal: we should include the content of the man pages that you get from a UNIX command prompt, but formatted in a manner consistent with the rest of our documentation.  As Tim noted in another message:

> At least for trial, there's a bunch of stuff that's *only* in the
> man page and not the online docs, and a bunch of stuff that's *only* in
> the online docs and not the man page (and stuff that's *only* in the
> core/development/policy section of the docs, and not in the Trial
> section...). Hopefully if they're all part of the same doc system, it'll
> be easier to have everything in the one place and easy to find.


It would definitely be better to have this information more centrally located and maintained.

This is a good goal, and something we are nominally already supposed to be doing.  (And in the process of writing this up, I discovered, oops, we're not, but more on that later.)

Lore already has a plugin which attempts to convert man pages (groff input) into HTML.  To be clear: it creates lore *input* documents, which are intended to be an intermediate state; they're supposed to be re-processed in the same way as the rest of the lore input, to produce lore output.

Lest it seem like editing groff directly is a thing only an insane person would want to do (and arguably, Moshe Zadka, the author of this feature, is in fact an insane person (we love you Moshe)), he designed it this way for a good reason.  The thinking at the time among several Twisted developers was that it would be more useful to go from groff (which has a fairly limited vocabulary) to HTML (which is pretty expressive) than the other way around, since you'd have to give up if you ran into some HTML nuance that couldn't be expressed in groff.

This is slightly good news for including the manpages in the current process being tested on the buildbot, because it just means you have to add one additional command-line to generate the input, and the .rst output can be generated along the same lines as what is already being done.  At least, this is all you need to do to get HTML that probably looks reasonable.  Getting manpages out is another matter.

Unfortunately there's no 'man2rst', so we can't keep man pages around as input for sphinx.  So, in order to continue with the same workflow we've had - with groff as source - we'd have to do several undesirable things:

Produce a convoluted document conversion process where we start with groff input, go through HTML, then end up with rst, then some more, different HTML generated from that.  One goal of this process would be for the document generation pipeline to be a simple, single command (I am hoping something like 'sphinx doc' and that's it...).
Keep maintaining lore2sphinx.  This was an explicit non-goal of the migration process, given that there are going to be some manual fixes required.  (Despite my suggestion that I think we should have fewer of these, I don't think that maintaining it long-term is a good idea.)
Keep maintaining lore, since that's where the manpage conversion code lives.  Since the whole point of this process is so that we can dump lore out of the nearest airlock, this strikes me as a dealbreaker by itself.

There is, however, an 'rst2man'. <http://docutils.sourceforge.net/sandbox/manpage-writer/rst2man.txt>, which appears to be the basis for sphinx's manpage generator.  And this is where we encounter the problem where it's somewhat difficult to get manpages as output: it seems to have exactly the issues that the lore groff-input thingy was designed to prevent.  It just chokes and dies on groff-unfriendly markup, and it requires several annotations which can't be provided in the HTML input.  That means the output generated from the groff->lore->rst pipeline will be missing a bunch of stuff that rst2man wants, such as ":Manual section:" and ":Manual group:" fields.  Which, in turn, means that this task will require a distinct set of manual (no pun intended) post-processing that the rest of the documents won't.

Of course, these limitations and the extra work might be worth living with for the reasons listed at the beginning of this message.

But there's good(-ish) news.  We don't currently generate readable HTML from manpages anywhere.  So if we don't have it in the next release, it won't be a regression.

The code used by the current release process, here - <http://twistedmatrix.com/trac/wiki/ReleaseProcess#Buildhowtodocumentsforwebsite> - has a bug in it, where it generates the lore input, but then uses a dictionary to avoid processing those generated input documents.  Nobody ever sees this half-generated input, because it's only available here <http://twistedmatrix.com/documents/10.1.0/core/man/>, a secret index to which I can't find a link anywhere in the browsable bits of the lore documentation!

Anyway, I apologize for the extremely long-winded enumeration, because ultimately it's just to say I agree with this:

> I think this is a worthwhile idea, but I'd prefer to leave it until after the main docs are converted (i.e. under a separate ticket).  lore2sphinx can be used on just the man files later on if need be, though it would take a little mucking around.

I shouldn't let all the effort of composing this message go to waste though, so I'll be more specific.

We should use lore and then lore2sphinx to convert these documents to rst, in a one-shot process similar to the one we're using to convert the rest of the documentation, manually add in the information required by 'rst2man', and add 'rst2man' as part of our build process.  Of course, part of this task will be making sure that the documentation buildbot turns red if we have any rst in the desginated 'man' documents that doesn't get turned into valid groff.  And we'll have to keep lore in the tree until this is done, because otherwise it'll be a pain to get the code installed for the man-page-conversion pipeline.

Again, since we're lacking it now, I don't think this is a high-priority task.  I would be happy to have even a couple of releases with no pretense at HTML manpages, until we can do this separate conversion.

Until then, I encourage our users to please enjoy these fine links from die.net: <http://linux.die.net/man/1/twistd>, <http://linux.die.net/man/1/trial>, <http://linux.die.net/man/1/conch>, <http://linux.die.net/man/1/cftp>, <http://linux.die.net/man/1/lore>.

(Since this is so low-priority, It's too bad I just spent over an hour writing an email about it.  Oh well.)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20100710/2734a4fe/attachment.html>