[Twisted-Python] Proposal (almost) to switch from Lore to Sphinx

[Kevin Horn]

On Tue, Nov 17, 2009 at 2:13 AM, Glyph Lefkowitz <glyph at twistedmatrix.com>wrote:

>
> On Nov 16, 2009, at 5:37 PM, Kevin Horn wrote:
>
> On Mon, Nov 16, 2009 at 3:11 PM, <exarkun at twistedmatrix.com> wrote:
>
>> On 06:09 pm, kevin.horn at gmail.com wrote:
>> >>
>> >er, sorry, forgot to post the link...
>> >
>> >here it is:
>> >http://twistedsphinx.funsize.net/
>>
>> Thanks.  It's good to be able to see (at least an early draft of) what
>> the outcome of this proposal will be.
>>
>> I'm glad the reactions have been good so far, but I want to be a wet
>> blanket for a moment and suggest that we try to avoid being influenced
>> excessively by the "new look" effect.
>
>
> Wet blanket away!  Every project needs a good "devil's advocate".
>
>
> Well!  If it's time for criticism I've got some ;-).
>
> I'd really like to see the stylesheets on this site adjusted to look as
> much as possible like the current Twisted style.  Use the verbatim
> stylesheets from twistedmatrix.com as much as possible.  I realize you'll
> have to have some custom elements, but it should be possible to at least
> avoid creating any new images and to stitch together the existing styles
> rather than make new ones wholesale.
>
> If we ever have the resources to do a site redesign again (and I hope we
> do, it would be nice to get a fresh look every few years) I want to be able
> to re-style the _entire_ site.
>
> We could do this with the Lore documentation too, but if we're going to
> make the investment I'd rather it be on something easier to maintain, as
> your conversion project promises to be.  I also suspect that it would be a
> lot easier to style the output from sphinx, as it is explicitly designed to
> be themable, and has actually been modified to have multiple, radically
> different stylesheets already.  Actually integrating this with the L&F of
> the rest of the Twisted site would go a long way towards demonstrating
> Sphinx's superiority in this area :).
>

The current look is the "sphinxdoc" theme, that comes with Sphinx.  My
intention is to create a new one for the Twisted docs that will be very
similar to the Twisted site.  I haven't really even started on this yet,
though.


> On the "things to do" side, while the mere presence of this page is itself
> unintentionally hilarious - <
> http://twistedsphinx.funsize.net/projects/lore/howto/lore.html> - and it
> will likely be removed from the final output, it does contain a good
> enumeration of all the things that need to be converted in one place.  I
> suspect that if you made all those UNHANDLED_TAG output warts go away, we'd
> be most of the way towards converting all of the documentation.
>

Heh, that is pretty funny.  If you'll notice though, almost all the
UNHANDLED_TAG stuff (on that page anyway) is related to the table on that
page.  I haven't even touched tables yet.  I don't think they'll be too
difficult, as long as there aren't any nested tables (there aren't in the
Twisted docs...I haven't looked at the Divmod stuff yet).  Since I think
there's only 2 tables in the Twisted docs, there kind of low on the priority
list.

FYI, those UNHANDLED_TAG messages are intentionally ugly, to make it easier
to find things that haven't been handled yet.

As far as what still needs to be done on conversion, this falls into a
couple of main areas:

- handling unhandled tags
- fixing some whitespace handling issues (notice that nested lists are
totally borked at the moment)
- going back through the "Using Lore" document, making sure all the various
tags are being handled correctly, and fixing any     problems
- building a "twisteddocs" theme

After this stuff is done, then the real grunt-work begins...fixing the bits
manually that can't be easily converted.  So far, I think this only really
involves 3 pages of Lore XHTML, but I'm sure I'll find other problems. :)

> The next "tier" of advantages (again IMO) also have nothing to do with the
> output, those advantages being:
>
> - a more approachable document format (in the sense that I think there are
> probably more people familiar with the Sphinx dialect of RestructuredText
> than are familiar with Lore's subset of XHTML...and the number of Sphinx
> users is growing)
>
>
> Actually, I think that this particular often-touted advantage is a wash.
>  The "approachableness" of ReST is questionable, especially once you get
> into the weird corner cases where the syntax just completely falls apart and
> turns into an incomprehensible line-noise jumble.  I mean, the syntax for
> tables reads like a joke about how trying to make the plain-text input look
> like the output is a bad idea: <
> http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#tables>.
>  Consider trying to re-flow the contents of a table cell after adding text
> to it, for example.  Plus, a _lot_ of people know HTML -- still many more
> than know ReST, I would argue -- and lore adds relatively little to HTML.
>

I'm not sure I agree here.  While it's true that lots of people know HTML,
I'm not sure that that helps more than it hurts.  First of all, XHTML is
subtly different from HTML, but more importantly, everyone  who "knows" HTML
doesn't know it to the same level, and has different opinions about what
different elements should be used for.  The "Using Lore" page spells things
out pretty clearly, but even some of those clear rules aren't followed very
well in the actual Twisted docs.  I think because people "know" HTML, they
just use what they know instead of paying close attention to the actual Lore
rules.  (going way out on the opinion limb here, though)

I guess my point is that I think it takes about the same amount of effort to
really learn the Lore subset of XHTML as it does to learn Sphinx's dialect
of RestructuredText.  And more people know Sphinx-RestructuredText to the
level practically usable for documentation (from using it on other
projects).  There's arguments either way I guess.

On a personal level, having written docs in other XML/XHTML dialects and in
Sphinx/ReST, I know I can write them way faster and with much less thought
to format in ReST.  And, IMO, those corner cases are very much in the
corner.  They don't come up very often, at least for me.

There's no defending ReST tables though.  You're totally right there.  Yuck.


> However, the motivation for choosing HTML was that "any day now" there
> would be a good, commonly used, GUI editor for HTML documents and we could
> easily annotate the output of one of those with the extra metadata that lore
> wanted.  That hasn't happened.  What *has* happened is that despite the
> difficulty involved in parsing and emitting ReST as compared to something
> fairly regular like HTML or XML, tools like these have been emerging:
>
>     http://kib2.free.fr/reSTinPeace/
>     http://blog.enthought.com/?p=127
>
> which let users edit ReST documents in a sort-of-wsyiwyg way, so you get
> immediate feedback when you put a "|" or a "`" in some obscure spot that
> makes your entire document disappear, rather than trying to hunt it down
> after an hour of writing.  Plus, docutils provides a document model for ReST
> so you can transform it programmatically and ignore its syntactic
> peculiarities.
>

I think the underlying reason this has happened is that ReST is less
flexible than HTML, and therefore less ambiguous.  I couldn't tell you why I
think that, though.  Just a feeling.


> Which all comes down to the fact that I'm mainly just restating this point
> as the primary advantage:
>
> - a much larger user base, and clear extension API, which allows us to take
> advantage of existing extensions, etc.
>
>
> The active user community means that not only is somebody else mantaining
> the basics, lots of other people are working on *tools* that we can use for
> more intense usage.  I am personally looking forward to using some of those
> GUI tools for actually editing the documentation; it's a much more
> streamlined workflow than running the 'lore' commandline and reloading in a
> browser.
>

Kevin Horn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20091117/5a4e7e14/attachment.html>