[Twisted-web] web2 log patch

[glyph at divmod.com]

On Fri, 20 Jan 2006 15:46:36 -0800, Ed Suominen <general at eepatents.com> wrote:
>I've expressed my admiration for the Twisted project and the brilliance
>of its developers on numerous occassions.  I like to think that my
>background in patenting things gives me an eye for spotting innovative
>work, and there's a *lot* of innovation going on with Twisted that,
>thankfully, Glyph et al. have essentially given to the public for its
>free use.

Hint to other people with criticism - this is a good way to start a message :)

(Thanks, Ed)

>I hope that background gives me a little room to concur in some of the
>criticism being aired here.  Perhaps we are long overdue for the kind of
>hearing that Andrea is creating with his willingness to speak his mind.

Yes.

I am frequently looking for criticism of various Twisted (and Divmod) projects; especially from competent people with specific goals in mind.  I have been heard to complain about this, actually - useful criticism is hard to find.  I probably hear more baseless hagiography than baseless insults.  It's equally useless :).

A pipe-dream of mine is to get a thorough efficiency review of the Twisted codebase from a really experienced C++ programmer; to have them just totally rip it apart and suggest (or better yet, of course, provide) a reactor implementation that is maximally efficient according to the best known practices for async I/O.  Jason Beardsley, can you hear me? :)

Perhaps we can turn this thread into a productive discussion.

>The Twisted team is great about sticking to good programming practices
>and showing professionalism and maturity in certain cases but comes
>across far differently in many others.  The religious fervor around unit
>tests is a positive example, but the goofy naming conventions (jelly,
>banana, manhole, etc.) that don't impart much if any information to the
>source code reader are a very contrary case in point.

There should *definitely* be more documentation around things like PB - and a bit less silliness in naming perhaps.  However, I do note that the coding standard allows for silliness in module names, but forbids it in class, method, and function names.  A certain amount of silliness is unavoidable.  Corporate people call silliness "branding". What information does "ActiveX" or "Vista" or "Cocoa" impart to the user? but if you have any examples of where the standard has been violated, or you would like to improve the documentation of what the various silly names mean, *please* go ahead.

(Although, probably PB doc bug discussions should go on the other mailing list...)

>Andrea points out
>another example below; there's a lot of just plain crud and useless
>amusements littering up the SVN tree, yet many classes and methods in
>the code are devoid of meaningful docstrings.

Twisted.Quotes is a single file, and I have actually received quite a bit of positive feedback about the file's utility in understanding the IRC culture of the Twisted development team and lubricating social interactions that take place there, especially for people less familiar with IRC.  It's amusing, and perhaps not as useful as code, but it's not "crud" or "useless".

>I once made the mistake of trying to check out SVN/branches -rHEAD and
>found that there are several *gigabytes* of utterly worthless cruft in
>there, the vast majority of which will never, ever be touched.  A huge
>debate on IRC ensued about the wisdom of keeping abandoned stuff in the
>HEAD revision of *any* SVN repository.  You can guess what my viewpoint
>was, and I figured it fit well into the "do everything excellently even
>to the point of pain" vision of the Twisted developers, but I was wrong
>about that.

This, on the other hand, is definitely a useful observation.  Let me try my hand at that productive-conversation thing I mentioned before!

I missed this discussion, and if I hadn't, I would have agreed with you :-).  /branches in the Twisted repository is insane.  I actually removed some branches recently: around r15627.

Not to be too hard on the developers responsible for it - much of the crap in there is auto-generated.  cvs2svn generates branches if you *sneeze*.

Some of it belongs to people who are hard to contact, and currently the rudeness of just deleting it without warning outweighs the benefits of having a smaller checkout of /branches.

Divmod's repository tries hard to keep /branches to a reasonable size: everything at http://divmod.org/svn/Divmod/branches/ is something we're actually working on.  You'll note there is nothing older than a week.  I would like it *very much* if Twisted could get to that point.

>I even saw a bit of this contradictory zeitgeist on the #twisted IRC
>channel recently, when someone was criticized for off topic posts about
>more general Python stuff.  I pointed out that there is a lot of utter
>nonsense on the IRC channel, including a lot of frankly silly,
>repetitive stuff about poking, eating shoulders, stabbing in the eye, etc.

I think I can half-defend this behavior.  The silliness is done by people who are active contributors.  In reality, they're just people who were formerly active contributors or who are friends of contributors, which is why I can only half-defend it.

The silliness is energizing to developers, as are on-topic questions.  However, new people who show up to ask off-topic but technical questions are soaking up the time that #twisted's denizens want to spend either being silly or working on Twisted.  If someone were to pay a Twisted developer to support Python (or some Linux distribution, as those questions are sometimes heard too) I'm sure that you'd find them very prompt and courteous, as well as knowledgeable.

In many cases, this support fatigue is related to half a decade spent in #python, answering the same question hundreds or thousands of times.  There's nothing wrong with simple Python questions, but there is an appropriate forum on IRC already, and there are plenty of people there who can help you.

>Sure, this is all free software and we're all largely in it to have some
>fun, but I think there's some reasonable grounds for not-always positive
>comments from the user base, and that hardly is "harming the community."

The harm to the community comes from the frequent implications of things like "Divmod told me to use Atop, and it ruined my business!".  Mr. Arcangeli hasn't ever stated that directly, of course, but he continues to strongly imply it.  I consider this *personally* very insulting, since Divmod has put a lot of money, time, and energy (argubaly more than was wise) into building a reputation with the open-source community.  That reputation has been immensely valuable, since we have gotten lots of help from tireless contributors like Matt Goodall.  Such a thing can be easily destroyed, and future contributors repelled, by a few misleading comments about the availability of support, or lack thereof, for our code.

The harm comes from continuously repeating the same vague, baseless differences of opinion with Twisted developers.  The fact that Andrea Arcangeli happens to prefer code that is in the library model rather than the framework model is not a serious criticism of Twisted.  Nor is the fact that he doesn't like to close over arguments rather than have them passed to him from the framework.

There are LOTS of problems with Nevow.  To mention a few: livepage.LivePage should be converted into a backward-compatibility wrapper around athena.LivePage.  The flatten() system can get stuck in a dead loop if you do apparently-innocuous things.  Lots of things are undocumented.  Fragments largely obviate the need for the evil 'ctx' and 'data' arguments on every renderer.  These are all problems about which very concrete things can be said.  Here are some more:

<http://divmod.org/trac/query?status=new&status=assigned&status=reopened&groupdesc=1&group=priority&component=Nevow&order=priority>

You might ask, "Why then, glyph, do you bother to respond?  Shouldn't you be fixing those problems?"  I do actually have a reason :-).  Comments like the ones we're talking about are discouraging.  There are lots of developers working on webby Twisted things that are frequently frustrated enough to throw their hands up and go play Crack Attack instead, just because of the volume of *legitimate* problems we have to deal with.  Responding to emphasize that this  rudeness isn't important feedback is important.  It turns a long-winded screed about how worthless all the Nevow team's efforts have been into an amusing flamewar ;-).

Still, I *would* rather be spending my time on more positive ways to keep them energized.