[Twisted-Python] documentation / kqueue / feedback

[glyph at divmod.com]

On 01:15 pm, dr.pythoniac at googlemail.com wrote:
>Seeing there is doc. available, I digged through it. After all TM seems
>worth the effort. From what I see, I love TM.

Great!  The rest definitely doesn't sound like you love it though ... 
;-)
>Links are often broken, though. Often seemingly useful doc. is old, 
>very
>old. Even current doc. isn't up to date.

Since the 8.0 release, we have had quite a few problems with links, 
specifically the links from the core documentation to the API 
documentation on the website.  We are working to fix this.  However, if 
you spot broken links, please feel free to report them in the bug 
tracker: http://twistedmatrix.com/trac/newticket

Saying that there are "often" broken links is just about as pointless as 
saying that you "often" find bugs.  Without specifics, this is not 
useful information; the reason we have not fixed specific problems is 
not because we believe there are no problems anywhere.
>They talk about "resource trees", yet I don't find them in the doc. I 
>find
>putchild() but all examples and docs indicate, that you can do it only
>within the root resource.

You can do putChild to any resource, as long as you understand its 
lifecycle.  A "resource tree" is simply a tree ( 
http://en.wikipedia.org/wiki/Tree_structure ) of resources ( 
http://twistedmatrix.com/documents/current/api/twisted.web.resource.IResource.html 
).

Our documentation assumes, and will continue to assume, a certain level 
of familiarity with basic computer-sciency jargon; if you feel that a 
particular phrase is too obscure, feel free to submit a patch 
elaborating on it.  However, the audience of the bulk of the Twisted 
documentation is fluent Python programmers.  If you want to write 
something for folks who are new to Python or programming in general, a 
separate set of documentation would be better than trying to re-explain 
everything in-line.  For example, explaining that we are referring to a 
directed graph of IResource providers would be fine, but adding an 
explanation of graphs or how python references or dictionaries work 
would not.
>OK, a little later, I
>have my first web server running an my first rpy works, spitting out
>"test". Wow.

Sounds like the docs are doing their job, then :).
>I'm wondering, why of all things "rpy" ? python won't generate sth. 
>like
>"rpyc" for that, which translates in lost speed, I assume. No 
>explanation.

"rpy" to mark the file as a specific kind of python file that needs to 
be run within the web server to produce a resource.  Implemented in this 
manner, python would not generate a "pyc" file even if it were called a 
".py"; it is not a module that you import.

As far as "lost speed" is concerned, it's not entirely clear that that's 
true, because the times when Python would have to check for the .rpyc 
are entirely different than the way that it does when you are importing 
modules.  Twisted could emit .rpycs as an optimization if you did a 
benchmark, discovered it did indeed matter, and submitted a patch.  This 
is, however, hardly the highest priority for twisted.web developers 
right now :).
>I'm on FreeBSD. Of course, I want to use kqueue. Now, another "funny"
>journey begins ...

>Intense googleing, and I mean "intense". Reading through years of this
>mailing list. No positive result.

Fair enough.  The kqueue reactor doesn't work and there's no 
documentation saying that it doesn't, except a failing buildbot: 
http://buildbot.twistedmatrix.com/builders/freebsd-py2.4-kq - also, 
Twisted's support for platforms is not clearly explained.
>I have the clear impression that twisted is something in between a toy 
>and a
>brilliant product. It's hackers, however, leave much to be desired in 
>terms
>of usefulness. As it is, it's a great and promising hobby but not a 
>useful
>product.

Many, many people (including me!) disagree.

I understand that you're frustrated, but this does nothing but serve to 
weaken your position.  You have some valid criticism, but when I read 
"twisted is not useful", I think that you are simply mistaken.
>Sorry, but doc strings don't replcae a halfway decent documentation and 
>a
>reasonable tutorial, considering the highly complex matter.

First google hit for "twisted tutorial":

http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html

Not perfect, perhaps, but I think it could definitely be described as a 
"reasonable tutorial" to core Twisted concepts.
>Sorry, but documentation that is often enough outdated and sometimes 
>offers
>broken links is next to useless.

If you were truly sorry, you would be contributing patches which fix the 
broken links and updates the documentation to be more recent.
>Kqueue seems to be vital to an event driven approach like TM, yet there 
>are
>multiple versions of pykqueue floating
>around, none of them properly working (and I didn't fumble around. I
>plain simply used the FreeBSD port), some of
>them with some mor and some with some less "annotations" (I refuse to
>call that doc.)

Kqueue is most definitely _not_ "vital" to something like Twisted.  It 
is an optimization of a very specific part of a Twisted application on a 
very specific platform.  *Most* realistic Twisted applications will be 
bottlenecked on application code long before something like kqueue (or 
epoll) will help.  Working from the amount of support from the 
community, most Twisted applications also don't run on FreeBSD.

I apologize if this response seems too harsh, but I am annoyed by people 
seeing a pet feature (variously: good ideas like HTTP/1.1, kqueue, and 
Cocoa GUI support, bad ideas like "block on a Deferred", and core API 
thread safety)  which isn't too important in Twisted and then claiming 
it's "vital" to the project's success.  The project is succeeding, so 
clearly it is not vital: QED.  It may be important to you: that's great. 
Do some work to support it.  Once it's supported, buildbot will listen 
to its tests and it won't break again.

For those projects which really do benefit from high multiplexing volume 
with something like kqueue, Twisted offers an extremely abstract API 
where the multiplexing mechanism is independent of the application code. 
You can, at any moment, replace the reactor and your application will 
keep working.  So even if kqueue support does not exist at all, you can 
add it with Twisted much more easily than if you wrote your own select() 
loop.

You might say, in fact, that Twisted is vital to a platform-specific 
tool like Kqueue, because otherwise almost nothing will use it.
>- How about getting 1 version of pykqueue properly running and into TM 
>?

Yes, how about that?  Maybe you could contribute to one of the numerous 
kqueue tickets already in the tracker.
>- How about freezing some TM version (like 8.0) and updating/matching 
>docs?
>(Of course, experiments are funny and intriguing, but quite some of us 
>out
>here need sth. stable for everydays work)

Twisted is very, very stable.  The next version (8.1) will break only 
code that is using APIs which have been outdated for literally _years_. 
There is no need to "freeze" anything, just update the documentation to 
what is most recent and do a new release including that documentation.
>- How about writing some complete docs and tutorials? Sth. along the 
>lines
>of "My first web server wit TM" (instead all those - sometimes seeming 
>to
>contradict each other - snippets)

Yes, how about that?  You can write documentation on your own site, and 
we will link to it.  Or, you could contribute to the core documentation.

The bottom line: complaining like this is not very useful.  If you have 
time to help, then contributing patches and responding to reviews in the 
tracker is helpful.  If you don't have the time to actually do any work, 
then describing specific problems (in as much detail as possible!) is 
useful.  If you don't have time even for _that_, then cash donations to 
the foundation are useful.

If you are interested in getting stuff in Twisted fixed, though, writing 
rambling complaints serves to do nothing but reduce our developers' 
already scarce motivation, make us think our work is not appreciated, 
and encourage us to spend time writing rambling responses like this one 
rather than fix the problems you're talking about.

(Which is not to say this has been a _complete_ waste of time.  If one 
out of every fifty people I write a message like this to understands 
what I'm saying and becomes a long-time contributor, then it's probably 
worth it.)