[Twisted-Python] documentation / kqueue / feedback

[Gabriel Rossetti]

glyph at divmod.com wrote:
>
> On 01:15 pm, dr.pythoniac at googlemail.com wrote:
>> Seeing there is doc. available, I digged through it. After all TM seems
>> worth the effort. From what I see, I love TM.
>
> Great!  The rest definitely doesn't sound like you love it though ... ;-)
>> Links are often broken, though. Often seemingly useful doc. is old, very
>> old. Even current doc. isn't up to date.
>
> Since the 8.0 release, we have had quite a few problems with links, 
> specifically the links from the core documentation to the API 
> documentation on the website.  We are working to fix this.  However, 
> if you spot broken links, please feel free to report them in the bug 
> tracker: http://twistedmatrix.com/trac/newticket
>
> Saying that there are "often" broken links is just about as pointless 
> as saying that you "often" find bugs.  Without specifics, this is not 
> useful information; the reason we have not fixed specific problems is 
> not because we believe there are no problems anywhere.
>> They talk about "resource trees", yet I don't find them in the doc. I 
>> find
>> putchild() but all examples and docs indicate, that you can do it only
>> within the root resource.
>
> You can do putChild to any resource, as long as you understand its 
> lifecycle.  A "resource tree" is simply a tree ( 
> http://en.wikipedia.org/wiki/Tree_structure ) of resources ( 
> http://twistedmatrix.com/documents/current/api/twisted.web.resource.IResource.html 
> ).
>
> Our documentation assumes, and will continue to assume, a certain 
> level of familiarity with basic computer-sciency jargon; if you feel 
> that a particular phrase is too obscure, feel free to submit a patch 
> elaborating on it.  However, the audience of the bulk of the Twisted 
> documentation is fluent Python programmers.  If you want to write 
> something for folks who are new to Python or programming in general, a 
> separate set of documentation would be better than trying to 
> re-explain everything in-line.  For example, explaining that we are 
> referring to a directed graph of IResource providers would be fine, 
> but adding an explanation of graphs or how python references or 
> dictionaries work would not.
>> OK, a little later, I
>> have my first web server running an my first rpy works, spitting out
>> "test". Wow.
>
> Sounds like the docs are doing their job, then :).
>> I'm wondering, why of all things "rpy" ? python won't generate sth. like
>> "rpyc" for that, which translates in lost speed, I assume. No 
>> explanation.
>
> "rpy" to mark the file as a specific kind of python file that needs to 
> be run within the web server to produce a resource.  Implemented in 
> this manner, python would not generate a "pyc" file even if it were 
> called a ".py"; it is not a module that you import.
>
> As far as "lost speed" is concerned, it's not entirely clear that 
> that's true, because the times when Python would have to check for the 
> .rpyc are entirely different than the way that it does when you are 
> importing modules.  Twisted could emit .rpycs as an optimization if 
> you did a benchmark, discovered it did indeed matter, and submitted a 
> patch.  This is, however, hardly the highest priority for twisted.web 
> developers right now :).
>> I'm on FreeBSD. Of course, I want to use kqueue. Now, another "funny"
>> journey begins ...
>
>> Intense googleing, and I mean "intense". Reading through years of this
>> mailing list. No positive result.
>
> Fair enough.  The kqueue reactor doesn't work and there's no 
> documentation saying that it doesn't, except a failing buildbot: 
> http://buildbot.twistedmatrix.com/builders/freebsd-py2.4-kq - also, 
> Twisted's support for platforms is not clearly explained.
>> I have the clear impression that twisted is something in between a 
>> toy and a
>> brilliant product. It's hackers, however, leave much to be desired in 
>> terms
>> of usefulness. As it is, it's a great and promising hobby but not a 
>> useful
>> product.
>
> Many, many people (including me!) disagree.
>
> I understand that you're frustrated, but this does nothing but serve 
> to weaken your position.  You have some valid criticism, but when I 
> read "twisted is not useful", I think that you are simply mistaken.
>> Sorry, but doc strings don't replcae a halfway decent documentation 
>> and a
>> reasonable tutorial, considering the highly complex matter.
>
> First google hit for "twisted tutorial":
>
> http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html 
>
>
> Not perfect, perhaps, but I think it could definitely be described as 
> a "reasonable tutorial" to core Twisted concepts.
>> Sorry, but documentation that is often enough outdated and sometimes 
>> offers
>> broken links is next to useless.
>
> If you were truly sorry, you would be contributing patches which fix 
> the broken links and updates the documentation to be more recent.
>> Kqueue seems to be vital to an event driven approach like TM, yet 
>> there are
>> multiple versions of pykqueue floating
>> around, none of them properly working (and I didn't fumble around. I
>> plain simply used the FreeBSD port), some of
>> them with some mor and some with some less "annotations" (I refuse to
>> call that doc.)
>
> Kqueue is most definitely _not_ "vital" to something like Twisted.  It 
> is an optimization of a very specific part of a Twisted application on 
> a very specific platform.  *Most* realistic Twisted applications will 
> be bottlenecked on application code long before something like kqueue 
> (or epoll) will help.  Working from the amount of support from the 
> community, most Twisted applications also don't run on FreeBSD.
>
> I apologize if this response seems too harsh, but I am annoyed by 
> people seeing a pet feature (variously: good ideas like HTTP/1.1, 
> kqueue, and Cocoa GUI support, bad ideas like "block on a Deferred", 
> and core API thread safety)  which isn't too important in Twisted and 
> then claiming it's "vital" to the project's success.  The project is 
> succeeding, so clearly it is not vital: QED.  It may be important to 
> you: that's great. Do some work to support it.  Once it's supported, 
> buildbot will listen to its tests and it won't break again.
>
> For those projects which really do benefit from high multiplexing 
> volume with something like kqueue, Twisted offers an extremely 
> abstract API where the multiplexing mechanism is independent of the 
> application code. You can, at any moment, replace the reactor and your 
> application will keep working.  So even if kqueue support does not 
> exist at all, you can add it with Twisted much more easily than if you 
> wrote your own select() loop.
>
> You might say, in fact, that Twisted is vital to a platform-specific 
> tool like Kqueue, because otherwise almost nothing will use it.
>> - How about getting 1 version of pykqueue properly running and into TM ?
>
> Yes, how about that?  Maybe you could contribute to one of the 
> numerous kqueue tickets already in the tracker.
>> - How about freezing some TM version (like 8.0) and updating/matching 
>> docs?
>> (Of course, experiments are funny and intriguing, but quite some of 
>> us out
>> here need sth. stable for everydays work)
>
> Twisted is very, very stable.  The next version (8.1) will break only 
> code that is using APIs which have been outdated for literally 
> _years_. There is no need to "freeze" anything, just update the 
> documentation to what is most recent and do a new release including 
> that documentation.
>> - How about writing some complete docs and tutorials? Sth. along the 
>> lines
>> of "My first web server wit TM" (instead all those - sometimes 
>> seeming to
>> contradict each other - snippets)
>
> Yes, how about that?  You can write documentation on your own site, 
> and we will link to it.  Or, you could contribute to the core 
> documentation.
>
> The bottom line: complaining like this is not very useful.  If you 
> have time to help, then contributing patches and responding to reviews 
> in the tracker is helpful.  If you don't have the time to actually do 
> any work, then describing specific problems (in as much detail as 
> possible!) is useful.  If you don't have time even for _that_, then 
> cash donations to the foundation are useful.
>
> If you are interested in getting stuff in Twisted fixed, though, 
> writing rambling complaints serves to do nothing but reduce our 
> developers' already scarce motivation, make us think our work is not 
> appreciated, and encourage us to spend time writing rambling responses 
> like this one rather than fix the problems you're talking about.
>
> (Which is not to say this has been a _complete_ waste of time.  If one 
> out of every fifty people I write a message like this to understands 
> what I'm saying and becomes a long-time contributor, then it's 
> probably worth it.)
>
Hello,

there are links that are always broken, like for example : 
http://twistedmatrix.com/projects/core/documentation/howto/process.html, 
if you look at this line :

"If you only need the final exit code (like 
|commands.getstatusoutput(cmd)[0]|), the 
|twisted.internet.utils.getProcessValue 
<http://twistedmatrix.com/projects/core/documentation/howto/twisted.internet.utils.getProcessValue>| 
function is useful."

The link being : 
http://twistedmatrix.com/projects/core/documentation/howto/twisted.internet.utils.getProcessValue

All those types of links in the howtos/tutorials are bad. Should I write 
a bugreport? If so, would an explanation like the one above suffice (I 
can't find all the links in the tutorials that are bad)?

Gabriel