[Twisted-Python] [Fwd: Re: Deferred documentation rewrite]

[Glyph Lefkowitz]

On Fri, Jul 31, 2009 at 11:35 PM, Edward Z. Yang <ezyang at mit.edu> wrote:

> Excerpts from Itamar Shtull-Trauring's message of Fri Jul 31 22:26:29 -0400
> 2009:
> > The problem with this is that it perpetuates the misunderstanding the
> > Deferreds *make* things asynchronous, even with the intro that says
> > otherwise. I think it's better to assume already asynchronous code,
> > handling the transition from synchronous to async in an intro event loop
> > howto.
>
> Either way, the function that the first segment of the new docs do belong
> somewhere.  The documentation that traditionally served this purpose
> has been removed.


I'm actually inclined to agree.  Understanding what Deferreds are *for* sort
of implies an understanding of why one would want to have a list of
callbacks in the first place, which implies a need to understand
asynchronous programming.

Realistically, I don't think there are a lot of people out there who have
managed to do anything at all useful with Twisted without undersanding
Deferreds.

I think it might be useful to lead into this section with a specific example
problem though, rather than a very vague general template for the category
of problems.  Once you've demonstrated a specific problem, you can always
expand the general category by briefly listing some other problems which are
similar.

As for perpetuating the misunderstanding of Deferreds making things
> asynchronous, I completely agree!  However, I think this is something
> that can be fixed by spelling out the distinction between "writing
> asynchronous code" and "interacting with asynchronous code", and not
> just omitting the important paradigm shift that comes with sync->async.


By the time you have to spell out that distinction, I think it's too late.
By the time the user is dealing with a Deferred, they should have a pretty
good idea that it's just a list of functions that will be called at some
point in the future; if you have to say "here's this magic thing, oh, by the
way, it turns out it's just a list of functions" they're still going to
think something's magic about it.

If you want an example that doesn't involve any reactor integration, you
might try showing a program that polls a directory for a file being added.
If you want to simultaneously wait for a file named "1" and a file named
"2", you need to use callbacks, otherwise "waiting" for 2 will prevent the
code for 1 from running immediately if 1 shows up first.

I'm not really sure this can be done in a simple enough way to be useful, so
take it with a grain of thought.

> A better comparative exposition might be with normal callbacks, e.g.:
>
>
> > "def foo(x, gotResultCallback): pass" vs. "def foo(x): # return
> > Deferred".
> >
> > At the very least having that async but callbacky version in the middle
> > helps understanding.
>
> I briefly gloss on this, but I agree that this is an important point
> that could be further expanded.  We could have implemented asynchronous
> mechanisms using normal callbacks, but we decided to use Deferreds instead.
>

Definitely agreed here.  For those users who are used to "normal" callbacks
(if there is such a thing), this explanation would be helpful to explain why
you need them rather than just passing functions around.  For users who
aren't even familiar with passing functions around, this would fill an
important intermediate gap.


> It's not clear to me if the common case of confusion of Deferreds occurs
> in people who "know callbacks" but "don't know Deferreds."  As an incoming
> developer who was familiar with asynchronous programming, my primary
> problem
> was the ill-defined behavior of callback chains (which I resolved by
> hunkering down and reading the source code) rather than any fundamental
> misunderstanding of what Deferreds were supposed to do.
>

I think we need an expanded version of the "visual explanation" here: <
http://twistedmatrix.com/projects/core/documentation/howto/defer.html> (one
that uses complete sentences and more than one diagram), as well as an
expanded analogy to try:except: blocks in 'synchronous' python.

I object to the classification of the behavior of callback chains as
"ill-defined".  It might have been poorly-explained, but that's very
different :).

> It also omits half the story: how you *create* Deferreds. There should
> > be a section on that as well.
>
> I agree.  In fact, it might be worth making the document a little longer
> to address this point, because I realize now that even if you're not
> writing asynchronous code, you'll often need to baton Deferreds to
> make the execution flow work the way you want them to.
>

+1.  Dealing with Deferred creation is a good way to introduce users to the
non-magicness of Deferreds.  It's less common to create one than to consume
one, but understanding how to create them should make it easier to
understand how to consume them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090801/7b297d95/attachment-0001.html>