[Twisted-Python] Question about deferreds

[glyph at divmod.com]

On 01:48 pm, dfarrell at mypublisher.com wrote:
>Hi all,

>I consider myself a Twisted "newbie", so this might seem like a simple
>couple of questions, hope you'll indulge me.

I hope a simple couple of answers will suffice.  Thank you for your 
clearly-worded questions :).
>When I first began experimenting with deferreds I was breaking up tasks
>into sequences of steps something like this.
(snip)
>When this runs I was under the mistaken impression that at the end of
>each callback (step1, step2, step3, etc) the deferred would yield back
>to the main loop (reactor) and allow other deferreds to run. Now I know
>that the callback functions run sequentially one after another with no
>break. So my first question is this: what is the advantage of this over
>just combining the callback functions into one big function?, something
>like this:

>d = Deferred()
>
>d.succeed(True)
>
>d.addCallback(OneBigStep)

Deferreds are not about breaking your work up into steps to save time. 
They're about breaking it up so that different systems can deal with the 
same asynchronous event in their own way.

For example, let's say you're working on a REST client that deals with 
invoices.  The server gives you data at a particular URL and you want to 
convert it into an "Invoice" instance.

Twisted itself isn't going to have that OneBigStep callback that 
converts HTTP straight into invoices for you, so you've got something 
that gives back a Deferred that fires with a string, getPage.  Then 
maybe there's an intermediary layer that deals with parsing XML or JSON 
or whatever.  Then there's a layer that takes the JSON or DOM structure 
and converts it into an Invoice.  Finally you've got your actual 
application code that wants to get the Invoice and *do* something with 
that Invoice.

And of course your final "do something with that invoice" step might 
need data retrieved asynchronously from other places, so it might need 
other Deferreds and other callbacks as well.

If you do have step1, step2, and step3 defined entirely in a single 
function, then yes, they should be in the same function.
>If step1(), step2() and step3() take a fair bit of time to complete,
>doesn't this effectively block the reactor loop for the entire exection
>of all the steps?

Yep.  That's the idea.  If you have long-running tasks that need to be 
split up into multiple steps, there are other facilities, *not* Deferred 
callbacks, that you can use.  For example, this:

http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.task.html#coiterate
>My second question is related to this; what is a recommended way to
>create the cooperative state machine I'm thinking of using deferreds?

(...)
>How do I get the execution of this to be this:

>1)      d1_step1
>2)      d2_step1
>3)      d1_step2
>4)      d2_step2
>5)      d1_step3
>6)      d2_step3

>I want to have the state machines I create cooperatively 'multi-task',
>yielding control back to the main loop at each state transition.

If you actually wanted to interleave Deferred callbacks like that it 
would be tricky.  But if you just want interleaved callbacks, use 
Cooperator as I mentioned above :).
>Thanks in advance for any help, guidance or references you can provide.
>I really enjoy Twisted, I'm still working on getting the 'Zen' of it.

Thanks!