[Twisted-Python] Design pattern for multi-stage web searches

[Jean-Paul Calderone]

On Sun, 5 Aug 2007 16:39:00 -0300, Luciano Ramalho <luciano at ramalho.org> wrote:
>Hello,
>
>I am a Google Summer of Code student working with the Zope 3/Grok
>community this year.
>
>I wonder if there is an established design pattern or examples that
>use Twisted to fetch data via HTTP when the process can take a
>variable number of requests to be completed.

This is usually done by "chaining" Deferreds together.  When a callback
or errback function on Deferred A returns Deferred B, B is chained to a:

    B.chainDeferred(A)

or

    B.addCallbacks(A.callback, A.errback)

So if you fetch a page and it turns out not to be the one you ultimately
want, a callback on that Deferred can initiate another fetch and return
the Deferred for that operation.  This is transparent to the caller of
the original function: their callbacks are only called once all of the
earlier callbacks have returned a None-Deferred value and all returned
Deferreds have fired.

I doubt I've explained this particularly well, so you might want to take
a look at http://twistedmatrix.com/projects/core/documentation/howto/defer.html to
really get an understanding.

Jean-Paul