[Twisted-Python] Design pattern for multi-stage web searches

[Luciano Ramalho]

Hello,

I am a Google Summer of Code student working with the Zope 3/Grok
community this year.

I wonder if there is an established design pattern or examples that
use Twisted to fetch data via HTTP when the process can take a
variable number of requests to be completed.

For instance, if you search for book records online, some sites allow
you to do an ISBN search which gives back the full book details, while
others will present an intermediate results page from which links to
the book details page can be obtained (many sites use a internal id
for their products, including books). Still others can return an
intermediate page containing a link to a different edition (!= ISBN),
when the given ISBN refers to an out-of-print edition.

So for each different metadata source I'd like to provide a chain of
callbacks to be processed. Anyone knows of references that may help
me? I browsed the GoF patterns but none of them fits this use case.

BTW, the resulting code will be open source under the ZPL and hosted
at http://svn.zope.org.

Thanks for a great piece of software!

Regards,

Luciano

PS. I've already implemented, using Twisted, a prototype of the
collector for Amazon.com and it's affiliates in Europe and Japan. But
I am Brazilian and I know their catalog is not very complete outside
of the markets where they operate. To be really international, a book
metadata collector must be pluggable to allow localized searches
depending on the ISBN prefix.