[Twisted-Python] Non-blocking http client?

[Jean-Paul Calderone]

On Sat, 13 Dec 2008 10:08:43 -0800, Erik Wickstrom <erik at erikwickstrom.com> wrote:
>Hi all,
>
>I have an application that is doing some web spidering.  Right now I'm
>using urllib to retrieve the URLs, but it is painfully slow.  I was
>wondering if it's feasible to swap out urllib with a twisted client
>that uses deferds so I can process urls in a more "parallel" fashion?
>
>I've done a bunch of Googleing, but I haven't come across anything
>that I can use as a drop in replacement.   If you can point me in the
>right direction I'd really appreciate it!

There is no drop-in replacement.  There is an HTTP client in Twisted,
`twisted.web.client´.  There is also a more featureful HTTP client in
development but not yet available in a release of Twisted.  You can
find an example of how to use the former in the Twisted Web examples,
<http://twistedmatrix.com/projects/web/documentation/examples/>, and
you can follow progress of the latter at <http://twistedmatrix.com/trac/ticket/886>.

Jean-Paul