[Twisted-Python] Non-blocking http client?

[Denis Bilenko]

I've been working on eventlet-twisted integration and it includes a drop-in
replacements for standard Python modules (including urllib and httplib) that
do not block the reactor.

There was no release yet and there're no documentation written.
(You can find the documentation for the original eventlet here
http://wiki.secondlife.com/wiki/Eventlet,
however, not all of it is applicable. You can also read the
documentation of greenlet here
http://codespeak.net/py/dist/greenlet.html)

However, usage is very simple.
1. add this line before using any eventlet functions
from eventlet.twistedutil import join_reactor
2. import eventlet.green.urllib instead of urllib
3. Use eventlet.twistedutil.deferToGreenThread function to make
"blocking" functions
from eventlet.green.urllib return Deferred.

Here's an example on how to use eventlet.green.httplib and twisted's
LineReceiver
together to build a toy http proxy:

http://devel.ag-projects.com/~denis/cgi-bin/hgweb.cgi/file/097d879e0228/examples/twisted_http_proxy.py

To get the repository:

hg clone http://devel.ag-projects.com/~denis/cgi-bin/hgweb.cgi/eventlet_twisted

The only dependency is py.magic.greenlet from py library.
The code is only tested on python2.5 linux & macos so it probably won't work
on python2.4/windows but that will be fixed sometime in the future.

On Sun, Dec 14, 2008 at 12:08 AM, Erik Wickstrom <erik at erikwickstrom.com> wrote:
> Hi all,
>
> I have an application that is doing some web spidering.  Right now I'm
> using urllib to retrieve the URLs, but it is painfully slow.  I was
> wondering if it's feasible to swap out urllib with a twisted client
> that uses deferds so I can process urls in a more "parallel" fashion?
>
> I've done a bunch of Googleing, but I haven't come across anything
> that I can use as a drop in replacement.   If you can point me in the
> right direction I'd really appreciate it!
>
> Thanks for your help!
> Erik
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>