[Twisted-Python] One big smile ...

[Abe Fettig]

On Fri, 2003-07-11 at 20:56, Thomas Weholt wrote:
> I got a list of RSS/RDF-urls and I iterate thru them in sequence, feeding
> each one to Mark Pilgrims Ultra Liberal Parser. The call to that
> parse-method is actually what's being using in deferToThread. The I add a
> callback-method which processes the result of the RSS/RDF-parsing. It's a
> plain dictionary. This dictionary is processed into a simpler list. That's
> it.
> 
> Alot of the things I'll be doing is like this; fetch a file from the net (
> here the parser fetches it itself ), process the data returned from that
> call and update some object or stuff the result into a database.

While Mark's feedparser does include it's own code for downloading the
feed over HTTP, it doesn't make sense to use it in a Twisted app.  You
should let Twisted download the rss file, thus avoiding the need to run
feedparser in a seperate thread.  It's easy enough to use the feedparser
parsing code on it's own once you've downloaded the file - that's what I
do in Hep.

I've attached two examples.  The first downloads my RSS feed using
Twisted, parses it with feedparser, and prints the results.  As you can
see, it's only a few lines of code.  The second example does the same
thing, but runs the parsing code in a seperate thread, so that you avoid
blocking even for the small amount of time feedparser takes to parse the
RSS (whether it actually makes sense to do this depends on the needs of
your application).

Abe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rss_demo.py
Type: text/x-python
Size: 407 bytes
Desc: not available
URL: </pipermail/twisted-python/attachments/20030712/25e187c9/attachment-0004.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rss_demo2.py
Type: text/x-python
Size: 701 bytes
Desc: not available
URL: </pipermail/twisted-python/attachments/20030712/25e187c9/attachment-0005.py>