[Twisted-Python] Scalability of an rss-aggregator

[Valentino Volonghi aka Dialtone]

Andrew Bennetts wrote:

>On Thu, Apr 01, 2004 at 12:13:03PM +0200, Valentino Volonghi aka Dialtone wrote:
>  
>
>>I went on testing and found some interesting things...
>>    
>>
>
>I just found something interesting too -- I just took a quick peek at
>feedparser, and the parse function looks like it fetches the page
>synchronously using urllib.  That's *not* a good thing to do from inside
>Twisted's main loop.  It looks like you want your parseFeed method to call:
>
>    r = FeedParser(baseuri)
>    r.feed(data)
>
>like feedparser.parse does internally.  (You'll need to do a little bit of
>work to return the same sort dictionary that parse constructs for you).
>  
>
I verified... The parser only downloads if as url is supplied, otherwise 
it does not.

Anyway, since it needs a StringIO like argument, now I do myself the 
conversion into StringIO and this
will make the parser always return as fast as possible from 
_open_resource().

I'm starting to think that this is some kind of 'race condition' inside 
twisted, or something similar.
BTW, I'll investigate more and more :)

-- 
Valentino Volonghi aka Dialtone
Linux User #310274, Gentoo Proud User
X Python Newsreader developer
http://sourceforge.net/projects/xpn/