[Twisted-Python] Scalability of an rss-aggregator

[Valentino Volonghi aka Dialtone]

Hi all,
attached you will find my rss-aggregator made with twisted.

It's really fast although when I tried with 745 feeds I got some problems.
When the download reached 300 parsed feeds (more or less) it locked till 
I pressed Ctrl+C and then it
processed the remaining 340 feeds in less than 30 seconds... I think 
that my design has at least an issue
but  I cannot find it so easily and I hope someone on this list can help 
me to improve it.

The "script" is heavily commented.

BTW When it finishes (with all 740 feeds) it reports an awesome 330 
seconds which is an impressive time, less than half a second
for each feed, and It downloads more than 50Mb of feeds from the net 
(with 745 feeds to download).

Thx for your help.

-- 
Valentino Volonghi aka Dialtone
Linux User #310274, Gentoo Proud User
X Python Newsreader developer
http://sourceforge.net/projects/xpn/

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: twisted-rss.py
URL: </pipermail/twisted-python/attachments/20040331/ef318972/attachment.ksh>