[Twisted-Python] etag and last-modified
Jacob Friis
lists at debpro.webcom.dk
Fri Nov 5 07:53:34 MST 2004
exarkun at divmod.com wrote:
>>Instead I'll let Feedparser do the download and use twisted for threads.
>>What is the maximum pool size I can use?
>
> Using threads to do this, there is no point using Twisted at all. Since Twisted is perfectly capable of downloading select web pages based on their headers, there's no reason to use threads.
But I need to download approx 150000 files several times every day from
approx 7000 servers. That's why I thought threading would be the solution.
Can I use Twisted for this?
Thanks,
Jacob
More information about the Twisted-Python
mailing list