[Twisted-Python] etag and last-modified

[exarkun at divmod.com]

On Fri, 05 Nov 2004 15:53:34 +0100, Jacob Friis <lists at debpro.webcom.dk> wrote:
>exarkun at divmod.com wrote:
> >>Instead I'll let Feedparser do the download and use twisted for threads.
> >>What is the maximum pool size I can use?
> > 
> >   Using threads to do this, there is no point using Twisted at all.  Since Twisted is perfectly capable of downloading select web pages based on their headers, there's no reason to use threads.
> 
> But I need to download approx 150000 files several times every day from 
> approx 7000 servers. That's why I thought threading would be the solution.
> Can I use Twisted for this?

  Threads are an _inferior_ mechanism for network concurrency.  Twisted can download multiple files simultaneously without using threads, and generally speaking, more efficiently than using threads.

  Jp