[Twisted-Python] Setting a timeout for HTTPDownloader

[Edd Dumbill]

Hello,

Firstly, thanks for Twisted, I'm just starting to use it and I'm
impressed with what I see.  I'm also grateful for the help folks have
given me on the IRC channel.

So here's my problem.  I'm adapting the spider from Twisted to my own
purposes.  One problem I've come across is this: that an HTTP connection
will be opened OK, but is just held open with no return for many many
minutes by a server.

So, I want to implement a timeout.  As I understand it, this is
different from the timeout given at reactor.connectTCP time, as it's
post-connect.  Certainly that's how it looks from here.

I guess that somewhere I want a callLater which stops a download after
so many seconds, and that's what I tried to do.  Unfortunately I have no
idea what to do to implement that.  Most of my attempts ended up in
raising exceptions that brought everything to a halt: my understanding
of the thread of execution throughout the various Twisted components is
currently very immature.

I'd be grateful for pointers on how to implement this, with reference to
twisted/web/spider.py.  Essentially what I want is a timeout to cause an
error that will result in SpiderSender.downloadFailed being invoked,
pretty much as for any other error relating to the HTTP connection does.

With many thanks,

-- Edd