[Twisted-Python] deferToThread - supported alternative to the (deprecated) setTimeout method

[Andrew Bennetts]

On Tue, May 23, 2006 at 09:49:23PM +0100, Paul Moore wrote:
[...]
> 
> The problem is that in rare cases, the blocking call can block
> indefinitely. In this case, I need to make the call time out. However,
> the underlying API offers no way to time out the call, so I have to do
> this externally.

There is a problem here Twisted cannot help you with -- you will leak threads
that you cannot interrupt, because they are blocked indefinitely.  setTimeout or
other solutions will allow you to paper over this fact, but you need to be aware
of it, because it will eventually stop your process from running.  There's no
way something like defer.setTimeout can magically cancel the underlying
operation for you (even though it unblocks the deferred waiting on that
operation), and part of the reason setTimeout is a deprecated is to force people
to be aware of that.

Also, deferToThread by default isn't going to give you with 50-100 parallel
threads.  The default threadpool size is 4.  You'll need to use
reactor.suggestThreadPoolSize to change that -- but realise that the default DNS
resolver in current Twisted releases use that threadpool too, and perhaps so
will other libraries, and you could starve those callers by swamping the
threadpool with your blocking calls.  So it may be better to use your own
threadpool, as twisted.enterprise.adbapi does, for example.  See
twisted.python.threadpool.

I don't suppose there's a non-blocking way to do what you want?

Oh -- and remember that while you can't kill threads, you *can* kill processes.
Consider using subprocesses to do your blocking work.

-Andrew.