[Twisted-Python] setTimeout in Deferred

[Andrew Bennetts]

On Wed, Mar 09, 2005 at 11:53:58PM -0500, Yun Mao wrote:
[...]
> 
> The problem is: many of the result-generating APIs do not have a timeout 
> option, for example, XMLRPC calls, Database queries, etc. What can we do?
> Are you suggesting to modify those APIs? I believe the framework should 
> have something related to that.

At the moment, the best thing to do is to modify those APIs, yes.
Fundamentally, cancellation needs to be supported by whatever code is
managing the operation that the Deferred is waiting on. 

There may be a better way to do this, see the discussion in
http://twistedmatrix.com/bugs/issue178 (and the mailing list thread it links
to).  Realistically, not all operations can be cancelled, even though the
clients of those operations may no longer care about the result.  Any
solution needs to have sensible support for this case, and at the moment
setTimeout doesn't.

[...]
> Even with APIs that have the timeout option, there could also be problems 
> when you use them in some form of combination. E.g. I'd write a procedure 
> which involves one DNSLookup and one getPage() sequentially. I could 
> specify timeout value with 2 seconds each, however what I really want the 
> total time to be less than 4 seconds in which case DNS could take longer. 
> It is not terribly hard to implement base on what we have now but it would 
> be nice if there is a framework support that can calculate the correct 
> timers for me.

Right, what is actually the case there is there are three different
operations:

   1. the DNS lookup
   2. the getPage
   3. your procedure that invokes 1 and 2.

The code responsible for 3 would also be responsible for cancelling 1 and 2
when 3 is cancelled.  No fancy timeout calculations would be required.

Currently we lack ways to cancel in-progress getPage operations (and
probably DNS lookups, I haven't looked), and we certainly lack any standard
APIs or framework for supporting cancellations.  This is the root problem
that needs fixing, rather than timeouts, I think.

> Another form of "cancellation" is also related to the chaining effect. 
> Suppose I want to start following two jobs at the same time:
> 
> 1: A()->B()-C()
> 2: D()->E()
> 
> However, I only need the result from any of of them. i.e. as long as one 
> of them is finished, the other thread is useless and should be canceled 
> ASAP to prevent from resource consuming. Right now, I'm using 
> DeferredList, but it doesn't cancel the other thread. A similar 
> requirement is that if either one of them got errback, the other should be 
> canceled.

Yeah, this would be nice... but of course there's no generic way to kill
another thread in python.  The thread needs to co-operate somehow.

> Anyway, being able to schedule flexible timers is a very big advantage in 
> async programming over multithreading to me. Otherwise, signals, locks, 
> etc could just drive me nuts. However, I still believe in Twisted has the 
> potential to have better framework support.

Magic bullets welcome ;)

-Andrew.