[Twisted-Python] Deferred's chainDeferred is too simplistic?

[Terry Jones]

I think it's worth spending some time thinking about whether chainDeferred
in t.i.defer.Deferred is too simplistic. I've thought for a while that it
could be more helpful in preventing people from doing unintended things
and/or cause fewer surprises (e.g., see point #1 at http://bit.ly/bp6iT5
which JP agreed with in the followup).

Those are smaller things that people can obviously live with. But a more
important one concerns deferred cancellation. See uncalled.py below.

Here are five examples of chainDeferred behavior that I think could be
better. In an attempt to fix these, I made a few changes to a copy of
defer.py (immodestly called tdefer.py, sorry) which you can grab, with
runnable examples, from http://github.com/fluidinfo/chainDeferredExamples
The tdefer.py code is meant as a suggested approach. I doubt that it's
bulletproof.

Here are the examples.

boom1.py:

    # Normal deferreds: this raises defer.AlreadyCalledError because
    # the callback of d1 causes the callback of d2 to be called, but d2 has
    # already been cancelled (and hence called).

    # With tdefer.py: there is no error because d1.callback will not call
    # d2 as it has already been cancelled.

    def printCancel(fail):
        fail.trap(defer.CancelledError)
        print 'cancelled'

    def canceller(d):
        print 'cancelling'

    d1 = defer.Deferred()
    d2 = defer.Deferred(canceller)
    d2.addErrback(printCancel)
    d1.chainDeferred(d2)
    d2.cancel()
    d1.callback('hey')


boom2.py:

    # Normally: raises defer.AlreadyCalledError because calling d1.callback
    # will call d2, which has already been called.

    # With tdefer.py: Raises AssertionError: "Can't callback an already
    # chained deferred" because calling callback on a deferred that's
    # already been chained is asking for trouble (as above).

    d1 = defer.Deferred()
    d2 = defer.Deferred()
    d1.chainDeferred(d2)
    d2.callback('hey')
    d1.callback('jude')


uncalled.py:

    # Normally: although d2 has been chained to d1, when d1 is cancelled,
    # d2's cancel method is never called. Even calling d2.cancel ourselves
    # after the call to d1.cancel has no effect, as d2 has already been
    # called.

    # With tdefer: both cancel1 and cancel2 are called when d1.cancel is
    # called. The additional final call to d2.cancel correctly has no
    # effect as d2 has been called (via d1.cancel).

    def cancel1(d):
        print 'cancel one'

    def cancel2(d):
        print 'cancel two'

    def reportCancel(fail, which):
        fail.trap(defer.CancelledError)
        print 'cancelled', which

    d1 = defer.Deferred(cancel1)
    d1.addErrback(reportCancel, 'one')
    d2 = defer.Deferred(cancel2)
    d2.addErrback(reportCancel, 'two')
    d1.chainDeferred(d2)
    d1.cancel()
    d2.cancel()


unexpected1.py:

    # Normally: prints "called: None", instead of the probably expected
    # "called: hey"

    # tdefer.py: prints "called: hey"

    def called(result):
        print 'called:', result

    d1 = defer.Deferred()
    d2 = defer.Deferred()
    d1.chainDeferred(d2)
    d1.addCallback(called)
    d1.callback('hey')


unexpected2.py:

    # Normally: prints
    #   called 2: hey
    #   called 3: None

    # tdefer.py: prints
    #   called 2: hey
    #   called 3: hey

    def report2(result):
        print 'called 2:', result

    def report3(result):
        print 'called 3:', result

    d1 = defer.Deferred()
    d2 = defer.Deferred().addCallback(report2)
    d3 = defer.Deferred().addCallback(report3)
    d1.chainDeferred(d2)
    d1.chainDeferred(d3)
    d1.callback('hey')


I wont go into detail here as this post is already long enough. Those are 3
classes of behavior arising from chainDeferred being very simplistic.
Comments welcome, of course.  Once again, runnable code is at
http://github.com/fluidinfo/chainDeferredExamples

Terry