Deferreds are beautiful! (A Tutorial)

  1. Introduction
  2. A simple example
  3. Errbacks
  4. addBoth: the deferred version of finally
  5. addCallbacks: decision making based on previous success or failure
  6. Hints, tips, common mistakes, and miscellaney
  7. Conclusion

Introduction

Deferreds are quite possibly the single most confusing topic that a newcomer to Twisted has to deal with. I am going to forgo the normal talk about what deferreds are, what they aren't, and why they're used in Twisted. Instead, I'm going show you the logic behind what they do.

A deferred allows you to encapsulate the logic that you'd normally use to make a series of function calls after receiving a result into a single object. In the examples that follow, I'll first show you what's going to go on behind the scenes in the deferred chain, then show you the deferred API calls that set up that chain. All of these examples are runnable code, so feel free to play around with them.

A simple example

First, a simple example so that we have something to talk about: 
#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
here we have the simplest case, a single callback and a single errback
"""

num = 0

def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)

def handleResult(result):
    global num; num += 1
    print "callback %s" % (num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"


def behindTheScenes(result):
    # equivalent to d.callback(result)

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()


def deferredExample():
    d = defer.Deferred()
    d.addCallback(handleResult)
    d.addErrback(handleFailure)

    d.callback("success")


if __name__ == '__main__':
    behindTheScenes("success")
    print "\n-------------------------------------------------\n"
    global num; num = 0
    deferredExample()
Source listing - listings/deferred/deferred_ex.py

And the output: (since both methods in the example produce the same output, it will only be shown once.) 

callback 1
        got result: success

Here we have the simplest case. A deferred with a single callback and a single errback. Normally, a function would create a deferred and hand it back to you when you request an operation that needs to wait for an event for completion. The object you called then does d.callback(result) when the results are in.

The thing to notice is that there is only one result that is passed from method to method, and that the result returned from a method is the argument to the next method in the chain. In case of an exception, result is set to an instance of Failure that describes the exception.

Errbacks

Failure in requested operation

Things don't always go as planned, and sometimes the function that returned the deferred needs to alert the callback chain that an error has occurred.

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
this example is analogous to a function calling .errback(failure)
"""


class Counter(object):
    num = 0

def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"


def behindTheScenes(result):
    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()


def deferredExample(result):
    d = defer.Deferred()
    d.addCallback(handleResult)
    d.addCallback(failAtHandlingResult)
    d.addErrback(handleFailure)

    d.errback(result)


if __name__ == '__main__':
    result = None
    try:
        raise RuntimeError, "*doh*! failure!"
    except:
        result = failure.Failure()
    behindTheScenes(result)
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample(result)
Source listing - listings/deferred/deferred_ex1a.py
errback
we got an exception: Traceback (most recent call last):
--- exception caught here ---
  File "deferred_ex1a.py", line 73, in ?
    raise RuntimeError, "*doh*! failure!"
exceptions.RuntimeError: *doh*! failure!

The important thing to note (as it will come up again in later examples) is that the callback isn't touched, the failure goes right to the errback. Also note that the errback trap()s the expected exception type. If you don't trap the exception, an error will be logged when the deferred is garbage-collected.

Exceptions raised in callbacks

Now let's see what happens when our callback raises an exception

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
here we have a slightly more involved case. The deferred is called back with a
result. the first callback returns a value, the second callback, however
raises an exception, which is handled by the errback.
"""


class Counter(object):
    num = 0

def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"


def behindTheScenes(result):
    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = failAtHandlingResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()


def deferredExample():
    d = defer.Deferred()
    d.addCallback(handleResult)
    d.addCallback(failAtHandlingResult)
    d.addErrback(handleFailure)

    d.callback("success")


if __name__ == '__main__':
    behindTheScenes("success")
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample()
Source listing - listings/deferred/deferred_ex1b.py

And the output: (note, tracebacks will be edited slightly to conserve space)

callback 1
        got result: success
callback 2
        got result: yay! handleResult was successful!
        about to raise exception
errback
we got an exception: Traceback (most recent call last):
--- <exception caught here> ---
  File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
326, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "./deferred_ex1.py", line 32, in failAtHandlingResult
    raise RuntimeError, "whoops! we encountered an error"
exceptions.RuntimeError: whoops! we encountered an error

If your callback raises an exception, the next method to be called will be the next errback in your chain.

Exceptions will only be handled by errbacks

If a callback raises an exception the next method to be called will be next errback in the chain. If the chain is started off with a failure, the first method to be called will be the first errback.

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
this example shows an important concept that many deferred newbies
(myself included) have trouble understanding.

when an error occurs in a callback, the first errback after the error
occurs will be the next method called. (in the next example we'll
see what happens in the 'chain' after an errback)

"""

class Counter(object):
    num = 0

def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"



def behindTheScenes(result):
    # equivalent to d.callback(result)

    # now, let's make the error happen in the first callback

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = failAtHandlingResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    # note: this callback will be skipped because
    # result is a failure

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()



def deferredExample():
    d = defer.Deferred()
    d.addCallback(failAtHandlingResult)
    d.addCallback(handleResult)
    d.addErrback(handleFailure)

    d.callback("success")


if __name__ == '__main__':
    behindTheScenes("success")
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample()
Source listing - listings/deferred/deferred_ex2.py
callback 1
        got result: success
        about to raise exception
errback
we got an exception: Traceback (most recent call last):
  File "./deferred_ex2.py", line 85, in ?
    nonDeferredExample("success")
--- <exception caught here> ---
  File "./deferred_ex2.py", line 46, in nonDeferredExample
    result = failAtHandlingResult(result)
  File "./deferred_ex2.py", line 35, in failAtHandlingResult
    raise RuntimeError, "whoops! we encountered an error"
exceptions.RuntimeError: whoops! we encountered an error

You can see that our second callback, handleResult was not called because failAtHandlingResult raised an exception

Handling an exception and continuing on

In this example, we see an errback handle an exception raised in the preceeding callback. Take note that it could just as easily been an exception from any other preceeding method. You'll see that after the exception is handled in the errback (i.e. the errback does not return a failure or raise an exception) the chain continues on with the next callback.

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
now we see how an errback can handle errors. if an errback
does not raise an exception, the next callback in the chain
will be called

"""

class Counter(object):
    num = 0


def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)
    return "okay, continue on"

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"

def callbackAfterErrback(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)



def behindTheScenes(result):
    # equivalent to d.callback(result)

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = failAtHandlingResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = callbackAfterErrback(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass



def deferredExample():
    d = defer.Deferred()
    d.addCallback(handleResult)
    d.addCallback(failAtHandlingResult)
    d.addErrback(handleFailure)
    d.addCallback(callbackAfterErrback)

    d.callback("success")


if __name__ == '__main__':
    behindTheScenes("success")
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample()
Source listing - listings/deferred/deferred_ex3.py
callback 1
        got result: success
callback 2
        got result: yay! handleResult was successful!
        about to raise exception
errback
we got an exception: Traceback (most recent call last):
  File "./deferred_ex3.py", line 97, in <module>
    deferredExample()
  File "./deferred_ex3.py", line 90, in deferredExample
    d.callback("success")
  File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line 243, in callback
    self._startRunCallbacks(result)
  File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line 312, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line 328, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "./deferred_ex3.py", line 34, in failAtHandlingResult
    raise RuntimeError, "whoops! we encountered an error"
exceptions.RuntimeError: whoops! we encountered an error

callback 3
        got result: okay, continue on

addBoth: the deferred version of finally

Now we see how deferreds do finally, with .addBoth. The callback that gets added as addBoth will be called if the result is a failure or non-failure. We'll also see in this example, that our doThisNoMatterWhat() method follows a common idiom in deferred callbacks by acting as a passthru, returning the value that it received to allow processing the chain to continue, but appearing transparent in terms of the result.

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
now we'll see what happens when you use 'addBoth'

"""

class Counter(object):
    num = 0


def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)
    return "okay, continue on"

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"

def doThisNoMatterWhat(arg):
    Counter.num += 1
    print "both %s" % (Counter.num,)
    print "\tgot argument %r" % (arg,)
    print "\tdoing something very important"
    # we pass the argument we received to the next phase here
    return arg



def behindTheScenes(result):
    # equivalent to d.callback(result)

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = failAtHandlingResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    # ---- this is equivalent to addBoth(doThisNoMatterWhat)

    if not isinstance(result, failure.Failure):
        try:
            result = doThisNoMatterWhat(result)
        except:
            result = failure.Failure()
    else:
        try:
            result = doThisNoMatterWhat(result)
        except:
            result = failure.Failure()


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()


def deferredExample():
    d = defer.Deferred()
    d.addCallback(handleResult)
    d.addCallback(failAtHandlingResult)
    d.addBoth(doThisNoMatterWhat)
    d.addErrback(handleFailure)

    d.callback("success")


if __name__ == '__main__':
    behindTheScenes("success")
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample()
Source listing - listings/deferred/deferred_ex4.py
callback 1
        got result: success
callback 2
        got result: yay! handleResult was successful!
        about to raise exception
both 3
        got argument <twisted.python.failure.Failure exceptions.RuntimeError>
        doing something very important
errback
we got an exception: Traceback (most recent call last):
--- <exception caught here> ---
  File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
326, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "./deferred_ex4.py", line 32, in failAtHandlingResult
    raise RuntimeError, "whoops! we encountered an error"
exceptions.RuntimeError: whoops! we encountered an error

You can see that the errback is called, (and consequently, the failure is trapped). This is because doThisNoMatterWhat method returned the value it received, a failure.

addCallbacks: decision making based on previous success or failure

As we've been seeing in the examples, the callback is a pair of callback/errback. Using addCallback or addErrback is actually a special case where one of the pair is a pass statement. If you want to make a decision based on whether or not the previous result in the chain was a failure or not (which is very rare, but included here for completeness), you use addCallbacks. Note that this is not the same thing as an addCallback followed by an addErrback.

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
now comes the more nuanced addCallbacks, which allows us to make a
yes/no (branching) decision based on whether the result at a given point is
a failure or not.

"""

class Counter(object):
    num = 0


def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)
    return "okay, continue on"

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"

def yesDecision(result):
    Counter.num += 1
    print "yes decision %s" % (Counter.num,)
    print "\twasn't a failure, so we can plow ahead"
    return "go ahead!"

def noDecision(result):
    Counter.num += 1
    result.trap(RuntimeError)
    print "no decision %s" % (Counter.num,)
    print "\t*doh*! a failure! quick! damage control!"
    return "damage control successful!"



def behindTheScenes(result):

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = failAtHandlingResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    # this is equivalent to addCallbacks(yesDecision, noDecision)

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = yesDecision(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        try:
            result = noDecision(result)
        except:
            result = failure.Failure()


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    # this is equivalent to addCallbacks(yesDecision, noDecision)

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = yesDecision(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        try:
            result = noDecision(result)
        except:
            result = failure.Failure()


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()


def deferredExample():
    d = defer.Deferred()
    d.addCallback(failAtHandlingResult)
    d.addCallbacks(yesDecision, noDecision) # noDecision will be called
    d.addCallback(handleResult) # - A -
    d.addCallbacks(yesDecision, noDecision) # yesDecision will be called
    d.addCallback(handleResult)
    d.addErrback(handleFailure)

    d.callback("success")


if __name__ == '__main__':
    behindTheScenes("success")
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample()
Source listing - listings/deferred/deferred_ex5.py
callback 1
        got result: success
        about to raise exception
no decision 2
        *doh*! a failure! quick! damage control!
callback 3
        got result: damage control successful!
yes decision 4
        wasn't a failure, so we can plow ahead
callback 5
        got result: go ahead!

Notice that our errback is never called. The noDecision method returns a non-failure so processing continues with the next callback. If we wanted to skip the callback at "- A -" because of the error, but do some kind of processing in response to the error, we would have used a passthru, and returned the failure we received, as we see in this next example: 

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
now comes the more nuanced addCallbacks, which allows us to make a
yes/no (branching) decision based on whether the result at a given point is
a failure or not.

here, we return the failure from noDecisionPassthru, the errback argument to
the first addCallbacks method invocation, and see what happens

"""

class Counter(object):
    num = 0


def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)
    return "okay, continue on"

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"

def yesDecision(result):
    Counter.num += 1
    print "yes decision %s" % (Counter.num,)
    print "\twasn't a failure, so we can plow ahead"
    return "go ahead!"

def noDecision(result):
    Counter.num += 1
    result.trap(RuntimeError)
    print "no decision %s" % (Counter.num,)
    print "\t*doh*! a failure! quick! damage control!"
    return "damage control successful!"

def noDecisionPassthru(result):
    Counter.num += 1
    print "no decision %s" % (Counter.num,)
    print "\t*doh*! a failure! don't know what to do, returning failure!"
    return result


def behindTheScenes(result):

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = failAtHandlingResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    # this is equivalent to addCallbacks(yesDecision, noDecision)

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = yesDecision(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        try:
            result = noDecisionPassthru(result)
        except:
            result = failure.Failure()


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    # this is equivalent to addCallbacks(yesDecision, noDecision)

    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = yesDecision(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        try:
            result = noDecision(result)
        except:
            result = failure.Failure()


    if not isinstance(result, failure.Failure): # ---- callback
        try:
            result = handleResult(result)
        except:
            result = failure.Failure()
    else:                                       # ---- errback
        pass


    if not isinstance(result, failure.Failure): # ---- callback
        pass
    else:                                       # ---- errback
        try:
            result = handleFailure(result)
        except:
            result = failure.Failure()


def deferredExample():
    d = defer.Deferred()
    d.addCallback(failAtHandlingResult)

    # noDecisionPassthru will be called
    d.addCallbacks(yesDecision, noDecisionPassthru)
    d.addCallback(handleResult) # - A -

    # noDecision will be called
    d.addCallbacks(yesDecision, noDecision)
    d.addCallback(handleResult) # - B -
    d.addErrback(handleFailure)

    d.callback("success")


if __name__ == '__main__':
    behindTheScenes("success")
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample()
Source listing - listings/deferred/deferred_ex6.py
callback 1
        got result: success
        about to raise exception
no decision 2
        *doh*! a failure! don't know what to do, returning failure!
no decision 3
        *doh*! a failure! quick! damage control!
callback 4
        got result: damage control successful!

Two things to note here. First, "- A -" was skipped, like we wanted it to, and the second thing is that after "- A -", noDecision is called, because it is the next errback that exists in the chain. It returns a non-failure, so processing continues with the next callback at "- B -", and the errback at the end of the chain is never called

Hints, tips, common mistakes, and miscellaney

The deferred callback chain is stateful

A deferred that has been called back will call it's addCallback and addErrback methods as appropriate in the order they are added, when they are added. So we see in the following example, deferredExample1 and deferredExample2 are equivalent. The first sets up the processing chain beforehand and then executes it, the other executes the chain as it is being constructed. This is because deferreds are stateful. 

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
The deferred callback chain is stateful, and can be executed before
or after all callbacks have been added to the chain
"""

class Counter(object):
    num = 0

def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    f.trap(RuntimeError)

def handleResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    return "yay! handleResult was successful!"

def failAtHandlingResult(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    print "\tabout to raise exception"
    raise RuntimeError, "whoops! we encountered an error"

def deferredExample1():
    # this is another common idiom, since all add* methods
    # return the deferred instance, you can just chain your
    # calls to addCallback and addErrback

    d = defer.Deferred().addCallback(failAtHandlingResult
                       ).addCallback(handleResult
                       ).addErrback(handleFailure)

    d.callback("success")

def deferredExample2():
    d = defer.Deferred()

    d.callback("success")

    d.addCallback(failAtHandlingResult)
    d.addCallback(handleResult)
    d.addErrback(handleFailure)


if __name__ == '__main__':
    deferredExample1()
    print "\n-------------------------------------------------\n"
    Counter.num = 0
    deferredExample2()
Source listing - listings/deferred/deferred_ex7.py
callback 1
        got result: success
        about to raise exception
errback
we got an exception: Traceback (most recent call last):
--- <exception caught here> ---
  File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
326, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "./deferred_ex7.py", line 35, in failAtHandlingResult
    raise RuntimeError, "whoops! we encountered an error"
exceptions.RuntimeError: whoops! we encountered an error


-------------------------------------------------

callback 1
        got result: success
        about to raise exception
errback
we got an exception: Traceback (most recent call last):
--- <exception caught here> ---
  File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line
326, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "./deferred_ex7.py", line 35, in failAtHandlingResult
    raise RuntimeError, "whoops! we encountered an error"
exceptions.RuntimeError: whoops! we encountered an error

This example also shows you the common idiom of chaining calls to addCallback and addErrback.

Don't call .callback() on deferreds you didn't create!

It is an error to reinvoke deferreds callback or errback method, therefore if you didn't create a deferred, do not under any circumstances call its callback or errback. doing so will raise an exception

Callbacks can return deferreds

If you need to call a method that returns a deferred within your callback chain, just return that deferred, and the result of the secondary deferred's processing chain will become the result that gets passed to the next callback of the primary deferreds processing chain 

#!/usr/bin/python

from twisted.internet import defer
from twisted.python import failure, util

"""
"""

class Counter(object):
    num = 0
    let = 'a'

    def incrLet(cls):
        cls.let = chr(ord(cls.let) + 1)
    incrLet = classmethod(incrLet)


def handleFailure(f):
    print "errback"
    print "we got an exception: %s" % (f.getTraceback(),)
    return f

def subCb_B(result):
    print "sub-callback %s" % (Counter.let,)
    Counter.incrLet()
    s = " beautiful!"
    print "\tadding %r to result" % (s,)
    result += s
    return result

def subCb_A(result):
    print "sub-callback %s" % (Counter.let,)
    Counter.incrLet()
    s = " are "
    print "\tadding %r to result" % (s,)
    result += s
    return result

def mainCb_1(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)
    result += " Deferreds "

    d = defer.Deferred().addCallback(subCb_A
                       ).addCallback(subCb_B)
    d.callback(result)
    return d

def mainCb_2(result):
    Counter.num += 1
    print "callback %s" % (Counter.num,)
    print "\tgot result: %s" % (result,)


def deferredExample():
    d = defer.Deferred().addCallback(mainCb_1
                       ).addCallback(mainCb_2)

    d.callback("I hope you'll agree: ")


if __name__ == '__main__':
    deferredExample()
Source listing - listings/deferred/deferred_ex8.py
callback 1
        got result: I hope you'll agree: 
sub-callback a
        adding ' are ' to result
sub-callback b
        adding ' beautiful!' to result
callback 2
        got result: I hope you'll agree:  Deferreds  are  beautiful!

Conclusion

Deferreds can be confusing, but only because they're so elegant and simple. There is a lot of logical power that can expressed with a deferred's processing chain, and once you see what's going on behind the curtain, it's a lot easier to understand how to make use of what deferreds have to offer.

Index

Version: 8.2.0