[Twisted-Python] Twisted CallLater: Advanced Usage

[Moshe Zadka]

Those of you who heard my talks, either at Europython or Strakt, probably
remember me going on about how to use reactor.callLater(0,) to partition
algorithms into bite-size chunks, that would not delay the main loop too
much. This a topic which is interesting to me. I'm still waiting for glyph
to write the ultimate "What is Blocking" treatise, but I may give up and
write something myself. In the meantime, I want to present to advanced
examples of what you can achieve by using this technique.

optFactorial is an optimized version of the factorial function from my
talk. The trick here is to do 10 steps (or whatever argument is given
to us, if the user wants to optimize further by tweaking) at a time.
I have not profiled: possibly, it only becomes worth the complexity
at a much higher step number. However, this is an important example if
you are writing algorithms. It showcases how the assignment-paradigm
and the tail-call paradigm can be combined seamlessly to produce a
more complicated, but more efficient, whole.

def optFactorial(n, steps=10, d=None, acc=1L):
    d = d or defer.Deferred()
    for i in range(steps):
        if n==1: 
            d.callback(acc)
            break
        n, acc = n-1, acc*n
    else:
        reactor.callLater(0, optFactorial, n, steps, d, acc)
    return d

os_path_walk is a reimplementation of os.path.walk, from Python's
standard library, in an asynchronous manner. Here the word "asynchronous"
is literal: while it will call the visit() function with the same arguments,
in the same order, it does not guarantee when it will do so: certainly,
it is not likely to finish its work by the time it returns. yield, explicit
and implicit .next() calls and recursion combine to make sure that we will
never delay the mainloop more than the time it takes to do one os.listdir.
Note that unlike the os.path.walk function, here an exception anywhere
terminates the loop and lets us now of the failure. 

This example, in particular, shows how frighteningly similar Twisted
can be to threads :) In effect, this is much like doing something
like

reactor.runInThread(os.path.walk,
                    path,
                    lambda arg, dirname, filenames:
                        reactor.callFromThread(visit, arg, dirname, filenames),
                    arg)

except without the overhead of thread spawning and thread communication
[mutexes, etc.] However, note that many of the thread caveats remain:
if you have other code running, this code cannot know how many of the
visit() functions will be called by the time it runs.

from __future__ import generators
def os_path_walk(path, visit, arg):
    d = defer.Deferred()
    def walk(path):
        files = os.listdir(path)
        visit(arg, path, files)
        for file in files:
            yield None
            if os.path.isdir(file):
                for _ in walk(os.path.join(path, file):
                    pass
    next = walk(path).next
    def caller():
        try:
            next()
        except StopIteration:
            d.callback(None)
        except:
            d.errback(sys.exc_info()[1])
        else:
            reactor.callLater(0, caller)
    caller()
    return d

-- 
Moshe Zadka -- http://moshez.org/
Buffy: I don't like you hanging out with someone that... short.
Riley: Yeah, a lot of young people nowadays are experimenting with shortness.
Agile Programming Language -- http://www.python.org/