[Twisted-Python] Breaking up long computations

[Nicola Larosa]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I think I drank enough kool-aid to see that "Deferred are not magical things
that turn long code into short", so I'm trying to break long, cyclical
computations into Twisted-compliant pieces.

Let's say I have a matrix and want to make some computation on its elements
(this does not involve Numeric or numarray, by the way, it's just an example):


data = ((1, 1, 1),
        (2, 2, 2),
        (3, 3, 3))

sqrSum = 0
for row in data:
    for num in row:
        sqrSum += num*num
print sqrSum


To be able to make one operation per event, I break the loops using iter()
and reactor.callLater(), and end up with this, actually working, script:


from twisted.internet import reactor

def computeRow(sqrSum, rowIter, dataIter):
    try:
        num = rowIter.next()
    except StopIteration:
        reactor.callLater(0, computeData, sqrSum, dataIter)
    else:
        sqrSum += num*num
        reactor.callLater(0, computeRow, sqrSum, rowIter, dataIter)

def computeData(sqrSum, dataIter):
    try:
        row = dataIter.next()
    except StopIteration:
        print sqrSum
        reactor.stop()
    else:
        rowIter = iter(row)
        reactor.callLater(0, computeRow, sqrSum, rowIter, dataIter)

data = ((1, 1, 1),
        (2, 2, 2),
        (3, 3, 3))
sqrSum = 0
dataIter = iter(data)
computeData(sqrSum, dataIter)
reactor.run()


(In both cases, the answer is 42. :^) )

Apparently, this allows Twisted to process any other events that may happen
in the middle of the computation. But I wonder, since I'm not using threads,
and Twisted runs the reactor and my code in the same process/thread, how
could such events get inserted into the queue?

I mean, if I'm appending non-delaying events from inside the computation,
one after the other, how can anything else get a chance to get started
*before* the computation completes?

Obviously, in such a case there would be no point in this exercise in
decomposing, so I feel I'm overlooking something fundamental.


- --
"Unlike some other scripting languages, Python syntax tends to be in
words rather than typographical syntax - it's very unlikely you'll be
writing lines of code that look like comic book curse words. As a result,
when you have to come back to your code six months later, odds are you
will still be able to understand it."  Samuele Pedroni and Noel Rappin

Nicola Larosa - nico at tekNico.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/MMKTXv0hgDImBm4RAgBLAJ9WdmIDeOSPJYLtYLQB5unbuhPajACeKzhl
s2bZSXJzeSEXLrJlpMYHtZE=
=9JKY
-----END PGP SIGNATURE-----