[Twisted-Python] Synchronization techniques

[Brian Granger]

> >With all that said, I have encountered a few highly unusual cases
> >where I really did want blockOn to exist.  These cases had the
> >characteristic that they couldn't be done any other way in Twisted.
> >The answer in this case is to ditch twisted and use a tool that is
> >better suited to the problem.  But in my experience these cases only
> >pop up about 0.00001% of the time.
>
> I am very curious about your 0.00001% case.  Not that I don't believe such
> cases exist, but in every case but one (twisted ticket #2545) the issue has
> actually been a documentation problem with Twisted, where it wasn't clear
> how to do something the "normal" way with Deferreds and such.  I'd like to
> know if there is another such doc bug we should be filing :).

The cases I am thinking about is not an example of a doc bug.  The
most relevant one is related to using Twisted in an interactive python
(really IPython most of the time) session.  There are two difficulties
we keep running into:

1.  The interactive python interpreter is a completely synchronous
universe - getting the reactor running in this context seems like a
hack.  The only way I have seen this done is by running the reactor in
a different thread.  The problem with this is that it is inevitable
that you end up wanting to do things with Deferreds in the main thread
where user code is running.  But, as I understand it, Twisted is not
thread safe, so at that point, you are playing with (threaded) fire.

2.  Users expect certain things in an interactive python session that
don't mesh well with Twisted and the asynchronous universe:

>>> psi = computeWavefunctionForHydrogen()
>>> psi.getEnergy(1)
-13.6
# here the user looks at the energy (a human if statement) and decides
if they actually want to
# make the following plot.  If the answer were not -13.6, they would
not make the plot.
>>> plot(psi.getState(1))

Even if you could get the reactor running in an interactive python
session it would be crazy to have to write something like (in an
interactive session):

>>> d = computeWavefunctionForHydrogen()
>>> def printAndPlot(psi, n):
>>>     print psi.getEnergy(n)
>>>     if abs(psi.getEnergy(n) - (psi.getEnergy(n)) < 1.0e-4:
>>>         plot(psi.getState(n))
>>> d.addCallback(printAndPlot, 1)

The problem we are having is that we would like to use Twisted network
clients in functions like computeWavefunctionForHydrogen.  But we
simply can't as there is no way of returning the actual result to the
user.  I can't emphasize enough that end users of such code
(scientists) "just want the damn result" (not a deferred) and are
willing to wait for it.  Thus in classes/functions that need to block
for an actual result, we don't use twisted - we use blocking sockets
instead.

In blockOn existed, the implementation of
computeWavefunctionForHydrogen could look like:

def computeWavefunctionForHydrogen():
    d = doRemoteComputationOnServerUsingTwisted()
    return blockOn(d)

The important point is that doRemoteComputationOnServerUsingTwisted is
the only place where anything involving the network is happening in
this process.  This type of code occurs mainly in clients.  In server
code you are usually also listening on sockets, so there could be
other asyncronous events that occur.  But in interactive client code,
the network event are often very contained and isolated.

One thing we have done in our blocking client code is to create
something we call a PendingResult object.  It is basically a fully
synchronous version of a deferred that allows a user to  poll for or
block on a result that is being computed elsewhere.  It is designed
for interactive usage, where only blocking sockets are used.

Brian