[Twisted-Python] Synchronization techniques

[glyph at divmod.com]

On 05:25 pm, ellisonbg.net at gmail.com wrote:
>The issue brought up in this thread is one that we (the IPython dev
>team) has thought about a lot.  There are really two questions about
>having true synchronization primitives (something like d.waitFor() or
>blockOn(d)):
>
>1.  Can they be implemented in python/twisted in a reliable and robust 
>way?

The real question here doesn't involve the words "in python/twisted" :).
>2.  Should you use them in your code?

>Unfortunately, the answer to (1) seems to be no - at least not without
>completely refactoring the core of twisted to support this capability
>from the ground level.  The tempting way of doing it currently is to
>call reactor.iterate() to spin the event loop at a particular point.
>We have code in IPython that does this:
>
>http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/blockon.py
>[DON'T USE THIS CODE!]
>
>For very simple things this code works just fine.

I think you mean, "in some cases this code appears to work".  Working 
"just fine" implies that it is robust and supported.  This is an 
actively deprecated programming style and there are numerous known 
problems (besides the ones you've already documented here) with doing 
it.
>d = functionThatReturnsADeferred()
>r = blockOn(d)          # spin the reactor until d fires.
>
>The problem is that if you start to use blockOn in various places in
>your code, a single call to blockOn (which calls reactor.iterate())
>could trigger other calls to blockOn (which will attempt to call
>reactor.iterate() again).  The twisted reactor is simply not designed
>to be doubly iterated like this - it leads to all sorts of really
>weird problems that are _impossible_ to track down.

These "weird" problems are the entirely predictable result of violating 
every assumption that code makes about its run-time environment when it 
is written and tested.  An analogous operation would be to write C code 
to forcibly delete Python objects rather than go through the garbage 
collector because you *really know*, *just this one time*, that you want 
to free that memory.  If you're careful to never touch that object 
again, you might be able to avoid a segfault, but I think most people 
would agree that all bets are off at that point.

This might seem like an exaggerated problem, but I have actually seen 
code like that more than once written by dyed-in-the-wool C programmers 
who didn't "get" how Python's object model worked.  It's the same with 
dyed-in-the-wool non-concurrent programmers approaching concurrent 
systems for the first time.
>This is why people are saying "it can't be done."  I should mention
>that it might be possible to implement these things in stackless.

If you were to implement these things in stackless, you would still have 
to deal with the inherently problematic issue of apparently "sequential" 
code being run recursively when it does not expect to be.  In order to 
prevent this, you would likely have a completely different programming 
model where something properly event-driven, like Twisted itself, were 
scheduling "user code" which was written using a different programming 
style.

For certain problems such things are a good approach.  For example, in 
AI code with extremely deeply nested ad-hoc decision trees modeled as if 
statements and for loops, the cost of stack ripping becomes high both 
conceptually and performance-wise, and it is more natural to model 
individual agents as individual control flows (or "cooperative 
threads").  This sort of code, though, would be written in a style more 
like Erlang, with almost no shared state at all.  Part of Twisted's 
appeal is that it makes mutable state-sharing between disparate systems 
straightforward.  In other words, it is a different programming model 
for a different set of problems that would require a different pile of 
infrastructure.

It may well be possible to implement such a layer on top of Twisted, but 
there is a curious thing that takes place when people begin to tackle 
this problem.  Pretty much everyone eventually comes to the realization 
that this isn't a good idea for their problem domain, and what they 
*actually* want is to wish away the difficulties associated with 
concurrency and pretend that they can "just block" and everything will 
be OK.  The ones who really, really need it (like people dealing with 
the aforementioned AI problems) already know their requirements and 
quietly go ahead and implement what they need, without any hand-wringing 
about how hard programming with Deferreds is or how they'd really like 
to block on one.
>So what about (2)?  Ignoring the fact that such constructs can't be
>implemented reliably (let's imagine they could be), should you want to
>use them?  I think the answer is this:

>The design of Twisted reflects the realities of an asynchronous, event
>driven world where things can and do go wrong when you least expect
>it.  The error handling decision tree of Deferreds are a reflection of
>this reality.  If you try to make this stuff go away (we have tried ma
>ny times - we are slow learners and very stubborn) you will be
>punished and there will be "weeping and gnashing of teeth."  This
>punishment will take the form of buggy code that is difficult to
>maintain and extend.

It sounds like we broadly agree here :).
>With all that said, I have encountered a few highly unusual cases
>where I really did want blockOn to exist.  These cases had the
>characteristic that they couldn't be done any other way in Twisted.
>The answer in this case is to ditch twisted and use a tool that is
>better suited to the problem.  But in my experience these cases only
>pop up about 0.00001% of the time.

I am very curious about your 0.00001% case.  Not that I don't believe 
such cases exist, but in every case but one (twisted ticket #2545) the 
issue has actually been a documentation problem with Twisted, where it 
wasn't clear how to do something the "normal" way with Deferreds and 
such.  I'd like to know if there is another such doc bug we should be 
filing :).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20070404/de501080/attachment.html>