[Twisted-Python] Re: PDF I thought may be of interest

[Andrew Francis]

Hi Michael and Colleagues:

> .. which is a paper by Rob von Behren, Jeremy Condit and
> Eric Brewer.

As a part of writing "Adventures in Stackless Python/Twisted Integration," I have read "The Why Events are" paper as well as the Ousterhout "Why Threads Are A Bad Idea (for most purposes). 

The "Why Events" paper led me to the Adya et al, "Cooperative task management without manual stack management" and the original Lauer and Needham  "On the duality of operating system structures." I also visited the Schmitt papers on the Reactor pattern. 

>From what I recalled, I preferred the latter three papers because they provide a much better analytic framework for describing the issues. In particular, the Adya provides terminology.

I have two basic sets of take-aways from the paper survey

1) "Events are great papers" state that threads are bad because of A,B, and C. "Threads are great papers" state that events are bad because of X,Y, and Z. As a programmer using these systems, perhaps I should pay special attention to issues A,B,C,X,Y and Z whenever applicable (and by the way papers thanks for telling me C and Y were issues since I didn't know). 

Perhaps out of all this, we can come up with appropriate guidelines for lockless programming?

2) The "Duality" paper argues that messaging passing and procedural systems can be described in terms of each other (bi-simulation?). I found this to be a very powerful way to look at Stackless Python and Twisted's threading models. 

A simple example. 

At the time I presented "Adventures," to avoid deadlock, I ran a reactor protocol instance in its own Stackless thread. This bugged me. It turned out that by replacing channels with deferreds (and doing a little buffering), I could replace the extraneous tasklet. My way of looking at it was, after all, protocol instances/deferreds are just another way of doing continuations... Use them, don't replace them....
 
> Looking slightly below the surface of things, it appears to
> be advocating something more like stackless's tasklets, rather than
> OS level threads (since it makes reference near the end to Erlang's >threading model).

A quote from the paper:

"To evaluate the ability of threads to support high
concurrency, we designed and implemented a simple
(5000 line) user-level cooperative threading package for
Linux. Our thread package uses the coro coroutine
library [15] for minimalist context switching, and it
translates blocking I/O requests to asynchronous requests
internally."

> The paper appears to be 5 years old already, so the impact
> of this paper appears minimal, but I thought it may be of interest to
> others. (The reason it  raised such a surprise for me is because it >would be as surprising to me as it would be to read of any of the core >twisted devs to suddenly come out in  favour of a threading approach >rather than non-threading!)

"The Why Events" is essentially the approach I took in regards to Stackless and Twisted. Right now, I am analyzing Stackless's scheduling model to see if improvements can be made. I feel I have much to learn...

That said, I feel inline callbacks are a step in this direction. As Stackless/PyPy concepts get into mainstream Python, I can see Twisted following in this direction. Already one can see that with stuff like Christopher Armstrong's Corotwine. 

Again, I think the key to incorporate threads and keep to the spirit of Twisted, simplicity and safety, is to come up with some guidelines for lockless programming.

Cheers,
Andrew