[Twisted-Python] Synchronization techniques
Paul G
paul-lists at perforge.com
Wed Apr 4 11:34:32 MDT 2007
Daniel Miller wrote:
-- snip --
> So anyway, I rewrote my server-side library to do it the twisted way
> and return deferred's instead trying rig up some way of waiting for
> them. I still think it would be super useful to be able to pseudo-
> block on a deferred (i.e. allow the reactor to process other events
> while waiting for the deferred). It is very annoying to have to
> rewrite many layers of code when twisted is introduced into a
> program. I did find gthreadless.py, and maybe that would do it.
> Unfortunately discussion on that seems to have been dropped some time
> ago...
you can still use that method *if* you're willing to patch your cpython with
the stackless changes and live with the requirement that this be done on all
cpythons your code is going to run on. short of that, or using evil bytecode
hacks, it's not really possible to implement - you need the ability to save,
switch and restore stacks plus call into the middle of a function. while
this can be done with generators, it is virtually equivalent to splitting
your functions into a top/bottom half and is not directly supported, so it
ends up looking ugly. if pypy ever gets to the point where it's usable in
production (has the required c extensions ported and runs close in speed to
cpython on non-trivial code), syntactic support for such usage could be
added. please note that if you hadn't wanted to take advantage of being in
an event loop, you could have easily stuck your code - unchanged - into a
thread pool (via apis helpfully provided by twisted) and just written a few
lines of code to adapt the interfaces.
this has little to do with twisted, which - imo - does a wonderful job of
making the (for some, abeit not me personally) difficult execution flow
model palatable and clear. twisted might be a very good framework, but it
isn't an async fairy that sneaks into your room at night and - if you've
been good - rewrites your code to be all twisty and eventful and leaves a
quarter under your pillow.
with all that said, the sooner you start thinking of a program as a series
of events being emitted and handled, the better for you and your twisted
code - everything will become very natural at that point instead of looking
like an idiosyncracy. while there are a few languages, like erlang for
example, that express event loops with cooperative threading syntax
(processes, messages and futures), i'm of the opinion that it's actually
harmful to hide the event loop details from people who haven't started
thinking that way yet - they should be simply syntactic sugar for when
you've already grokked the model. it's inherently impossible to write good
concurrent code without this insight, including preemptiver multithreading
code as well. writing preemption and smp safe mt code boils down to
understanding the chain of processing each 'event', where such event is
access to shared data, goes through and the interaction between two or more
such events descending down this chain - this is the only effective way i've
found to think about memory barriers and cache coherency for example. just
like grokking functional programming, grokking this will make you (the
impersonal pronoun version) a better programmer (fwiw, it's made functional
programming easier to get for me personally).
cheers,
-p
More information about the Twisted-Python
mailing list