[Twisted-Python] Flow, generators, coroutines etc.

[Michael]

On Sunday 28 May 2006 13:39, Phil Mayers wrote:
...
> All that said, the BBC Kamaelia project uses a generator-based
> consumer/producer pipeline as its underlying primitive, and seems to get
> along fine with it. But a consumer/producer component is not the same as
> a lightweight thread of course - broadly a c/p will only ever interact
> with its input, local variables, output and library calls. A thread
> might be expected to interact with other threads and shared data, from
> whence all difficulty springs.

</delurk>

[ Since someone else raised us, I'd like to say a couple of things, since I
  think one is related to the conversation in hand, and the other is relevant
  to a conversation started at Europython last year about our work feeding
  into Twisted in a good way (ie a way useful to people using Twisted). ]

Hopefully both points are useful.

Given we're now churning out systems at work with Kamaelia which are getting
/close/ to what I'd be happy with as production systems, so it //seems// to
be working out OK so far as you say. One thing that's less obvious about
the term consumer producer (even though its correct) is people tend to
think pipelines, and many systems don't tend to be (we've invented the
term graphlines instead, for want of a better name).

As part of getting to that stage - getting back to why I was replying - as
well as pure generators, we also properly support threaded components now
(albeit with a couple of caveats). The only difference between those and
standard components is the baseclass you inherit from and the fact your
main method doesn't have a yield statement in it. Threaded components are
also useful for forcing true system quiescence. This also gives a fairly
simple migration route for code for integration.

This also means bridging sync and async worlds is pretty trivial - for example
a crappy interactive web-tester application can look like this: (would be
nicer to use sys.argv...)

pipeline(
   ConsoleReader(),                 # This uses raw_input in a thread.
   TCPClient("www.google.com", 80), # Async and uses a selector component
                                    # running in another thread so the
                                    # system can be quiescent
   ConsoleEchoer(),                 # Normal Async
).run()

(The reason TCPClient fires up a selector component is because it expects to
be in a situation where lots of networking stuff is happening. If it's not the
CPU/memory load addition is minimal)

There's more useful examples than the above, but I quite like the above
because of simplicity. (The system we've got for dumping DVB-T transport
streams, and transcoding them is much more fun, but longer winded :)

On the second point...

I think it's worth mentioning that our goal of trying to make it possible
to integrate our code with twisted by Europython this year was woefully
optimistic (we've been spending time on optimisations instead, which will
probably actually simplify integrating with twisted). (Integrating as in
"work well with", not as in "viewed as a good idea by everyone involved
with twisted :)" ) [[ Sorry Tommi, I know there were aspects you wanted to 
nick... :-( ]]

<relurk>

Best Regards,


Michael.
--
Michael Sparks, Senior Research Engineer, BBC Research, Technology Group
michael.sparks at rd.bbc.co.uk, Kamaelia Project Lead, http://kamaelia.sf.net/