[Twisted-Python] Reactors and New Asynchronous APIs

[Bob Ippolito]

On May 5, 2005, at 5:52 AM, Patrick Mézard wrote:

> 2- I am considering the integration of a custom database API (like  
> Berkeley DB)
> in a twisted application. I could easily wrap the API in a C++  
> python module and
> access it synchronously from twisted, at least in a prototyping phase.
> Obviously, if the blocking calls are too long it would make the  
> whole framework
> completely useless. Then, the calls could be made from twisted  
> managed threads.
> Assuming the module code was written to minimize GIL contention it  
> would
> probably greatly improve the situation. The most efficient solution  
> would be
> to plug the asynchronous API directly into the reactor:
>
> Is it possible to extend reactors with new asynchronous API  
> implemented in
> separate client modules?

No, that's fundamentally impossible.  Unless you have some resource  
the API you're using understands (i.e. a file descriptor), then  
there's no way you're going to be able to wake up a blocking call to  
select(), WaitForMultipleObjects, or whatever.

What you probably want to do is run your code in a separate thread,  
and notify the main thread when something happens.  (e.g.  
callInThread, etc.).  See threadedselectreactor (in svn) for an  
inversion of this idea -- which runs (the blocking part of) Twisted  
in a separate thread and notifies your foreign event loop when  
something happens.

> With my little knowledge of the twisted framework and quick glances  
> at the
> selectreactor and iocpreactor implementation, I think the answer is  
> no.
> The practical solution would be to duplicate the reactor code and  
> extend it
> with the new API calls.
>
> I do not know if it would be easy or not to provide a pluggable  
> asynchronous
> framework *at C module level*. Basically, the reactor is nothing  
> else than a
> thread-safe event queue with a (post/get)-event API. If this  
> abstraction can be
> implemented at C level, then plugging new asynchronous handlers  
> would be clearly
> doable, _iocp.c is a good example for that. Now, I am not really  
> knowledgeable
> (yet) about python C module coding, and I guess it is not exactly  
> easy to bind
> C callbacks from one python C module into another using normal  
> python code as
> glue. In the end, this is just another plugin/component system like  
> COM to
> write so...

If the reactor just polled until something happened and spun 100%  
CPU, then sure, you could integrate whatever the hell "asynchronous"  
API you wanted.  That's not how anything (should) work in a multi- 
tasking operating system.  That's not even driven.

-bob