[Twisted-Python] Synchronous Code Fishbowl

[Ed Suominen]

The dialogue copied below shows yet another example of the frustrations
that arise when trying to bridge the chasm between Twisted and the
synchronous rest of the world.

This latest episode inspired the idea of a synchronous code fishbowl
that offers well-behaved Twisted code a deferToQueue() method for
running badly-behaved blocking code.

See http://foss.eepatents.com/sAsync/browser/branches/syncbridge.py

Comments?

- Ed Suominen


-------- Original Message --------
Michael Bayer wrote:
> why are you using twisted *with* threads ?  didnt we all agree that was
> sort of unnecessary ?

Twisted operates without needing to run *in* a thread by doing
everything asynchronously. Every call made via the Twisted event loop
must either return (synchronous) results very quickly or return an
immediate reference to an eventual result (thus operating
asynchronously). The objects that hold those immediate references are
called "deferreds" in TWisted parlance.

Twisted runs most all of its internal operations with appropriate
chunking and use of select() to keep the asynchronous event loop humming
along nicely, but it has no control over how other libraries do things.
When an external library like SQLAlchemy presents a blocking call like
select(...).execute.fetchall(), the only way to make that call "play
nice" with Twisted by immediately returning a deferred to the eventual
result is by having TWisted run it in a thread. It includes the
deferToThread() function for that express purpose.

> (this goes to my point that twisted is a pain in the butt...)

Perhaps, but asynchronous code is a whole different way of thinking that
some of us actually find cleaner and more intuitive than blocking +
threads. What I'm trying to do with sAsync is put the "pain in the butt"
associated with making synchronous and asynchronous code work together
firmly behind the scenes for Twisted users, at least as far as
interacting with SQLAlchemy is concerned.

> anyway, SA 0.2 uses the same idea for sqlite threading as 0.1....it
> maintains each connection thread locally since you cant share a sqlite
> connection between threads.   im also not sure what youre talking about
> with a "thread local copy of the engine", if thats something youre doing
> on your end, you might want to not do that.  use Connection objects
> instead theyre much more portable.

Mike, I remember you telling me a while ago that SA somehow figures out
how to make things work with the engine. That was after I spent a lot of
time trying to re-engineer the wheel, to great frustration on my part. I
followed your advice, ditched my thread-local code, and got SA
transactions just fine (based on the engine) to run in arbitrary
threads. Now it's not working in SA 0.2. Could you point out what's
changed in regard to the engine vs. threads vs. sqlite?

I'd rather not change everything under the hood (e.g., using connections
somehow) if I don't have to. The lesson of not trying to re-engineer
what SA handled fine on its own is still with me. :-)

Best regards, Ed

> On May 27, 2006, at 6:57 PM, Ed Suominen wrote:
> 
>> My sAsync project relies on Twisted to maintain a thread pool. It runs
>> database transactions in some available thread via Twisted's
>> deferToThread() function. See the transact function in
>> http://foss.eepatents.com/sAsync/browser/trunk/sasync/database.py for
>> details.
>>
>> That arrangement created no problem with SQLite and SA 0.1, even though
>> SQLite does not allow sharing of connections between threads. Somehow,
>> SA 0.1 made the transaction run in its thread with a thread-local copy
>> of the engine and table that was created in the main thread. I never
>> figured out how, but it just worked.
>>
>> It doesn't work in SA 0.2, however. I either get the error when trying
>> to use the main-thread table in the threaded transaction, or have no
>> table available in the thread because it wasn't created there. Now I'm
>> trying to figure out how to put humpty dumpty back together again. Any
>> suggestions?
>>
>> Best regards, Ed