[Twisted-Python] Simplifying database access in Twisted

[Glyph Lefkowitz]

On Thursday, June 19, 2003, at 04:13 PM, Thomas Weholt wrote:

> Is there no other way to do simple database access in Twisted? 
> Something
> more like the DB-API 2.0?

Feel free to just use DB-API modules directly, if that's appropriate.  
twisted.enterprise exists to facilitate non-blocking access; if your 
database queries are quick, you can block your server.  The API 
certainly is a bit grotty from disuse; a few cleanups and some better 
documentation would probably be better.

Speaking of being grotty - Bob's examples were correct, but there is a 
more natural API that you can use.  (The methods he suggested using 
pre-date Deferreds.)  Assuming you want to run a bunch of code which 
munges some data from a database, you can write something like this:

--

def myInteraction(t):
     t.execute('select * from my_numbers')
     # need some arbitrary code, so I'll sum these in memory rather than 
db
     sum = 0
     while True:
         row = t.fetchone()
         if not row:
             break
         num = row[0]
         sum += num
     return sum

def printx(x):
     print x

cp.runInteraction(myInteraction).addCallback(printx)

--

Assuming that your 'interaction' method doesn't touch any 
non-thread-safe state, this will work for pretty much any massaging you 
want to do to your data.  Since it's a Deferred, you can return it to 
various places in Twisted that can already manage waiting on deferreds 
for you, including woven.

runInteraction is probably the only method you really need, and it has 
the added benefit of allowing you to combine reads and writes in a 
single function, and rolling back any changes if there is a bug in your 
Python code.

> PS! I don't mean to step on anyones toes or anything, so forgive me if 
> I'm a
> bit harsh. But this might be the bitter end of my journey into the 
> realms of
> Twisted. Quick, fast and easy access to a database will make-or-break 
> all of
> my current projects, all of which are based on Twisted.

I can understand your frustration (twisted.enterprise is crap, 
especially the docs), but this is a bit much.

If you feel you're getting to this point with any bit of Twisted, 
remember that it's all just python code.  Twisted provides these 
components together in order to ease deployment and provide as complete 
an environment as possible, not to ram them down your throat.  There 
are many parts of Twisted which could use improvement, and we've 
attempted to do quite a few things that we are not expert at.

Certainly implementing your own database threadpool on top of 
reactor.callFromThread/callInThread or making database calls 
synchronously is easier than throwing out all of Twisted and starting 
over.