[Twisted-Python] Using sqlalchemy in twisted.

[Peter Cai]

I'm not quite sure, but I think I'm pretty careful of sharing objects
between threads.
1st, I only cached as few as possible orm objects.  I tried to detach them,
but I found that if I detach them,  I can't access any of their fields any
more.

2nd, I create new orm objects based on client request, pass them to class
Database and then merge them to scoped sessions, change, commit and then
discard these objects.

3rd, I switch to sqlite frequently to check if there is any database
operation outside Database, because sqlite doesn't allow multi-thread
access.

Actually it seems to work until 2 or 3 days ago suddenly cases hang the
server.

Ah, as I've already written lots of code in ORM, I think maybe I should try
to change Database to use a dedicated thread to handle all database
operations.

That might be a bottle neck of my application, but I really can't give up
orm as these mapper classes are used everywhere in my application.

On Thu, Mar 5, 2009 at 3:04 AM, Valentino Volonghi <dialtone at gmail.com>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On Mar 4, 2009, at 3:28 AM, Peter Cai wrote:
>
>  The code doesn't work.   When I limit the thread numbers to 1
>>
>>   reactor.suggestThreadPoolSize(1)
>>
>> Everything goes fine.  Other wise the server would be blocked and must
>> be killed by "kill 9 ...".
>>
>> The result conflicts with my understanding of sqlalchemy.  Since I
>> don't share any object between threads, there should be no problem!
>>
>
> I'm pretty sure you can't say this with full certainty and actually you are
> just wrong based on the code you showed. deferToThread doesn't use
> the same thread every time you call it, there's absolutely no guarantee
> in that and sqlalchemy keeps state around in that thread related to that
> object it returned. If you want to do any operations you need either to
> detach the object from the session before returning it and do any
> modification
> on the same object in another thread after reattaching it (pretty
> cumbersome).
>
> Or write your own threadpool that gives names to threads so that you can
> have a guarantee that you always call the same thread when working with
> a bunch of objects.
>
> Nonetheless though sqlalchemy is a huge project and I'm pretty sure it has
> some deadlocks and race conditions around which means that even taking
> care of these issues you'll have other problems when dealing with the orm.
>
> If you want to use sqlalchemy don't use its orm but just the query builder,
> it's the only sane way to integrate with twisted. Which anyway IMHO is the
> best way to use it anyway because it uses a lot less memory since it
> doesn't
> have to always cache objects because you control everything and can make
> that call when you really need it.
>
>  Ah....  It always have risk to use something you haven't tried
>> before ....
>> I think I have no choice but always set thread pool size to 1 ...
>>
>
>
> Not entirely true.
>
> - --
> Valentino Volonghi aka Dialtone
> Now running MacOS X 10.5
> Home Page: http://www.twisted.it
> http://www.adroll.com
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Darwin)
>
> iEYEARECAAYFAkmu0NoACgkQ9Llz28widGXBawCg32svBsLqsIRLzvzOThgR4sA0
> 5UkAoIgNfyUDPl9c0nwSud0sem3aKjz5
> =2XIX
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
look to the things around you,the immediate world around you, if you are
alive,it will mean something to you ——Paul Strand
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090305/9a952ddc/attachment.html>