[Twisted-Python] Using sqlalchemy in twisted.

[Valentino Volonghi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Mar 4, 2009, at 3:28 AM, Peter Cai wrote:

> The code doesn't work.   When I limit the thread numbers to 1
>
>    reactor.suggestThreadPoolSize(1)
>
> Everything goes fine.  Other wise the server would be blocked and must
> be killed by "kill 9 ...".
>
> The result conflicts with my understanding of sqlalchemy.  Since I
> don't share any object between threads, there should be no problem!

I'm pretty sure you can't say this with full certainty and actually  
you are
just wrong based on the code you showed. deferToThread doesn't use
the same thread every time you call it, there's absolutely no guarantee
in that and sqlalchemy keeps state around in that thread related to that
object it returned. If you want to do any operations you need either to
detach the object from the session before returning it and do any  
modification
on the same object in another thread after reattaching it (pretty  
cumbersome).

Or write your own threadpool that gives names to threads so that you can
have a guarantee that you always call the same thread when working with
a bunch of objects.

Nonetheless though sqlalchemy is a huge project and I'm pretty sure it  
has
some deadlocks and race conditions around which means that even taking
care of these issues you'll have other problems when dealing with the  
orm.

If you want to use sqlalchemy don't use its orm but just the query  
builder,
it's the only sane way to integrate with twisted. Which anyway IMHO is  
the
best way to use it anyway because it uses a lot less memory since it  
doesn't
have to always cache objects because you control everything and can make
that call when you really need it.

> Ah....  It always have risk to use something you haven't tried
> before ....
> I think I have no choice but always set thread pool size to 1 ...


Not entirely true.

- --
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkmu0NoACgkQ9Llz28widGXBawCg32svBsLqsIRLzvzOThgR4sA0
5UkAoIgNfyUDPl9c0nwSud0sem3aKjz5
=2XIX
-----END PGP SIGNATURE-----