[Twisted-Python] Help w/ low adbapi connection pool limits?

[Steve Freitas]

Hi all, I'm new to Twisted, and it's about to show. :-) I hope you can help 
me.

I'm writing a PostgreSQL-backed XML-RPC server with Twisted 1.0.6 on Linux. My 
problem is that if I hammer it with repeated, high-speed queries from a 
client, the client receives a Fault 8002. I'm probably just not setting some 
limit variable somewhere high enough. Here's my server code:

from twisted.web import xmlrpc, server
from twisted.enterprise import adbapi
class takeOne(xmlrpc.XMLRPC):
    def xmlrpc_dbtest(self):
        dbpool = adbapi.ConnectionPool("pyPgSQL.PgSQL", host='127.0.0.1', \
		user='postgres', database='template1')
	dbpool.connect()
        d = adbapi.Augmentation(dbpool)
        result = d.runQuery("select * from auth where uname='foo'")
	dbpool.close()
        return result

(In an earlier iteration I'd omitted the "dbpool.close()" call. Putting that 
in made it work considerably better, but it still fails under heavy load.)

Individual queries from the interactive interpreter work fine:

[steve at lothar]$ python
>>> import xmlrpclib
>>> s = xmlrpclib.Server('http://localhost:7080/')
>>> s.dbtest()
[{'baseObj': ['foo', 'd00d']}]

But when I hit it hard for a simple load test, it goes for a few seconds and 
then falls over:

>>> while i < 500:
...     x = s.dbtest()
...     i = i + 1
...
Traceback (most recent call last):
  File "<stdin>", line 2, in ?
  File "/usr/lib/python2.2/xmlrpclib.py", line 821, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.2/xmlrpclib.py", line 975, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.2/xmlrpclib.py", line 853, in request
    return self.parse_response(h.getfile())
  File "/usr/lib/python2.2/xmlrpclib.py", line 896, in parse_response
    return u.close()
  File "/usr/lib/python2.2/xmlrpclib.py", line 571, in close
    raise apply(Fault, (), self._stack[0])
xmlrpclib.Fault: <Fault 8002: 'error'>

Afterwards the connection pool seems to still be available:

>>> s.dbtest()
[{'baseObj': ['foo', 'd00d']}]

But I've got lots of connections in process still:

[steve at lothar]$ netstat -atn
tcp        0      0 127.0.0.1:5432          127.0.0.1:48841         TIME_WAIT 
tcp        0      0 127.0.0.1:7080          127.0.0.1:48839         TIME_WAIT
(repeat these guys about 150 times each, plus several being serviced)

The connections eventually go away as (I presume) they get serviced.

So I expect I'm just hitting a database connection pooling limitation, as a 
similar test against an XML-RPC function which didn't access a database ran 
just fine. I'm not sure if the limitation is in Twisted, or in PyPgSQL, and I 
don't know enough about how to enable traceback logging in Twisted (is there 
any?) to tell. Ideally I'd like it to handle several thousand backlogged 
requests without burping.

Help! :-)

Thanks for your time,

Steve Freitas