[Twisted-Python] Twisted webserver performance issues

[James Y Knight]

On Nov 8, 2009, at 1:08 PM, James Y Knight wrote:

>>
>> Can we attach gdb to the process and trigger an all-threads stack
>> dump when it happens?
>
> That's a good idea. Unfortunately, currently it seems to be working
> quite smoothly. :)

The intermittent long pauses were happening again, so I got a gdb  
"thread apply all bt". I discovered that there were two threads not in  
sem_wait waiting for GIL to be released. And both were waiting in  
nanosleep. One had the GIL held, the other had released it. The one  
with the GIL held was sleeping inside of sqlite3_prepare, called from  
pysqlite_statement_create.

We were using the pysqlite shipped with python 2.5. Fortunately, the  
latest version of pysqlite standalone has already corrected that  
function to release the GIL around the call to sqlite3, so I  
downloaded and installed it. Hopefully that'll fix the remaining issues.

I also added an Expires header to the static content (of 24hours),  
although I think web browsers are pretty good at not reloading  
referenced content frequently even without an explicit expires, so I  
doubt that it really matters.

James