[Twisted-Python] Twisted webserver performance issues

[Maarten ter Huurne]

On Sunday 08 November 2009, James Y Knight wrote:

> When I last looked into the performance issues, I found that sometimes
> trac appears to block for long periods of time without releasing the
> GIL. That seems to be the core of the performance issues, currently.
> When it's responding normally, it's perfectly snappy. But, sometimes,
> it blocks for 10sec at a time.

How much memory does Trac use? Python garbage collection works in 
generations, which leads to recently allocated objects to be checked for 
garbage more often than older objects.

http://docs.python.org/library/gc.html

When collecting the oldest generation (2), we saw the Python VM freeze for 
several minutes. This was on a process that had several GB of allocated 
memory though. There was enough RAM to contain all data, but it's still 
possible the OS (Linux) swapped out pages there were not used for a long 
time to get more memory for caching.

We decided to work around the issue by disabling mark-and-sweep collection 
and relying on reference counting only. We had to add some code to forcably 
break reference cycles, but this was only needed for a small number cases.

Bye,
		Maarten