[Twisted-Python] profiling twisted

[Maarten ter Huurne]

On Wednesday 27 June 2007, Markus Schiltknecht wrote:

> Now, I'm wondering if there's a profiling tool, which can quickly tell
> me what events are blocking my server. I have even deferred the file I/O
> to separate threads, as I've suspected them to make the server behave
> poorly when under load. But there still seems to be something
> blocking...  What do you use to find the callbacks which block (for too
> long)?

Does your server process use a lot of memory?

I had problems with a random simple request taking a long time every now and 
then and it turned out to be the Python garbage collector. The collection 
itself can take some time if you have a lot of objects. If some of the Python 
process ends up in the swap instead of main memory, garbage collection will 
block until it is swapped back in, which can slow it down dramatically.

Using multiple threads does not give you any performance advantage in this 
case: during garbage collection the entire Python VM is frozen.

One thing you could try is:
  import gc
  gc.disable()
in the startup code of your server.

Note that this does not turn off all memory recycling; it only turns off the 
mark-and-sweep part of the garbage collector. Objects which have a zero 
reference count are still collected.

Bye,
		Maarten