[Twisted-Python] finding out what's blocking the reactor

[Chris Withers]

On 14/05/2019 16:39, Scott, Barry wrote:
> On Monday, 13 May 2019 15:26:01 BST Chris Withers wrote:
>> Hi All,
>>
>> On some busy instances of the app I'm working with/on, we see websocket
>> connections being dropped due to not ping/ponging in time, I'm fairly
>> sure this is as a result of some user-written callbacks blocking the
>> reactor for long periods of time (seconds...).
>>
>> What's the best way to debug a twisted application to find out what's
>> blocking the reactor?
> 
> Is the process CPU bound or IO bound?

Yes ;-)

> If CPU bound then cprofile or the tool Glyph mentioned would help.
> 
> This advice is from a Linux perspective.
> 
> If its IO bound the you may be waiting on a blocking operation.
> You could use strace on the process to see what its doing.

Good call on strace, will do so next time one of these happens.

> As its hung for seconds you could connect gdb to the process and see a python
> backtrace using the py-bt command. You are likely to catch the offending code.

I'm failing miserably to get gdb to find debug symbols for python. I've 
installed the system python's debug symbols rpm, I've installed gdb .py 
that ships in the python source tree (so at least I *have* a py-bt 
command now!) - what am I likely to be missing?

> You could adding logging to your code to show the time of callback entry and
> exit.

Right, but if the callback yields, that will be falsely shown as 
blocking, no?

cheers,

Chris