[Twisted-Python] Threadpool analysis?

[Phil Mayers]

Jonathan Vanasco wrote:
> 
> On Apr 11, 2007, at 5:02 PM, Phil Mayers wrote:
> 
>> Why are you using a threadpool with Twisted?
> 
> 
> I'm not using one directly -- I'm using deferToThread , which twisted 
> manages via its own threadpool.  Stuff is locking up, so i'm using 
> manhole to try and see wtf is going on.

Ok. What are you running in deferToThread?

More generally, you said things keep locking up. How, what are the 
symptoms? What OS and version are you using, how many outstanding 
requests have you deferToThread'ed, etc.

Answering your original question more directly, you can monitor basic 
info about that usage like so:

import time

COUNT = 0

def myDeferToThread(*p, **kw):
     global COUNT
     COUNT += 1
     stime = time.time()
     def cb(val):
         etime = time.time()
         global COUNT
         COUNT -= 1
         print "deferToThread returned in %.1fsec" % (etime-stime,)
         return val
     return deferToThread(*p, **kw).addCallback(cb)

You can using a task.LoopingCall to report the outstanding number 
periodically. You can even get sophisticated and use an increasing 
sequence number like so:

OUTSTANDING = {}
SEQUENCE = 0

def myDeferToThread(*p, **kw):
     global SEQUENCE
     global OUTSTANDING
     SEQUENCE += 1
     argstr = 'p=%r kw=%r' % (p, kw)
     OUTSTANDING[SEQUENCE] = argstr
     def cb(val):
         args = OUTSTANDING[SEQUENCE]
         del OUTSTANDING[SEQUENCE]
         print "%s returned" % (args,)
         return val
     return deferToThread(*p, **kw).addCallback(cb)

...or something along those lines.

Some of which might help you.