[Twisted-Python] Re: twisted and threading

[Paul Moore]

Jp Calderone <exarkun <at> divmod.com> writes:

> twisted-10152003 <at> atlas.lcs.mit.edu wrote:
> > 
> > Ah, but that doesn't work.  As I stated the c library (not mine) calls
> > potentially block forever (or at least longer than I want to wait to
> > shutdown the twisted server).  The thread created by deferTothread may
> > never complete which makes shutting down the server gracefully
> > impossible (or so I thought?).
> > 
> 
>    There is no portable way to terminate a thread without its 
> assistance.  Python supports no API for this - save one, the 
> ridiculously named "setDaemon" Thread method.  Twisted doesn't expose 
> this, nor call it internally, as it can lead to segfaults.
> 
>    Perhaps this should be parameterizable (defaulting to the current 
> behavior, of course), so that poorly behaved libraries can be dealt with?
> 
>    Alternatively, since daemonized threads might lead to this anyway, 
> perhaps you should just add a shutdown event os.kill(os.getpid(), 9) 
> <wwinkink>.

Sorry this is from long ago, possibly the OP has found a solution by now, but I 
thought I'd share my approach in case it was of more general interest...

I have hit the same problem recently (C library calls that might hang, and I 
want to give up and quit regardless in the end). What I came up with was a 
function deferToDaemonThread, which put the call in a daemon thread (one that 
the Python main thread will not wait for). It steals a lot of code from 
deferToThread, but creates a dedicated daemon thread for the call rather than 
using the thread pool.

Hope it's of interest,
Paul.

------------------------------------------------------------------------------

from twisted.internet import reactor
from twisted.python import failure
from twisted.internet import defer
from threading import Thread

def deferToDaemonThread(f, *args, **kw):
    """Run function in thread and return result as Deferred."""

    def putResultInDeferred(d, f, args, kw):
	"""Run a function and give results to a Deferred."""
	try:
	    result = f(*args, **kw)
	except:
	    f = failure.Failure()
	    reactor.callFromThread(d.errback, f)
	else:
	    reactor.callFromThread(d.callback, result)

    d = defer.Deferred()
    thread = Thread(target=putResultInDeferred,
	            args=(d, f, args, kw))
    thread.setDaemon(1)
    thread.start()
    return d