[Twisted-Python] Re: Taking a look at pending deferToThread actions

[Ed Suominen]

Allen Bierbaum wrote:
>> > - Has anyone extended threadpool to allow code to remove queued
>> > functions from the pending queue?
>>
>> How about just having the functions themselves check to see if they
>> really should run their intended task? (Being careful about
>> thread-safety, of course).
> 
> Thanks for the idea.  I will give it a try.

You also might want to take a look at my taskqueue sub-package in
Twisted-Goodies, http://foss.eepatents.com/Twisted-Goodies. One feature
that might come in handy is the IWorker implementors' ability to resign
when they feel that they are no longer able to run jobs. Any jobs that
the resigning worker has queued up individually at that point are
returned to the main queue for dispatching to other workers, current or
future.

The whole thing revolves around a priority queue, which might be useful
to you, too. Also, a ProcessWorker implementation of IWorker is pending
that spawns a separate Python interpreter to run tasks. If the
ProcessWorker hangs, you'll be able to just kill it and move right
along, without hanging your main Python interpreter.

> I am using threads because the calls I am making are into a
> non-blocking library.  I may see about rewriting it in the future to
> be more twisted-friendly, but for now I have to live with it.

That was my original motivation for writing taskqueue, with its
ThreadWorker implementation of IWorker. It's used in sAsync, which makes
a wonderful but totally synchronous database API (SQLAlchemy) play very
nicely (IMHO) with Twisted.

Debian packages are in the works for both, by the way.

Best regards,
Ed Suominen