[Twisted-Python] Re: CPU intensive threads

[Nathaniel Haggard]

These CPU intensive threads are python/C extensions generated with
pyrex.  Instead of messing with the GIL I will use processes as David
suggests.  It would be nice to have a process pool implementation that
was as easy to use as deferToThread.

Evidently they use the perspective broker for inter process
communication in quotient.searchup, and that may be a good starting
place.

Thanks,
-Nate



On 27 Jul 2005 17:10:42 -0400, David Bolen <db3l at fitlinxx.com> wrote:
> Jp Calderone <exarkun at divmod.com> writes:
> 
> > POSH doesn't elimate the transport overhead.  I've done some basic
> > investigation, and it's incredibly slow.  mmap() is probably a
> > better solution in most cases, although I am not convinced multiple
> > processes are called for in this case.
> 
> Me neither, but it would be a logical way to progress if you couldn't
> resolve things in the single process.  Good to know about POSH though.
> 
> > A solution which hasn't been suggested yet is to drop the native
> > thread and use a cooperative Python thread.  With this approach, you
> > can choose to schedule it however you like, including /not/
> > scheduling it when you have other more important tasks to complete.
> 
> Well, but you'd still have the problem of ensuring that it was
> yielding back at a reasonable frequency wouldn't you?  So it would be
> similar to sprinkling in a time.sleep(0) in a non-cooperative thread,
> and still subject to cases where it might not be that simple.
> 
> -- David
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>