[Twisted-Python] Re: CPU intensive threads

[Jp Calderone]

On 27 Jul 2005 11:05:43 -0400, David Bolen <db3l at fitlinxx.com> wrote:
>Nathaniel Haggard <natester at gmail.com> writes:
>
>> Is there a way to set the priority of the main part of twisted so that
>> it can run CPU intensive threads and still service connections.
>
> [snip]
>
>If you can't afford the time to transfer requests and/or results to a
>separate process over a normal channel, and are on a Posix platform,
>you might also investigate POSH (http://poshmodule.sourceforge.net),
>which implements object sharing in shared memory between processes,
>which would eliminate the transport overhead but still let you
>separate the processing into distinct processes.  It's an early
>development project that I've only experimented with but if it fits
>your needs it might do well.  (It comes with a simple
>producer/consumer example that could probably be used as a starting
>point)
>

POSH doesn't elimate the transport overhead.  I've done some basic investigation, and it's incredibly slow.  mmap() is probably a better solution in most cases, although I am not convinced multiple processes are called for in this case.

A solution which hasn't been suggested yet is to drop the native thread and use a cooperative Python thread.  With this approach, you can choose to schedule it however you like, including /not/ scheduling it when you have other more important tasks to complete.

Jp