[Twisted-Python] Perspective Broker and thread safe!

[Jean-Paul Calderone]

On Fri, 6 Feb 2009 17:30:19 +0100, "B. B." <thebbzoo at gmail.com> wrote:
>Hello, I am relative new to twisted, though I have manage to make a simple
>client server solution ( using perspective broker ) that I am now try to
>refactor a little.
>
>The solution is now something like :
>
>Using perspective broker, clients connects to the server ( setting up a
>remote reference to each other ).
>Via a PB call, each client can request the server do some processing. The
>result of  the processing is distributed to each connected client - and the
>client asking for the request gets a simple status report!
>
>This solution works quite well, the only bottleneck is that all the
>"thread-safe" processing on the server is done in the mainloop of the server
>reactor. I want to do concurrent processing on the server!

That requires that your server be capable of doing the processing
concurrently.  If the processing is CPU bound, then you need multiple
CPUs.  If it is disk I/O bound, you need multiple disks.  etc.

>The function doing the processing on the server, also distribute the result
>to all the connected clients, and returns the result to the klient asking
>for the request.
>
>My question is :
>If I just "defer" that function "toThread" ( like  threads.deferToThread(
>myServerProcessingFunction ) ). Do I have potential problems, when several
>threads try to distribute its result to the connecting clients???

If you use deferToThread with your "processing" function, then the function
must be thread-safe.  Since nearly no Twisted APIs are thread-safe, it cannot
use them directly.  It may only use reactor.callFromThread to send events
back to the reactor thread.  So, in general, yes, the potential for problems
exists.

>
>If I have a problem:
>Can I make a quick fix as follows:  Within each thread ( started with the
>threads.deferToThread function ), I do the distributing to the clients by
>using the "reactor.callFromThread"
>( like reactor.callFromThread( distributeToAllClient , myResult ) where the
>function distributeToAllClient are using the remoteReference for each client
>to send the "myResult" )

This may work.

>Or is the only solution, to let the myServerProcessingFunction returns the
>"myResult" to be distributed to all connected clients, and do the
>distribution in the reactor mainthread???

There are always lots of possible solutions.  Since you are particularly
interested in performance, it will probably be necessary for you to create
a way to measure how well your application is performing and then repeat
these measurements using different implementation techniques.  That is the
only way you'll know which one is best.

Jean-Paul