[Twisted-Python] Understanding Deferreds/callback further

[Yi Qiang]

On 11/8/06, Phil Mayers <p.mayers at imperial.ac.uk> wrote:

> To be honest, the problem still seems a little abstract to me, and I'm
> having a hard time figuring out exactly what you're trying to achieve.
> Surely the clients have to submit *some* form of completion status to
> the PB server?


Ok.  The thing I am trying to design/code is a job distribution system, for
a distributed computation model.  There is a server which interacts with the
job database (which is just a bsddb).  The server accepts connections from
clients and distributes jobs to them.  The server also accepts connections
from a 2nd type of client, which submits jobs to the server.

The "Monitor" portion of the clients seem superfluous - what does it
> achieve that having the worker submit the completion does not?



For the clients whose sole purpose it is to get and process jobs, I
initially envisioned it containing 2 classes.  One class is the Monitor
class and one is the Worker class.  The Monitor's responsibility would be
coordinating the Workers since there can be multiple workers running on the
same client.  The monitor would coordinate tasks such as watching for
activity from the workers (i.e. update the output from workers as it comes
in) and also the monitor needs to be able to prematurely end a workers task
if the job is marked as unnecessary.
The workers of course have to submit their results back to the server, but
in the system I envisioned the monitor takes care of that by continuously
polling the status of the workers.

You're right in that this system complicates things since now I have to
handle communication between the client and the server, and on the client
side I need to implement communication protocols between the monitor and the
worker.  However, it does seem to buy me some flexibility.  Also, this will
keep the Worker class quite simple since all it actually does it execute an
external program to do it's job.

Without a more concrete description I'm afraid I don't have much more
> useful to add.


I hope this clears it up some more and am looking forward to your feedback.
I think the most difficult problem I am facing right now is to decide on a
good design.  This is kind of vague in text and I hope to be able to show
you some code soon, when I figure out how to publish a mercurial repository
:)

Thanks,
Yi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20061108/4a053aff/attachment.html>