[Twisted-Python] Using Twisted for distributed computation / experiment running?

[Brett Viren]

robomancer <robomancer at gmail.com> writes:

> I'm looking to use Twisted for distributing computation over a small
> number (~10) of PCs.  I'm wondering if anyone else has some experience
> with this -- particularly if there is already a solution out there
> that I can use, so that I'm not reinventing the wheel. 

[OT: non-twisted discussion follows]

It seems you are wanting a hybrid between a batch queueing system and
a data management system.

Any reason not to use something like Torque (nee' OpenPBS) for the
batch part?

http://www.clusterresources.com/pages/products/torque-resource-manager.php

Or, if your nodes are also interactively used (ie, workstations by
day, batch nodes by night) you might look at condor:

http://www.cs.wisc.edu/condor/

Both are free-ish.


More info on what your data is like is probably needed for ideas on
the second part.

-Brett.