[Twisted-Python] clustering or process group replication

[Laran Evans]

jarrod roberson wrote:
> We are working on something similar for a project at work also.
> We are using ZeroConf ( aka formerly Apple Rendezvous now Apple Bonjour 
> ) for cluster discovery
> and a Perspective Broker interface for exchanging information.

 From what I've read about ZeroConf it seems like a LAN-only discovery 
mechanism. For clustering a web-server LAN only is fine. But there are 
certainly cases where it would be valuable to integrate WAN nodes.

If I were to build a toolkit I would want it to be able to run optimally 
on a LAN while maintaining the ability to achieve reliability and high 
performance on a WAN as well.

The folks at the AccessGrid at least would I'm sure appreciate that ability.

> We considered reliable multi-cast but decided against it because our 
> clusters are not going to be very big and
> are going to use a kind of inversion of control pattern instead of 
> distributing all the data everywhere.

The keyword in reliable multicast is "reliable". As soon as you have 2 
processes that you need to keep in sync, some form of reliable 
communication is necessary. And by virtue of the fact that there is more 
than one process, sending messages falls under the title of multi-cast.

So, I'm very interested to know how you integrated the IOC pattern to 
achieve reliability. This sounds a bit like a push vs. pull paradigm. If 
  that's the case I can point you to a paper which discusses the 
achievable properties of push, pull and push-pull scenarios.

> But I am interested in a COMMON IDIOM for clustering twisted servers, I 
> would definately say ZeroConf needs to be
> the discovery mechanism, no need to re-invent that wheel and a Twisted 
> implemenation is easy

ZeroConf does definitely look solid. And I'm 100% with you on 
not-reinventing the wheel. So I'll finish reading some of the ZeroConf 
specs to see what it can do.

- Laran