[Twisted-Python] clustering or process group replication

[Laran Evans]

Well, there are a couple of things to consider. Here are a few.

1/. How many processes in the group. Most protocols work well in groups 
up to a few hundred. When you get over a thousand or so things start to 
get complicated. And above that there are protocols which use epidemic 
techniques, so-called "gossip" or "rumor"-based protocols. Bimodal 
multicast is one such protocol.

2/. Network environment. Protocols that run exclusively on a LAN can 
make certain assumptions that WAN-inclusive protocols can't. So, some 
protocols shift things around a bit, intentionally slowing things down 
to accomodate the WAN nodes. So the whole system runs some factor 
slower, but reliability is maintained.

Beyond that there are all sorts of techniques, each of which makes 
certain assumptions. So, which protocol to utilize depends on what type 
of environment you want to create for applications.

Spread's definitely a solid choice. JGroups, Ensemble, Horus, Isis, 
Totem, Transis and Eternal are all top-notch as well, each of which 
provides slightly different properties. There has been some talk lately 
on the JGroups mailing list about adding python extensions to JGroups. 
JGroups is what's used for clustering in Tomcat and JBoss I believe. 
Though I could be wrong about that.

There's one design in which I'm quite interested which was initially 
proposed for an international stock exchange. You can read about that 
protocol here:
http://portal.acm.org/citation.cfm?doid=380749.380771.
This particular design is specifically designed to work on the Internet. 
  And is based on TRMP.

So, I hope this helps.

- Laran

Ian Duggan wrote:
> What would be the steps involved in integrating a few of the various protocols 
> involved here. Is there a chance of having a common API, or do the "large 
> blocks on the internet" variants need something different than something 
> optimized for "small blocks on the lan". 
> 
> I don't have a really good notion of all these things, but I'm considering 
> trying to integrate the spread.org stuff into twisted to achieve clustering 
> of servers into a logical virtual server, for redundancy. There might be a 
> simpler way to do the same thing. I'm still exploring.
> 
> Does anyone know of an example API for integrating several of the protocols 
> into the same framework? How do these types of protocols break down into 
> categories? 
> 
> --Ian
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

-- 
- Laran Evans