[Twisted-Python] Twisted-friendly Message bus

[Paul G]

----- Original Message ----- 
From: "Chaz." <eprparadocs at gmail.com>
To: "Paul G" <paul-lists at perforge.com>
Cc: "Twisted general discussion" <twisted-python at twistedmatrix.com>; "Phil 
Mayers" <p.mayers at imperial.ac.uk>
Sent: Wednesday, June 28, 2006 8:52 AM
Subject: Re: [Twisted-Python] Twisted-friendly Message bus


> If you go to the Spread website and read the documentation and source
> code it mentions both the number of computer limitations and the
> performance hit. Secondly, the system uses a static file to list the
> machines that participate in the communication.
>
> First, it won't support more a 200 computers, or so. It appears that
> deep in the bowels of the code is a byte sized field! The code is pretty
> convoluted, so I never did manage to find out what specifically was the
> field they mentioned. If you are going to build a system that involves a
> hundred machines or less you are probably alright.

yes, for my purposes, a small number is perfectly fine.

> Secondly, as for the performance a search will point out how costly
> reliable group communications are in general. I would suggest you load
> the software up and run the demo, tracking the messaging passing with
> Ethereal. Once this begins to scalable upwards to hundreds of machines
> you will see the problem.

there isn't an incredible amount of cost for baseline group communication. 
the incremental cost is attached to several incremental levels of delivery, 
ordering and consistency guarantees. assuming i wanted delivery guarantees, 
but didn't care about ordering or consistency, i imagine the cost wouldn't 
be much higher than plain udp.

with spread specifically, i would imagine there's also extra cost associated 
with handling the WAN case and paritioning, which i wouldn't need. however, 
that's something i could live with if it weren't too great.

> Third, the configuration of a Spread cluster is fixed based on a config
> file. This was a problem for me; I wanted my system to scale upward by
> just adding a machine to the cluster and not having to change a config
> file and rebooting the daemon machines! It doesn't make for a very
> reliable approach (after all if IM systems could do it why couldn't
> non-IM systems?)

there new version (currently in beta) removes this limitation.


> Now if you think Spread is the way to go, and it might be for small
> clusters of machines, you should do a little research into things like
> the SWIM protocol (of which I am doing a twisted implementation right
> now and will be releasing to the community) or other approaches to
> group-like communication (do a google of overlay networks, which I will
> be also releasing). You will find a lot of relevant information. In fact
> they will probably lead you to a group at Cornell (and a few people
> there) and some people at the U of I at Champaign.

although i hadn't seen swim (doesn't help much, in any case), i've done 
quite a bit of digging andread quite a few papers. i've got several use 
cases i'm trying to find the right solutions for:

1. a lightweight message bus (a la mq) for a cluster of machines on a 
low-latency lan; must be fast, light, usable from python, php and c, not 
implemented in java, and perferably have an async api. this would be used to 
implement a cluster-wide event service.
2. same as above, but with much relaxed requirements for use as a log 
message handling service (think sending apache logs through it to logging 
servers).
3. a lightweight group communications framework which would allow me to 
implement multiple finely tuned semantics for consistency and ordering 
guarantees. this would be used to implement distributed data structures. can 
assume low latency lan environments, doesn't need to care about partitions 
(this would eventually run over myrinet/infiniband/friends).

unfortunately, the preponderance of work done in this area centers around 
'overlay networks', p2p and other such beasts; this introduces complexities 
which make the solutions too heavy for my use. it appears that the stuff 
you're interested in is along the same lines; that doesn't mean it (or you) 
suck(s), just that it's not what i'm looking for ;]

moreover, i would ideally like to use something very thin and hackable; 
spread is already too much of a monolithic black box for use case #3 - the 
dds stuff i want to do would allow me to select consistency guarantees from 
none, to modest to very stringent for each specific case, eg i wouldn't want 
to pay for serialization if i were replacing, say, php's session storage 
stuff and intended to store transient data.

the berkley ninja/sedo stuff is the closest i've seen, but it's in java (not 
a huge problem, since i could port it) and the code appears to be 
unavailable (large, some would say insurmountabl, problem).

if anyone's got any ideas, i'm all ears. it looks like i'll be implementing 
#1 on top of twisted (locally, not for twisted project proper) in the near 
future (unless the dev roadmap changes *again*), so i'm definitely eager to 
hear about alternatives to sticking this on top of pb (not the right tool 
for the job, imho) or spread. in this case, java solutions are out (i'm not 
an anti-java zealot, i won't discuss the reasoning here).

-p