[Twisted-Python] Perspective broker remote calls to multiple servers seem to run sequentially

[Sean Hollingsworth]

Geez .... what a HUGE duh moment. This will ultimately run on multiple
remote servers, but, while doing initial testing, I have been running it on
just my dev box. And, of course, I keep forgetting that twisted (well,
python) doesn't make use of multiple cores. I wasted all yesterday trying to
figure out why this was running the way it was.....
A million thanks.

Sean

On Fri, Jul 3, 2009 at 7:52 AM, Johann Borck <johann.borck at densedata.com>wrote:

> Sean Hollingsworth wrote:
> > Let me start off by warning everyone that I am extremely new to
> > twisted (I've been using it maybe a week) and it's highly likely some
> > things don't work the way I think they should. Or, I'm just using them
> > incorrectly.
> >
> > [..]
> >
> > The problems:
> >
> > Right now, for development and testing, I have three servers running
> > on localhost on three ports.
> Are you running them in a single process?
> > When I run my test code, it seems that each Perspective Broker runs
> > sequentially, rather than concurrently. For example, if i watch the
> > debugging output from the server, I see the following:
> >
> > server 0 processing dataval1
> > server 0 processing dataval2
> > server 0 processing dataval3
> > server 0 processing dataval4
> > server 1 processing dataval*5*
> > server 1 processing dataval6
> > server 1 processing dataval7
> > server 2 processing dataval8
> > server 2 processing dataval9
> > server 2 processing dataval10
> >
> > My understanding is that the perspective brokers would work
> > concurrently. Is that incorrect? My guess is that they should work
> > concurrently and I am just doing something wrong in my code, due to my
> > very, very limited understanding of how they work. Well, how much of
> > twisted works, really.
> >
>
> > Below is my the relevant code for the client (I've taken out code that
> > just deals with prepping data or debugging). Please keep in mind that
> > this is mostly testing code, while I get a better understanding of how
> > perspective brokers work and was cobbled together from examples and
> > docs found online.
> >
>
> [...]
>
> With the client code, nothing seems wrong, since it doesn't enforce one
> job to be finished before starting the next one - the problem has to be
> on the server side. Actually I suspect you're running the same service
> on 3 ports but within one process, right?. If this is the case it's no
> wonder the order of jobs completed matches exactly the order they're
> arriving in  because it all happens in a single thread. If you want
> parallel processing you will have to use multiple processes. As a side
> note (which doesn't seem to apply here): if the work they're doing is
> not CPU-bound, but IO-bound (meaning the work is nothing but waiting for
> something to complete, which for some reason can't be done in an
> asynchronous manner, the most prominent examples being blocking database
> interfaces), you also have the alternative of using threads.
>
> I think the basic problem is you're assuming twisted to somehow schedule
> the work to different cores/cpus, which it doesn't do by default. You
> might want to look at ampoule https://launchpad.net/ampoule, which is a
> process pool, there was another similar thing called AsynQueue, but the
> site is down atm: http://foss.eepatents.com/AsynQueue.
>
> hth,
> Johann
>
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20090703/4d481e25/attachment.html>