[Twisted-Python] Using Twisted with MQ

[David Mitchell]

Thanks Chris,

That's pretty much along the lines of what I was planning to do - I'm
just wondering how much load I'll be able to throw at it.

Oh well, guess there's one very good way to find out ;->

Regards

Dave Mitchell

On 09/06/07, Chris Mellon <arkanes at gmail.com> wrote:
> On 6/8/07, David Mitchell <monch1962 at gmail.com> wrote:
> > Hello all,
> >
> > I've got what I think is a potential application for Twisted, but I'm
> > having trouble figuring out whether Twisted is going to be the best
> > path or not.  I've worked through the Twisted Finger tutorial and read
> > through the "Ball of snakes" O'Reilly book, but I'm not sure I've got
> > my head around Twisted sufficiently.
> >
> > I have to simulate a server that reads messages off IBM's MQ queues,
> > and posts responses back to different MQ queues.  I've used both pymqi
> > and Python/COM (on Windows only) to perform this sort of task in the
> > past, but now I need something that's going to be able to respond to
> > incoming MQ messages at a very high rate.
> >
> > I suspect I'm going to have to use reactor.callInThread to spawn off
> > threads within Twisted in order to handle IO with the MQ queues, as
> > I'm not aware of any event-based interface to MQ and therefore any
> > reads/writes from queues are potentially going to block for a while.
> >
> > Is Twisted suited for this sort of application?  Better yet, is anyone
> > aware of info I can access regarding this sort of application with
> > Twisted? - Google hasn't been much help...
> >
> > Thanks in advance
> >
> > Dave Mitchell
> >
>
> I haven't written any real code that does this, but I was just
> investigating it for feasability myself a couple days ago. I wrote a
> thread that looked basically like this:
> (pseudocode, as I don't have any of the code in front of me)
>
> while True:
>     try:
>         res = mq.get() #blocking call, with a fairly long timeout - 5-10 seconds
>         reactor.callFromThead(someCallbackFunc, res)
>     except Timeout: #pymqi will translate the timeout to an exception
>         pass
>     if self.exit.is_set(): #threading.Event to check for termination.
>         self.terminate()
>
>
> This just spins happily along, and dispatches processing into the
> Twisted event loop. I didn't really test it for performance or
> latency. Used reactor.callInThread to dispatch the writes.
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>