[Twisted-Python] Multiple Twisted Application Plugins in the same directory?

[Beau Hargis]

I am using the new twisted/plugin setup. I am not using mktap. So, in my project directory (which is included in the syspath) I have a dir 'twisted/plugins' into which I am going to put each of the services/applications. I do not want them running in the same process because I want to be able to stop one without affecting the others. Some of the services process a lot of data from 80-100 simultaneous connections so I need to be able to use multiple processors on the machine. The directory structure looks something like this:

Project\
...project.py
...Processors\
......ProcessorX.py
......ProcessorY.py
...twisted\
......plugins\
.........Service1.py
.........Service2.py

For one of the services I want to use pollreactor or epollreactor, so each of the Services has only 4 lines in it:

from twisted.internet import pollreactor
pollreactor.install()
import Project.Processors.ProcessorX
theService = Project.Processors.ProcessorX.ProcXServiceMaker()

This all works great until I want to add a second service in the same manner. When I add the second service I get the error about the reactor already being installed. So 'twistd --help' will only list one of the services and scrolling up the screen one can see the error it spits out when it attempts to evaluate the second Service. If I start the first service, it finds it and then I comment out any reference to importing the reactor, then it will successfully evaluate the first AND second Service and allow me to start the second one.

This is the error that it spits out:

  File "/usr/lib/python2.5/site-packages/twisted/internet/pollreactor.py", line 183, in install
    main.installReactor(p)
  File "/usr/lib/python2.5/site-packages/twisted/internet/main.py", line 24, in installReactor
    "reactor already installed"
exceptions.AssertionError: reactor already installed

And sometimes it will be a random failure (which is probably a cache interaction). It seems to be the combination of reactors. If I install the pollreactor in one and the epollreactor in the other then it works, but only one time. Then I check again and it fails. It seems to be loading up all the plugins in the same context.

----- Original Message ----
From: Jean-Paul Calderone <exarkun at divmod.com>
To: Twisted general discussion <twisted-python at twistedmatrix.com>
Sent: Wednesday, September 12, 2007 2:26:29 PM
Subject: Re: [Twisted-Python] Multiple Twisted Application Plugins in the same directory?

On Wed, 12 Sep 2007 14:06:50 -0700 (PDT), Beau Hargis <beau at subobscur.us> wrote:
>I started creating my services as twisted application plugins. When adding a second service as a plugin in the same directory as the first I get an error that the reactor was already installed. If I comment out the import of the reactor in one then I can start the other application. Neither the tutorial nor the source indicate what is going on except that it checks to see if a reactor is already installed at some point. What is the nuance here? Should I just go back to .tac files for these apps?

Could you describe what you're trying in a little more detail?  In particular,
I'm not sure if you're using mktap to create a tap file with multiple services,
if you're using the original plugin system (using tml files) or if you're using
the new plugin system (twisted.plugin).

Generally speaking, there's no problem running more than one service in a
single process.  A service is just an object which receives startup and
shutdown notification.  Anything beyond that is an application-specific
thing.

Jean-Paul

_______________________________________________
Twisted-Python mailing list
Twisted-Python at twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python