[Twisted-Python] Multiple simultaneous SelectReactors?

[Glyph Lefkowitz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Monday, April 14, 2003, at 08:36 PM, Peter Hansen wrote:

> There appear to be serious problems in several areas of 
> twisted.internet
> if one tries to create multiple SelectReactors to use at the same time.

That would be putting it mildly.

> Why would one want to?  For acceptance testing, when it can be 
> desirable
> that the code under test run in the same process as the test code, but
> it's not appropriate to share the Reactor.

I'm aware of this use-case, but it was difficult to write for a variety 
of reasons.  History plays a large role in Twisted's, um, history.  We 
try very hard to support backwards compatibility for many versions, so 
that early adopters do not get burned by quicksand interfaces, and 
sometimes a bit of ugliness results.  This is one of those cases.

As someone new to Twisted, your time would probably be better spent 
coding your acceptance tests to use spawnProcess and actually invoking 
different Python processes for the different components.  It's 
relatively easy to communicate between them once this is done.

> 1. Is Twisted expected to support this in principle, if not currently
> in practice?

Yes.  If you read the CVS logs, you'll notice a trend that more things 
these days take a "reactor" argument than they used to.  We have been 
slowly refactoring towards this for a while.

> 2. Are the globals "reads" and "writes" that are used in
> twisted.internet.default there solely for optimization purposes, or is
> there some more nefarious purpose?  Is it considered absolutely 
> critical
> that this optimization is in place?  (If so, it probably makes it
> impossible to fix this limitation of Twisted.)

I would prefer that this optimization remain in place (changes to code 
this far down into the core is a severe problem), but it wasn't 
originally an optimization.  It's present there as a 
backwards-compatibility hack for twisted.internet.main, which is so 
deprecated that you will get sick and die if you even look at it.  It 
can probably be changed at this point.

> 3. Is it well-known that the abstract.FileDescriptor class does a quiet
> little fallback to the global reactor if you don't pass it a reactor 
> reference,
> and that Port fails to pass in its own reactor reference when it 
> creates
> a Server object in doRead()?

Yes.  Trust me, this was the best compromise at the time :-).

> We've strung together a variety of little changes in the process of
> figuring this out.  Once we're more confident of what we're doing,
> I'd be happy to supply the fixes, but I'm not going to waste my time
> if everyone thinks SelectReactor.doSelect() absolutely must do those
> optimizations.

I would be happier with a simple subclass that was 
slightly-slower-but-multiple-reactor-capable.  However, if you were to 
supply a patch that actually caught all these nuances and fixed them in 
a sensible way, I seriously doubt I would turn it down if you undid a 
tiny (and dubious) optimization :).

> Also note: I'm working on the most recent released version, not CVS.
> The slightest hint that this stuff is significantly changed in CVS
> will send me there, but until now we weren't confident enough with
> Twisted to work from unreleased code.  (Actually, attempting to
> resolve this has slightly lowered my confidence in Twisted, but
> only just a bit.)

The unreleased code doesn't fix this in particular, but it does include 
a LOT of bug fixes.  We are currently struggling to get a release out, 
thanks to the heroic efforts of Kevin Turner.  Hopefully it won't be 
much longer now :).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (Darwin)

iD8DBQE+m2h8vVGR4uSOE2wRAhFTAKCjmEHVoQI1hAGZHXq7Y9DHlPSzFACeMOh6
bmy8goeb0OEsB/1b12qBz80=
=6aR2
-----END PGP SIGNATURE-----