[Twisted-Python] Problem with trial using the reactor

[Jonathan Lange]

On 6/7/07, Alex Lang <alex.lang at mail.mcgill.ca> wrote:
> Hello,
>
> Here is a problem regarding the distributed trial test runner. The
> slave instances receive data from the master, telling them which tests
> to run. This requires use of the reactor on the slave-side.
> Unfortunately, when the testCase does the cleanup of the reactor, it
> cleans up everything including the trial slave.
>
> Any ideas on what should be done about this?
>

This is a roundabout way of answering your question, but it's helpful
for me to write it down, and I think it will inform the discussion.

In terms of Trial's requirements:
* We want to write and run tests that perform asynchronous operations,
which essentially means tests that return Deferreds.
* We want to be able to run these tests in non-Trial test runners.
* We want tests to be isolated as much as possible.
* We want to be able to use Trial to test the reactor itself.

Some awkward facts:
* There is exactly one reactor per Twisted process. This reactor
cannot be cleanly restarted in a cross-platform way.
* The current Twisted test suite (i.e. what is loaded when you do
'trial twisted') does not play nicely with the reactor. Changing this
will take a significant amount of work.[1]

So, in response to all of these things:
* Trial presents a blocking interface for running any given test:
TestCase.run(). This allows Deferred-returning tests to be used in
non-Twisted test runners.
* By design, Trial totally obliterates the reactor in each call to
TestCase.run(). This helps ensure test isolation
 * Trial's design operates on the assumption that the caller of
TestCase.run() will not be doing anything with the reactor.

However, all of this presents problems:
* The vast bulk of Twisted's code base cannot be used to implement a
Trial runner. This is what Alex is discovering now.
* Trial's cleanup code is buggy and fragile. Changing anything is
likely to break hundreds of tests.
* Tests that use twisted.trial.unittest.TestCase run in a
highly-specialized, somewhat unrealistic environment.

Various solutions have been proposed:
* Using a customized reactor wrapper in trial that tracks resources
created by tests and cleans up only those resources.
* Making reactors restartable.
* Allowing for multiple running reactors. i.e. A reactor that could be
started within another running reactor.

I believe that only the third of these solutions would actually help
solve Alex's problem. However, as I understand it, the solution would
require extensive API changes across most of Twisted, as things which
once assumed a global reactor must now take 'reactor' as a parameter.

I'd rather not offer any recommendations in this email. It's difficult
enough figuring out what the problem is and what the options are.

jml

[1] For example, Trial does two calls to 'reactor.iterate()' in its
cleanup. If these calls are removed, tests fail and there are a
significant number of cleanup errors. I discussed some of the issues
here in an email last year.