[Twisted-Python] automated tests for a server application

[Tom Most]

On Tue, Nov 6, 2018, at 8:43 AM, Kyle Altendorf wrote:
> On 2018-11-06 11:28, Chris Withers wrote:
> I guess I'm still not clear on what the point of using a 'fake' reactor 
> over a 'real' one is.  Not that I'm an expert here...

There are a bunch of advantages! Including:

1. Your tests become more deterministic --- even *fully* deterministic.
2. Fake I/O is way faster --- no syscalls!
3. Fake I/O is more reliable --- no worries about failing to allocate a port.
4. You gain full control over time. This makes it easy to test behaviors like timeouts and backoff timers.
5. You can run multiple fake reactors. Great for testing distributed systems!

At an extreme, you can make the whole suite synchronous (meaning, Deferreds fire immediately). treq.testing[1] can be used like this. [2] is an example of a suite in this style (this suite is actually run under the Django test runner, which knows nothing of reactors or Deferreds, though I don't recommend doing that).

There are some potential pitfalls:

1. As with any test double, you can trick yourself. I find faking I/O in this way is less prone to these problems than tools like unittest.mock, though.
2. Manually controlling time can harm composability. Application-level Twisted tends to eventually require some sort of time-related behavior, if only the humble reactor.callLater(0, ...) to avoid stack overflow. (Use of the cooperator from twisted.internet.task is another common one.)
3. Some (old) bits of Twisted don't allow passing a reactor or use the one passed in all code paths[3].

You can combine fake I/O with techniques like generative testing, too, though I don't have any public examples to point at. And of course you can also combine with other forms of dependency injection, like passing in an endpoint.

FWIW twisted.test.proto_helpers and twisted.test.iosim *are* public API, despite their presence in the test package. They could certainly use docs, and there is a ticket to move them somewhere more appropriate. For the moment the best way to learn to read them is to look at Twisted's own test suite. I found reading the tests for HostnameEndpoint helpful [4].

---Tom

[1]: https://treq.readthedocs.io/en/latest/testing.html
[2]: https://github.com/twm/yarrharr/blob/e31a488f8884fb82f1dba733585bef4a7ad86968/yarrharr/tests/test_fetch.py#L282
[3]: https://github.com/twm/yarrharr/pull/277/commits/1a5b5832edbf1fb1b1f45e9d99b65dad51ada566#diff-29ffe25b52ad7bddee9f2f08544e899cR98
[4]: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/test/test_endpoints.py