[Twisted-Python] Thoughts about testing

[David Ripton]

On 2005.10.26 13:14:56 -0400, Glyph Lefkowitz wrote:
> On Wed, 2005-10-26 at 04:56 -0700, David Ripton wrote:
> 
> > Things I particularly like about py.test, vs. unittest:
> 
> Can we compare to trial as it stands please?  I don't think anyone is

Sorry.  Haven't used trial lately, and didn't want to make outdated
comparisons.  unittest is the lingua franca; if I accidentally say
something useful, the trial guys can surely translate.

> > 1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe()
> 
> This is cute - more in the way it deals with printing appropriate
> context than the fact that you actually use 'assert' - but it's a shame
> that they resorted to a language hack rather than a simple function like
> 'test()' or something.  Using 'assert' makes it impossible to test with
> -O or -OO.  I personally don't use those options, but some people do,
> and they should be testable.

$ python -O `which py.test` test_assert.py
inserting into sys.path: /home/dripton/py-dist
/home/dripton/py-dist/py/test/cmdline.py:30: UserWarning: Assertions are
turned off! (are you using python -O?)
  py.std.warnings.warn("Assertions are turned off!"
...

So I think this is a purely theoretical problem.  I can respect
disliking the magic assert on general principles, though, even though
it's very handy.  (Kinda like the print statement.)

> > 2. You can use module-level test functions.  No need to make a test
> > class unless it's actually helpful.  (Yay Java.)
> 
> Handy in some cases, but from what I've seen in my brief survey of
> py.test-based tests the only thing this does is make it more of a
> convention to smear unit test state across a bunch of shared globals ...

Well, you can abuse globals to carry state across test methods in
unittest and trial as well.  (Init them at the module level rather than
in setUp.)

> > 3. Tests execute in predictable order.
> 
> ... and then since there is now global state smeared across your whole
> module, the tests have to run in an exact order because each one sets up
> things for the next, rather than proper setUp/tearDown encapsulation.

If someone insists on writing bad tests, I don't think there's much a
Python framework is going to do to stop them.  It's just not a very
authoritarian language.  "Stop, or I'll say 'stop' again!"  But I think
most slackers tend to skip writing bad tests and go straight to writing
no tests.

I admit to reusing expensive-to-create objects across multiple test
functions/methods for performance, but I have been conditioned by years
of xUnit to treat such objects as immutable.  (Okay, in the case of
database connections, I put a rollback call in the teardown, which is
not quite the same thing.)

There are other uses for predictable order, even if the tests are fully
independent.  Like running the test you're currently trying to fix
first, or moving the slowest ones to the end.  Knowing what your program
is going to do is sometimes handy.  (unittest is random enough that one
of my co-workers insisted it must be multi-threaded.)

> > 6. You can fill your tests with print statements, and only see their
> > output if the test fails (or you pass the -s option).
> 
> This is like a replication of trial's AND unittest's worst feature - the
> fact that tracebacks are deferred until execution is finished, so tests
> which hang intermittently are un-debuggable.  I believe it is only for
> the buildbot that -e is not the default.  Speaking of which, let me go
> update TwistedEmacs...

Yeah, that makes sense.  But *if* you're going to have deferred
tracebacks, then having print statements silenced when the test passes
can be nice, so that you can lie in wait for intermittent bugs without
generating too much noise.

-- 
David Ripton    dripton at ripton.net