Hilarious dream-logic (was Re: [Twisted-Python] [patch] (etc)

[Jonathan Lange]

On 5/19/06, Andrea Arcangeli <andrea at cpushare.com> wrote:
> On Fri, May 19, 2006 at 10:41:35AM +1000, Justin Warren wrote:
> > break existing code, ie: regression testing. I'm personally a big fan of
> > this feature of test suites.
>
> I'm a big fun of test suites as well. I only disagree with wasting time
> delaying the integration of a valid bugfix just because the unit test
> doesn't exist yet. I absolutely never said unit tests are a waste of
> time. For the last months I've always said that "unit tests" are
> _welcome_ at the top of the cpushare-twisted webpage.
>
> > Incidentally, those with slightly more CS knowledge know that it is
> > possible, though by no means easy, to build a system that is provably
> > correct. An investigation of the Z specification language may prove
> > enlightening. I think you mean that a test suite proves that the system
> > passes all the tests, not that it is bug free.
>
> What I mean is that a test suite cannot prove the code is bug free. Nor
> that any bugfix is correct. If nothing else because the test suite may
> be buggy too. This is obvious.
>
> Clearly a test suite is welcome and can only help, but its mandatory
> requirement for any change to the code sounds way excessive.
>

Andrea,

Your main objection is that requiring unit tests for bug fixes is
excessive.  I think it would be fair to say that you believe that
following such a requirement means wasting time.

Many people have defended this requirement.  They have said that writing a test:
- ensures the bug stays fixed
- communicates important information about decisions in the code
- provides a clear [and fast -- jml] way of reproducing the bug
- confirms the existence of the bug

You raised a concern that unit tests do not prove the absence of bugs.
 You have also stated that unit tests are desirable, but not
necessary.  I would like to respond to these points.

First, although unit tests do not *prove* the absence of a bug, they
provide strong evidence for its absence.  When fixing code, there must
be some check that the fix is correct.  The check generally takes one
of four forms: informal induction from the code; manual
experimentation; automatic experimentation (i.e. unit tests) and
formal verification.  The first provides unacceptably weak evidence.
The fourth proves the matter, but is too difficult to attempt for most
bugs.  This leaves the second and the third.  I would consider either
acceptable for demonstrating a bug's absence.

Second, without unit tests the amount of work required to maintain
code increases over time, and so they are necessary.  Each of the
advantages listed above has a corresponding disadvantage: the bug will
probably appear again; decisions in code lie undocumented; bugs cannot
be easily reproduced and so cannot be easily determined as either
fixed or unfixed.  Each disadvantage costs time. There is a finite
supply of time and an arbitrarily large supply of bugs.

If you wish to argue against our requirement for unit tests, then you
must persuade us either that the evidence they provide is not strong
enough to provide the listed benefits, or that the work in writing
unit tests is significantly greater than the work of not writing unit
tests.

Regards,
jml