[Twisted-Python] Thoughts about testing

[Antony Kummel]

Hi Jp,

--- Jean-Paul Calderone <exarkun at divmod.com> wrote:

> >An alternative feature could be
> >enabling the user to specify that a certain delayed
> >call or thread is allowed to remain after the test,
> >and then Trial won't complain. The only question
> >remaining is how to do it. Simple: use a different
> >process. Run the tests in a different process, and
> >create a new one each time the reactor is dirtied.
> >py.execnet is a nice example of this concept.
> 
> Running tests in a child process is an interesting
> idea.  It 
> provides a much greater degree of isolation between
> tests than 
> essentially any other approach, which is great. 
> Isolation is 
> great for unit tests.  Unfortunately, it comes with
> a lot of 
> overhead.  While there are techniques for optimizing
> the 
> implementation of such a feature, running each test
> method in a
> different process would probably add at least 4
> minutes to 
> Twisted's test suite.  This is basically
> unacceptable (Twisted's 
> suite takes way too long to run already).

Well, it will certainly be non-optimal, but let me add
some more points in its favour:
1. It may still be faster than debugging unit tests
that fail because of a dirty reactor or other problems
that it resolves.
2. You don't have to run each test method in a
different process, only the ones who leave the
environment dirty. Of course people are likely to be
lazy if they get the chance, but it makes more sense
to leave it up to them. And of course it could be
optimized (is a pool of process what you had in mind?)
3. This feature brings us very close to completely
distributed unit testing. This will make it possible
to easily run tests simultaneously on different
computers which would actually make it faster than the
current method with very little effort (given enough
computers).
4. A mechanism allowing for running unit tests at
random locations (processes or machines) will probably
be good for making the tests themselves span more than
one computer. Despite your justified objections I
think this could be quite a powerful tool.
5. The same mechanism can be useful for another
problem I often run into: dirtying of the interactive
prompt environment. When using a relatively rich tool
such as PyCrust, for example, I have to close it and
start it again every time I make a grave enough
mistake (for example when I need to reload nested
modules). If the code I write at the prompt was to run
in a different process, I could simply click a button
to restart that process without it having an annoying
impact.

> >The second thought is this: there seem to be
> popping
> >up different testing toolkits each with their own
> very
> >nice extensions and features
> >(http://testoob.sourceforge.net/,
> >http://codespeak.net/py/current/doc/test.html).
> Trial
> >cannot benefit from this, having branched away at
> the
> >pyUnit level. I think Trial's special features can
> be
> >relatively easily formulated as plugins to a
> >plugin-oriented testing framework (especially if
> the
> >clean reactor requirement is relieved), and so can
> the
> >other testing packages. What this means, is that
> the
> >best thing anyone who wants the world of unit
> testing
> >to develop, and to benefit from it, is to push for
> a
> >plugin-oriented pyUnit, and for an implementation
> of
> >Trial (and the other tools) as a plugin for that
> >framework. I think.
> 
> Rather than hearing about the plethora of new
> testing libraries 
> appearing, I'd like to hear about features they
> provide that are
> valuable for writing tests.  I would certainly like
> to borrow 
> py.test's magical assert.  What other features are
> test authors 
> finding useful in some of these projects?

First of all, TestOOB already allows you to run test
in a different process (although I didn't go down to
the details -- this may be a little different than
what I talked about). If the plugin-oriented framework
I proposed existed, the feature we discussed above
would have already been available to us.

Secondly, it is not hard to think about ways to make
unit testing easier. Take a graphical UI for example.
In the current situation, any such improvement will be
useful for only a small group of users, and that group
is likely to grow ever smaller with the emergeance of
new frameworks. And when doing unit tests, we all
desparately want the same thing -- to make it easier.
With a common base, properly designed, we can all
benefit from everyone else's efforts, and I think this
is good enough a reason.

I join you in wanting to hear what test authors are
finding useful in these projects.

Antony Kummel



		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs