[Twisted-Python] Distributed Trial Test Runner

[Andrew Bennetts]

Alex Lang wrote:
> Hello,
> 
> For my Summer of Code project, I am implementing a distributed trial
> test runner 
> (http://code.google.com/soc/psf/appinfo.html?csaid=770D0FCD95DAFF9E).
> The following is basically what needs to be done to get distributed
> trial working, given a master machine and slaves:
> 
> 1. Transfer the relevant code to all slave machines. (with conch)

This part is a bit short on detail.  "relevant code" isn't possible to determine
automatically, in general.  You never know what dependencies code might have.
Probably this user needs to be given some control over this, e.g. a setting for
the directory to copy (or rsync?).

I guess the trick will be that once a slave is set up to make sure that giving
it the current code for testing with is quick (copying the whole source for a
large project every time would suck, especially if you want to use a slave
that's on a relatively slow link).  rsync is definitely tempting here!

Another possible way to transfer the changes is as a branch URL, revision ID,
and diff.  i.e. the output of "svn info | grep ^URL:", "svn info | grep
^Revision:" and "svn diff" for SVN checkouts (but keep in mind not all projects
use SVN, so if you go this route you'd want to make part extensible).

> 3. Find a good test-to-unique-id mapping (probably using FQPNs +
> something else for test_suite-generated tests?)

The scheme discussed on #twisted along these lines sounded ok to me.

> I think that is all. Any thoughts? Comments?

Make sure that trial options that don't work when distributed give graceful
errors, rather than tracebacks or wrong results.  Options like "--debug" and
"--coverage" spring to mind, but there may be others.  Similarly, you should
also make sure that "--until-failure" does something sensible.  Basically,
look at each of the options described by "trial --help" and figure out what
distributed trial should do with it.  This is a relatively small thing, but this
sort of polish makes a big difference to the end user experience.

I also agree with L. Daniel Burr that there could be some overlap with Buildbot
in this project.  I guess you probably weren't thinking of running a master
daemon perpetually for slaves to connect to, but maybe that's not a crazy idea.
Also Buildbot has already deals with reporting results (in a minimal way),
updating source trees according to instructions from a master, and running
commands on those trees.  It also deals with managing slaves.

Because running tests requires a much more elaborately and carefully configured
environment than compiling a pre-processed C file, thinking about disttrial as
being more like buildbot than distcc might make good sense.

-Andrew.