The 'pre-run tasks' unit testing anti-pattern

The fundamental idea behind unit tests is that you run them all the time. You will probably run a single test every two or three minutes as you make changes to your code, and a suite of tests at least every twenty minutes, probably oftener. This means that every change you make to a piece of logic is followed by running the test for that piece to validate the change.
Given this style of usage, everything about your tests should be built to suit. Some the obvious and oft repeated best practices to support this include

  • Ensuring that every unit test be atomic, testing just a single piece of logic. A fairly good rule of thumb is that you would have one test per method, and your methods don't have more than a few (<15) lines

  • Ensuring that every unit test have a fresh state and not depend on any previous tests to create that state

Now, the 'pre-run' anti-pattern surfaces where you have a set of infrastructure tasks that need to get run to set up a suitable environment for running tests. I'm not referring to the things you'd do in your setUp() method (which sets up the program state rather than the environment in which the program runs), but rather to situations where (say) your tests require a set of xml files to run against and these need to be copied into the test directory from somewhere else.
This is something I've noticed happen when developers run tests only from ant, rake or whatever build script they're using. There, it's natural to just add these tasks to the dependencies of the test task and be done with it. You don't have a problem because the dependency graph in your script ensures that the environment is as it should be for those tests.
This increases the threshold for unit testing and low thresholds to running tests frequently are very important, because the effect they have over time in the behavior patterns of a team can be quite dramatic. As a consequence of these external dependencies, developers are discouraged from actually using unit tests the way they're meant to be used, because you're now forced to run your tests from a build script.
You want to encourage people to run tests in the manner I described above. Ideally, you should be able to do a fresh checkout of a code-base and go in and run any one test directly and have it go green.

No comments: