Grow Your Test Harness Naturally

I Have Worked With Many Testing Organizations Where a common pattern is repeated over and over. It goes something like this—we realize there is more manual testing to do than time available, we decide to automate the testing, and we begin working on a test harness. Several weeks later, we have the start of a harness, but it’s barely useful and we still have not written any tests. At this point, we're behind—so we abandon the harness and revert to manual testing.

The most common reason for abandoning a test harness is that the team writing it does not completely understand the needs of the team that will be using it. But even if there is a single team that will be both writing and using the test harness, the harness effort seems to take on a life of its own. Sometimes, the harness builders get so carried away with harness building that they never get around to test writing. It becomes a matter of pride that the harness should be able to deal with any possible testing requirement. No bell is too small; no whistle too trivial. Occasionally, the harness is actually completed but it sits on the shelf because it does not have the features that the testers need or because the testers don't have the technical skills to use it.

At my current company, we decided to try something different. We already enjoy the benefits of Agile planning and design and wanted to see if the same just-in-time philosophy would be as successful when building a test harness.

We had an extensive suite of unit-level tests but very few automated tests at the system or functional level. We wanted to start with a smoke test that could be run under our automated build system—CruiseControl—after every check in. (See Jeffrey Fredrick's article in the September 2004 issue of Better Software for more about CruiseControl.) A smoke test is a small set of system-level tests that run quickly and give you confidence that the product is ready for further testing. The name comes from hardware testing. If you turn the hardware on and smoke comes out, there is probably something wrong.

We had attempted to automate system-level tests before. We started by building a harness so that . . . well, you guessed it, that project was abandoned. This time, we decided to take a page out of the Agile developer’s rulebook and build the harness one test at a time.

The First Test
My company's product Agitator is a tool for automating, creating, and managing developer tests for Java code. Agitator has a great deal of complexity under the covers, but the basic algorithm for using it is quite simple. First, you agitate a Java class. Agitator then generates observations that describe what the code does, which you turn into assertions that describe what the code should do.

I decided that the first test should agitate a simple Java class that our CTO uses to introduce Agitator in demos and then check the observations.
public class SimpleMath {
    public static int add(int x, int y) {
         return x + y;
    }
}

When I agitate this class, Agitator observes that the return value of the " add()" method always equals x + y. I started writing a test that uses the same steps that I had performed in the product's user interface.

public class Observ ationSmokeTest extends TestCase {
      public void testXPlusY() {
            agitate(SimpleMath.class);
            OutcomeHarness outcome = getOutcome(SimpleMath.class,
                                                                                    "add(int x, int y)" );
            outcome.assertObservationExists("@RETURN == x + y");