Automated test tools are powerful aids to improving the return on the testing investment when used wisely. Some tests inherently require an automated approach to be effective, but others must be manual. In addition, automated testing projects that fail are expensive and politically dangerous. How can we recognize whether to automate a test or run it manually, and how much money should we spend on a test?
In the last article, I wrote that not only do we have to test the right things, but we also have to test them the right way in order to get a positive return on our testing investment. Static, structural, and behavioral test techniques are each good for finding certain kinds of bugs during certain phases of the project. However, there's another decision we have to make as test professionals for each test suite: Automated or manual?
Some tests are well-suited for automation—indeed, some kinds of tests can't be done manually in any meaningful way. Other tests, conversely, as a practical matter are either most effective when done manual or only doable manually. Test automation projects are just as risky as system development projects, and just as many fail. As with test techniques, selection of the appropriate choice in this regard will have a serious impact on the return on test investment.
When Test Automation Makes Sense
Let's start with the tests that ideally are automated. These include:
- Regression and confirmation. Rerunning a test against a new release to ensure that behavior remains unbroken—or to confirm that a bug fix did indeed fix the underlying problem—is a perfect fit for automated testing. The business case for test automation outlined in Software Test Automation is built around this kind of testing.
- Monkey (or random). Tests that fire large amounts or long sequences of data, transactions, or other inputs at a system in a random search for errors are easily and profitably automated, as described in Noel Nyman's article.
- Load, volume, and capacity. Sometimes, systems must support tremendous loads. On one project, we had to test how the system would respond to 50,000 simultaneous users, which ruled out manual testing! Two Linux systems running custom load generating programs filed the bill.
- Performance and reliability. With the rise of Web-based systems, more and more automated testing is aimed at looking for slow or flaky behavior on Web systems.
- Structural, especially API-based unit, component, integration. Most structural testing involves harnesses of some sort, which brings you most of the way into automation. Again, the article I wrote with Greg Kubaczkowski, "Mission Made Possible," provides an example.
Other tests that are well-suited for automation exist, such as the static testing of complexity and code standards compliance that I mentioned in the previous article. In general, automated tests have higher upfront costs—tools, test development, environments, and so forth—and lower costs to repeat the test.
When to Focus on Manual Testing
High per-test or maintenance costs are one indicator that a test should be done manually. Another is the need for human judgment to assess the correctness of the result or extensive, ongoing human intervention to keep the test running. For these reasons, the following tests are a good fit for manual testing:
- Installation, setup, operations, and maintenance. In many cases, these tests involves loading CD-ROMs and tapes, changing hardware, and other ongoing hand-holding by the tester.
- Configuration and compatibility. Like operations and maintenance testing, these tests require reconfiguring systems and networks, installing software and hardware, and so forth, all requiring human intervention.
- Error handling and recovery. Again, the need to force errors—by powering off a server, for example—means