- that people must stay engaged during test execution.
- Localization. Only a human tester with appropriate skills can decide whether a translation makes no sense, is culturally offensive, or is otherwise inappropriate. (Currency, date, and time testing can be automated, but the need to rerun these tests for regression is limited.)
- Usability. As with localization, human judgment is needed to check for problems with the facility, simplicity, and elegance of the user interface and workflows.
- Documentation and help. Like usability and localization, checking documentation requires human judgment.
There's no return on investment in trying to automate these kinds of tests, typically. One of my clients once spent thousands of dollars and staff hours trying to automate configuration and compatibility tests, only to give up months into the effort.
In some cases, tests can be done manually, automated, or both.
- Functional. Functionality testing can often be automated, and automated functional testing is often part of an effort to create a regression test suite or smoke test. However, it makes sense to get the testing process under control manually before trying to automate functional testing. In addition, you'll want to keep some of the testing manual. As Cem Kaner, James Bach, and Bret Pettichord write in Lessons Learned in Software Testing , manual testing is adept at finding different kinds of functional bugs than automated testing.
- Use cases (user scenarios). By stringing together functional tests into workflows, you can create realistic user scenarios, whether manual or automated. The trick here is to avoid automation if many workflows involve human intervention.
- User interface. Basic testing of the user interface can be automated, but beware of frequent or extensive changes to the user interface that can incur high maintenance costs for your automated suite.
- Date and time handling. If the test system can reset the computer's clocks automatically, then you can automate these tests.
Higher per-test costs and needs for human skills, judgment, and interaction push towards manual testing. A need to repeat tests many times or reduce the cycle time for test execution push towards automated testing.
For any given test, manual or automated, we can do a quick, first-order cost benefit analysis. First, calculate the cost to design and implement the test. Next, add the cost to repeat the test multiplied by the number of times the test you need to repeat it.
The benefit of the test can be calculated using the cost of quality technique I discussed in the first article, but you might find it hard to come up with precise bug estimates for any given test. If so, consider the following approach. Estimate the likelihood of bugs in the feature or function you plan to test, in rough fractions like 25%, 50%, 75%, or 100%. Now, estimate the cost of such bugs, on average. The likelihood o f the bugs multiplied by the cost of the bugs gives you the benefit. If the benefit exceeds the cost to create and repeat the tests, then you have a business case for the test.
For example, suppose that you plan to spend $25,000 on tools and labor to create a performance test for your e-commerce application. You expect to spend a about $1,000 in maintenance and execution costs per test run, and you expect to run the test ten times over the course of a year. The test will cost $35,000. Suppose that, based on your study of your system and what you know about e-commerce sites, you think there is a 25% likelihood of slow performance.
Suppose you expect 100,000 potential customers per year, at an average