profit of $10 per customer. If half of your customers will abandon their transaction—and your site—if transactions are handled too slowly, then your estimated loss is $125,000 from poor performance. Spending $35,000 to mitigate the chances of such a loss makes sense.
If you expected only 10,000 potential customers, then you’d need to figure out how to run the test for $12,500 or less, for example by outsourcing a single run. If a test can be run manually as well as with automation—though performance tests typically can't—then you might want to consider the cheaper manual test, performed less frequently. Alternatively, you'd want to see if people were comfortable skipping the test based on your analysis.
Of course, there might be other considerations that would affect your analysis. Perhaps the word-of-mouth issues that would accompany bad performance make spending more money a priority for sales and marketing. However, if there's no real business case for the test, then you should not expect to get a positive return on investment from it, regardless of the other benefits.
Reasons to Be Careful with Automation
Automated testing is a huge investment, one of the biggest that organizations make in testing. Tool licenses can easily hit six or seven figures. Neophytes can't use most of these tools—regardless of what any glossy test tools brochure says—so training, consulting, and expert contractors can cost more than the tools themselves. And then there’s maintenance of the test scripts, which generally is more difficult and time consuming than maintaining manual test cases.
Capers Jones, in Estimating Software Costs , cites failure rates for large, complex system development efforts as high as 50%—or higher. A test automation project can be just as complex as developing software, and, indeed, Dorothy Graham and Mark Fewster cite similar failure rates for automation projects in Software Test Automation . Two anecdotes will serve to illustrate this point.
One of my clients will not allow me to use the phrase "test automation" in front of him. When I do refer to it, I jokingly call it "the A word." His organization spent over half a million dollars on tools and consulting to try to automated their complex software package. While the tool vendor, the vendor’s salesperson, and the consultant all cashed their checks, what my client has to show for that investment is a maze of unmaintainable automated test scripts and some nice looking test tool boxes on a shelf.
These failures don't necessarily have to do with incompetence of the automation itself. Remember my cautionary tale in the second article, when we focused exclusively on (automated) functional and regression testing to the exclusion of (manual) installation and setup testing. This was a well-designed, highly maintainable test system, one of the most elegant test automation solutions I've ever worked on. But it didn't solve the real test problem, delivering an accurate assessment of the users' and customers' experiences of quality.
Sadly, these are not isolated incidents. Many who’ve tried their hands at test automation have unhappy stories to relate. And gravely, when an investment this big goes awry, the loss of credibility for the associated parties is huge. People get fired for losing this kind of money. You'll want to position yourself for success when embarking on test automation by doing your homework with a book like Software Test Automation . Also, keep in mind that much of the work associated with automated testing must be done manually, such as the design of the test actions and data, the calculation of the expected result, and the analysis of the results. Successful test automation efforts don’t