Return on Investment
So what is the return on an investment in test automation?
Here is an example from my experience. An insurance firm with a highly specialized offering decided to outsource their software development but retain the software test responsibility. They reasoned that while programming was not their core expertise, the expected functionality of the system was.
Due to varying regulations, a steady stream of new offerings, and the constant state of adjustment of actuarial and premium tables, the software development effort was extensive and constant. The manual test effort required four weeks of dedicated work from a full-time team of five people. Yet even with this level of effort production problems were frequent, resulting in anything from downtime to misquoted policies to lost business.
Since a four-week test cycle was the most that the business could tolerate given the demand for delivering new functionality, and since her request for additional staff had been turned down, the senior test manager decided automation was her only answer. She bit the bullet and invested six months for two of her key testers, plus an outside consultant to help them select the right tool and set them on the right path for their test library design and management procedures.
At the end of the six months, her team had automated their existing test cases, and the execution could be completed in only one week. It would have taken even less time, but overnight batch processes and time-dependent factors required at least five calendar days of execution.
But here is the beauty: she did not advertise her accomplishments. She did not reduce her test staff, or promise test cycles of one week. Instead, she reinvested the time savings into expanded test analysis and coverage, and also in creating a daily "sanity check" that ran 150 test cases first thing every morning—in production—to check for problems with the infrastructure, data tables, interfaces, and other potential sources of failure. Her logic was simple: if she only did what she was doing before, then she would get the same results—production problems.
The payoff? Post-production incidents went down 80 to 90 percent. The savings to the company, as she was told when she got her raise and bonus, exceeded the entire budget for the test department.
The Real Payoff
If you want to justify test automation, don't look at your test budget, look at the cost of failure for the system you are testing. What does it cost the company or its customers in time, resources, and money if defects escape to production? What is it worth to deliver on time with a quality product?
If you do this analysis, you may find that test automation justifies a bigger test budget, not a smaller one.