We've all heard the pitch: test automation saves testing time and resources. Test tools can execute tests faster than a person can, and in most cases they can do so in an unattended mode. So test automation should reduce test cycle time or the number of testers needed. Right? Not exactly. But Linda Hayes will tell you what it does save.
Many a test tool investment has been justified by a handy little spreadsheet that calculates the return on investment based on the reduction in the number of resources and amount of time being spent on testing. It's common to hear that test automation saves testing time and resources.
But is this true? Do companies really reduce their testing headcount and test cycle time as a result of automation? In my experience, the answer is no. Frankly, I have never seen a company reduce their staff or their schedule because they bought a test tool. Test tools do not save testing time or resources.
But does this mean that test automation has no value? Again, the answer is no. The key is to understand the true purpose—and value—of test automation. I submit that automation can save more time and money than are budgeted for the entire test organization and test cycle. How can that be, you ask?
Investment in Automation
Granted, automation is not cheap. Not only does it take time and effort to evaluate and select a test tool: it takes money to buy it and time to learn to use it; and more time to actually implement it; and the most time of all to maintain the test library as the application changes. In fact, experience shows that it takes between five and ten times as long to develop and debug an automated test as it does to execute it manually.
Okay. So you have to make the investment to get the return, and in most cases you will execute the same test at least ten times over the life of the application or even within the same release cycle. So, what's the problem?
The problem is that the goal of test automation should not be to reduce either test resources or cycle time. The goal should be to reduce the risk and cost of software failure by increasing test coverage.
Think about it. If your management's primary goal is to reduce the cost of testing, then I can propose a foolproof way of cutting 100 percent of your testing budget without even buying a tool. It's simple: stop testing! Just quit it. You can save all that time and cut all the staff. No problem, right?
Well of course this is a problem, you say. Then we would be shipping defective software, and that could lead to production downtime, customer crises, and all manner of development and support costs.
Precisely. It's really all about reducing the risk and cost of software failure.