The Cost of Software Quality

cost of quality. Second, while it is often cheaper to prevent problems than to repair them, if we must repair problems, internal failures cost less than external failures.

A Hypothetical Case Study
Let's use a hypothetical case study to illustrate the use of this cost of quality technique to analyze return on the testing investment. Suppose we have a software product in the field, with one new release every quarter. On average, each release contains 1,000 "must-fix" bugs—unacceptable defects—which we identify and repair over the life of the release. Currently, developers find and fix 250 of those bugs during development, while the customers find the rest. Suppose that you have analyzed the costs of internal and external failure. Bugs found by programmers costs $10 to fix. Bugs found by customers cost $1,000 to fix.

As shown in the "No Formal Testing" column in Figure 1, our cost of quality is three-quarters of a million dollars. It's not like this $750,000 expenditure is buying us anything, either. Given that 750 bugs escape to the field, it's a safe bet that customers are mad!

Suppose we calculate that bugs found by testers would cost $100 to fix. This is one-tenth what a bug costs if it escapes to our custo mers. So, we invest $70,000 per quarterly release in a manual testing process. The "Manual Testing" column shows how profitable this investment is. The testers find 350 bugs before the release, which cuts almost in half the number of bugs found by customers. This certainly will make the customers happier. This process improvement will also make the Chief Financial Officer happier, too: Our total cost of quality has dropped to about half a million dollars and we enjoy a nice fat 350% return on our $70,000 investment.

In some cases, we can do even better. For example, suppose that we invest $150,000 in test automation tools. Let's assume we intend to recapture a return on that investment across the next twelve quarterly releases. Would we be happy if that investment in test automation helped us find about 40% more bugs? Finding 500 bugs in the test process would lower the overall customer bug find count for each release to 250. Certainly, customers would be much happier to have the more-thoroughly tested syste m. In addition, cost of quality would fall to a little under $400,000, a 445% return on investment.

This is a huge improvement over the initial situation. We are realizing a quantifiable and substantial return on our testing investment. We are also making our customers happier.

Is This For Real and How Do We Get There?
Cost of quality analyses on software process improvement bear out these figures. In Principles of Quality Costs , Campenella presents a case study from Raytheon that describes reductions in the cost of software quality from a whopping seventy percent of the total production cost to twenty to thirty percent. To put these percentages in more concrete terms, suppose you currently develop, deploy, and support systems at an average cost of $1,000,000 each. Reductions in the cost of software quality like Raytheon achieved would reduce this average cost to around $500,000. Can your organization use an extra $500,000 per system? While testing is only part of the investment in quality, it is an important part. Certainly a substantial investment is justifiable to achieve such phenomenal gains.

To get started, you’ll need a management team wise enough to look at the cost of quality over the entire life of the software release. A management team that ignores the long term and focuses just on the budget