We all have different attitudes and policies toward finding and fixing defects. The choice about whether and when to fix defects depends upon many factors, one of the least understood being the actual cost of fixing a defect. In this column, testing expert Johanna Rothman shares a formula for calculating the system test cost to fix defects and how to factor that into the bigger picture of your project.
Dan is a developer working on a project, along with four other people. They spent the first eight months of the project developing the product without fixing their defects, unless the defect prevented them from moving forward with development. Dan and his team thought it would be more cost-effective to fix their defects all at once. So at month nine, about a month away from the desired release date, they decided it was time to fix the defects.
Avery is a project manager at a company with a virtual lock on the market. Because the customers are captive, each customer wants a beta copy immediately, so they can start using the software. Concerned that a beta with many defects would make their customers angry, Avery decided it would be more cost-effective and less risky for the developers to find and fix the majority of the defects before they started system test.
Two projects, two completely different approaches toward finding and fixing defects. We all have different attitudes toward fixing defects, specifically which ones to fix and when. The choice to fix or not to fix depends upon many factors: the kind of product you're developing; the risks associated with shipping known or unknown defects; your development processes; and what it will cost you to fix the defect when you do fix it.
One of the least understood factors is the actual cost of fixing a defect. This cost feeds back into your choice of development lifecycle and development process and can help you decide how much risk you're willing to take to ship or not ship the product. However, many people don't actually know what it costs their organizations to fix a defect. If you're not sure either, here is an estimation technique to measure that cost:
In system test, when people are 100 percent dedicated to finding and fixing defects, count the number of fixes. You know how many people (developers, testers, and anyone else) worked on the project, and you know the duration of system test. This allows you to calculate the cost to fix a defect during this phase of the project. Here's how you can find the average cost to find and fix a defect:
Average cost to fix a defect =
(Number of people * number of days) * cost per person-day
(Number of fixed defects)
Note that the number of defects you find isn't enough information—it's the number of defects you fix. Detecting defects is only the first step. Locating the failure, deciding how to fix it, developer testing (a.k.a. unit testing) the fix, system testing the fix, and looking for other defects this fix caused is why the fix value is what's important. Let's look at some examples. In these examples, I assume a person-day is $500.