We all have different attitudes and policies toward finding and fixing defects. The choice about whether and when to fix defects depends upon many factors, one of the least understood being the actual cost of fixing a defect. In this week's column, testing expert Johanna Rothman shares a formula for calculating the system test cost to fix defects and how to factor that into the bigger picture of your project.
Dan is a developer working on a project, along with four other people. They spent the first eight months of the project developing the product without fixing their defects, unless the defect prevented them from moving forward with development. Dan and his team thought it would be more cost-effective to fix their defects all at once. So at month nine, about a month away from the desired release date, they decided it was time to fix the defects.
Avery is a project manager at a company with a virtual lock on the market. Because the customers are captive, each customer wants a beta copy immediately, so they can start using the software. Concerned that a beta with many defects would make their customers angry, Avery decided it would be more cost-effective and less risky for the developers to find and fix the majority of the defects before they started system test.
Two projects, two completely different approaches toward finding and fixingdefects. We all have different attitudes toward fixing defects, specifically which ones to fix and when. The choice to fix or not to fix depends upon many factors: the kind of product you're developing; the risks associated with shipping known or unknown defects; your development processes; and what it will cost you to fix the defect when you do fix it.
One of the least understood factors is the actual cost of fixing a defect. This cost feeds back into your choice of development lifecycle and development process and can help you decide how much risk you're willing to take to ship or not ship the product. However, many people don't actually know what it costs their organizations to fix a defect. If you're not sure either, here is an estimation technique to measure that cost:
In system test, when people are 100 percent dedicated to finding and fixing defects, count the number of fixes. You know how many people (developers, testers, and anyone else) worked on the project, and you know the duration of system test. This allows you to calculate the cost to fix a defect during this phase of the project. Here's how you can find the average cost to find and fix a defect:
|
Average cost to fix a defect = |
(Number of people * number of days) * cost per person-day (Number of fixed defects) |
Note that the number of defects you find isn't enough information-it's the number of defects you fix. Detecting defects is only the first step. Locating the failure, deciding how to fix it, developer testing (a.k.a. unit testing) the fix, system testing the fix, and looking for other defects this fix caused is why the fix value is what's important. Let's look at some
examples. In these examples, I assume a person-day is $500.
Dan's project is uncovering large numbers of defects now that they're in system test, and even though most of the defects are quick to fix, some take very long. Avery's project is uncovering very few defects in system test, but because there's such a long time between the time each defect is uncovered, it appears to take a long time to fix a defect. Using the calculation above, here are Dan and Avery's data for system test:
|
Company (for a specific release) |
Number of people |
Cost per person-day |
Number of days |
Number of system test fixes |
person-days |
Average days to fix during system test |
System test cost to fix a defect |
|
Dan |
5 |
$500 |
40 |
125 |
200 |
1.6 |
$800 |
|
Avery |
10 |
$500 |
20 |
30 |
200 |
6.7 |
$3,333 |
As long as you're looking at the whole picture
About the author
Johanna Rothman, known as the “Pragmatic Manager,” helps organizational leaders see problems and risks in their product development. She helps them recognize potential “gotchas,” seize opportunities, and remove impediments. Johanna was the Agile 2009 conference chair. She is the technical editor for Agile Connection and the author of these books:
- Manage Your Job Search
- Hiring Geeks That Fit
- Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects
- The 2008 Jolt Productivity award-winning Manage It! Your Guide to Modern, Pragmatic Project Management
- Behind Closed Doors: Secrets of Great Management
- Hiring the Best Knowledge Workers, Techies & Nerds: The Secrets and Science of Hiring Technical People
Johanna is working on a book about agile program management. She writes columns for Stickyminds.com and projectmanagementcom and blogs on her website, jrothman.com, as well on createadaptablelife.com.
