Inspections are a great way to find defects in software work products, but how well do your inspections work? Do they actually reveal defects? Are they cost-effective, compared to testing or other quality approaches? Do they pay for themselves? The only way to answer such questions is to collect and analyze some data from your inspections.
Assessing Inspection Impact
Three measures of inspection impact are effectiveness (the percentage of the defects present in a work product that your inspections discover), efficiency (the average effort required to find a defect), and return on investment (the time your organization saves from inspections compared to their cost).
Calculating effectiveness requires that you count the defects found by inspection, those discovered in later development or testing stages, and those that customers report. If you know your inspection effectiveness, you can estimate how many defects remain in a deliverable following inspection.
Efficient inspections discover many defects per hour of effort. There's a paradox here, though. A successful inspection program leads to process improvements that reduce the number of mistakes that developers make. So, as your product quality improves, the cost of discovering each defect will increase. You need to judge whether decreasing inspection efficiency metrics truly indicate higher product quality or if they mean that your inspections are not working as well as they should.
Return on investment is the net savings from your inspections divided by the detection cost. The net savings are the costs that your team avoided by finding defects early, rather than late. If you know the average cost of finding and fixing a defect during system test or in operation, you can estimate the potential savings from each defect found by inspection. Detection cost includes the effort spent on each inspection plus the overhead costs of your organization's inspection program.
The basic dimensions of software measurement are size, time, effort, and quality. The following data items will help you assess your inspection effectiveness, efficiency, and ROI.
Size.Planned lines of code or document pages that you planned to inspect
Size.Actual lines of code or document pages that were actually inspected
Time.Meeting duration of the inspection meeting
Effort.Planning total labor hours spent on planning, scheduling meetings, assembling the inspection package, and the like
Effort.Overview total labor hours spent on the overview stage
Effort.Preparation total labor hours spent on individual preparation
Effort.Meeting total labor hours spent in the inspection meeting
Effort.Rework total labor hours spent correcting and verifying defects
Defects.Found.Major number of major defects found
Defects.Found.Minor number of minor defects found
Defects.Corrected.Major number of major defects corrected during rework
Defects.Corrected.Minor number of minor defects corrected during rework
Number.of.Inspectors number of people who participated in the inspection meeting
You can calculate several useful metrics from these data items; some are listed below. To jump-start your inspection data storage and metric calculations, use the simple spreadsheet available from www.processimpact.com/pr_goodies.shtml.
Defect.Density number of defects found per unit of material inspected
Effort.Inspection total labor hours expended on the inspection
Effort.per.Defect average total labor hours expended to find a defect
Effort.per.Unit.Size average labor hours expended to inspect a document page or a thousand lines of code
Rate.Inspection average quantity of material inspected per meeting hour
Rate.Preparation average quantity of material covered per labor hour of preparation
Rework.per.Defect average number of labor hours needed to correct and verify a defect
As a general metrics guideline, if you don't plan to analyze the data, don't waste time collecting it. You can analyze data accumulated from multiple inspections in several ways:
- Track averages, such as the number of lines of code inspected per hour