Practical Test Reporting, Part 2

[article]
Summary:

In Part 1, David Freeman wrote about how to start a basic test metrics program. In Part 2, he shows how to combine the historical information to predict how your future projects may track–kind of like creating your own metric "magic eight ball."

Historical Analysis and Trends--The Real Value Metrics
You've created a way to track and report progress for a day or an entire project. You can develop graphical information and begin to quantify the work the test team performs. Your boss is impressed. Imagine if you had several projects worth of data. You now see how your projects tend to perform historically. By transferring the daily report information to the project summary information and then to a testing-history repository, you can detect common trends in your projects and use this information to change those trends and predict future project progress.

Let's take a look at a snippet of summarized historical project information. Figure 1 shows summary information for several projects.

Figure 1
Average summary information is presented at the top of the sheet. Across the top is a set of columns labeled Project Pass %. Each column will measure a value specified in the appropriate row when the project is at a certain Pass %. For example, Phone Dialer 1.0 had 6,686 tests when the project was at 10% pass. The sheet also shows a number of averages that have been derived from the completed projects.

Find the Phone Dialer 1.0 project in the Project column. There are only a few pieces of data that have been transferred from the project summary.

The following four pieces of data allow us to calculate some additional measures:

• The length of the test cycle is determined by calculating the number of days between the start and end dates (in this case, including weekends).
• The number of days into a project a particular Pass % was accomplished is determined by finding the difference between the date the Pass % was attained and the start date of the project. This lets us see how long it takes for each
project to reach each milestone.
• The percent of the project time it took to reach each Pass % milestone is calculated by dividing the number of days into the project by the total length of the test cycle. A percent is used to normalize the data.
• The number of tests that changed between each milestone is determined as well as the percent of change.

The average values (noted at the top of the report) of Length of Test Cycle, Days into Project, % into Project, Number of Tests, and % Change of Tests for all the projects that have been measured allow us to look for emerging patterns. (All the data here is from real projects, though the project names have been changed.) Figure 2 is an example of some derived data.

Figure 2
The X-axis of this chart shows the Pass % milestones (100% is not shown, as rarely are all tests executed and passed). The Y-axis shows the percent of test cycle time it takes to reach the milestone. For example, on average it takes about 15% of the testing cycle to reach the 10% Pass milestone. It takes 36% of the project timeline--or an average of thirty-eight days of a 108-day project--to reach 50% Pass. So what is the crux of this data? Here are several ways to interpret what you've collected:

• On average, projects reach the 90% Pass milestone 53% of the way into the testing cycle.
• The 90% milestone can be attained in just half the allotted testing time.

Or, an even more disturbing interpretation of this data:

• On average, it takes the last 47% of the project's testing cycle to pass the last 10% of tests.

What could cause the timeframe discrepancies over the course of the project?

• Testers are gung