Practical Test Reporting

[article]
Part 1

other information. To reflect the complexity of actual product test efforts, the Dialing test area has been added and a Summary created.

Figure 3: Total Executed and Executed & Pass has been added to provide data regarding product quality.

Term

Definition

Total Executed Total tests that have been executed at least once
Executed & Pass Percentage of the Total Executed that have passed
Test status The results of a test when executed

 

Label

Calculation

Total Executed Pass + Fail = Total Executed
Total Executed % Standardizes Total Executed as a percentage of the test executed

Executed & Pass

As the project progresses, the Total Executed % should increase. If not, check the following:

  • Are testers not receiving new code or an improved product?
  • Are more test cases being added? If so, why weren't they written sooner?
  • Are requirements changing?
  • Are testers testing?

Executed & Pass should also increase over the course of the project. Observing historical data, you should be able to predict what the Total Executed and Executed & Pass levels will be based on the expected quality of product.

Managing the data
Basic test management and reporting can be accomplished with a spreadsheet. Tests can be recorded by area and reported on as frequently as you like. There are some inherent limitations (e.g., a tester needs to actively manage each test version).

This system is good for a single tester or a few testers with well-defined responsibilities. It can be modified to be as simple or complex as required, referencing external spreadsheets and summary reports.

What about larger testing efforts? Most of the reporting here is based on data collected by a commercial test management system. That system failed to provide useful reporting, so we built our own. Since the tests and results were captured in a database, we continued to use spreadsheets. Using information from databases, intermediary applications, and spreadsheets, we created a reporting system that was accurate and unchallenged.

Weighting
The volume of data presented should also demonstrate another useful point: weighting. In most cases, it's safe to assume that areas with larger volumes of tests are more important (Regression testing efforts often have large test volumes but aren't considered a top priority). Given the mathematical nature of the data, weighing the tests to provide accurate reporting for your organization should be straightforward. Document the process so your readers understand the significance.

Reporting Intervals
It is important to find a time-based reporting pattern. My organization developed an application that queried the data from the database several times a day, and the end-of-day report was considered the report of record. Daily reports may provide too much information for your project.

The Week in Review--Summarizing the Project's Progress
You're generating volumes of data at least daily and are more aware of the testing effort. Now begins the real use of the data--generating theoretical possibilities and testing targets.

Once a system has been created for the daily report, this information needs to be put into perspective. Telling the project team you've passed 45% of the tests is meaningless without context (e.g., if the team can see that yesterday's testing progress showed 55% Pass and today's is 45%). Consolidate each daily report into a project summary.

What is transferred from the daily report to the project summary? The number of tests for each area and total number of tests are recorded. Since adding and removing tests affects the Pass % (and all other percentages), it helps to see what changed during the course of the project and how that affected the testing effort. Requirement changes, scope creep, removed functionality,

User Comments

5 comments

About the author

David Freeman's picture David Freeman

David Freeman has been testing COTS and n-tier software since 1993, managing testing efforts since 1998, and a nerd for much longer. He can be reached at a david.freeman@yahoo.com.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Sep 22
Sep 24
Oct 12
Nov 09