It's Not About the Bugs
Software testing is not about finding bugs. It's about delivering great software. No customer ever said with a straight face, "Wow! You found and fixed 65,000 bugs-that must be really great software!" So, why do many currently use bug counts as a measurement tool? The answer is simple: Bugs are just so darn countable that they are practically irresistible.
They can be counted, tracked, and used for forecasting. And it is tempting to do numerical gymnastics with them, such as dividing them by KLOC (thousand lines of code), plotting their rate over time, or predicting their future rates. But all this ignores the complexities that underlie the bug count. Bugs are a useful barometer of your process, but they can't tell the whole story. They merely help you ask useful questions.
So What Should We Measure?
Here are some thoughts:
How many staff hours are devoted to a project? This is a real cost that we care about. How effectively did your whole team (project managers, developers, and testers) go from concept to delivery? Instead of treating these groups as independent teams with clear-cut deliverables to each other, evaluate them as a unit that is moving from concept to code. Encourage the different specialties to work together. Have program management make the spec more crisp. Have development provide testability hooks. Have the test team supply early feedback and testing.
How many bugs did your customer find? What are customers saying about your product? Have you looked through the support calls on your product? What is customer feedback saying to you about your software's behavior in the field?
How many bugs did you prevent? Are you using code analysis tools to clean up code before it ever gets past compilation? Are you tracking the results?
How effectively did your tests cover the requirements and the code? Coverage metrics can be a useful, though not comprehensive, indicator of how your testing is proceeding.
Finally, a squishy but revealing metric: How many of your own people feel confident about the quality of the product? In some aircraft companies, after the engineers sign off on the project, they all get on the plane for a quick test flight. Assuming that none of your fellow engineers have a death wish, that's a metric you have to respect! It not only says that you found lots of bugs along the way, but that you are satisfied with the resulting deliverable.
I recently saw an email signature that sums it up: We are what we measure. It's time we measure what we want to be.
For More Info:
Software Metrics: Ten Traps to Avoid by Karl Wiegers
This paper describes how to successfully measure software development activities.