It's Not About the Bugs


What's wrong with finding bugs? Nothing, unless you send the signal that bug-finding is all you measure and care about. Face it. People tend to perform those actions for which they are rewarded. So, if you don't track and value bug prevention, then it's not likely your team will prevent many bugs. In this column, testing authority Harry Robinson explains why and points to some more productive things to measure.

Scene 1:
You are picnicking by a river. You notice someone in distress in the water. You jump in and pull the person out. The mayor is nearby and pins a medal on you. You return to your picnic. A few minutes later, you spy a second person in the water. You perform a second rescue and receive a second medal. A few minutes later, a third person, a third rescue, and a third medal. This continues throughout the day.

By sunset, you are weighed down with medals and honors. You are a hero. Of course, somewhere in the back of your mind there is a sneaking suspicion that you should have walked upriver to find out why people were falling in all day. But, then again, that wouldn't have earned you as many awards.

Scene 2:
You are sitting at your computer. You find a bug. Your manager is nearby and rewards you. A few minutes later you find a second bug. And so on. By the end of the day, you are weighed down with accolades and stock options. If the thought pops up in your mind that maybe you should help prevent those bugs from getting into the system, you squash it-bug prevention doesn't have as much personal payoff as bug hunting.

What You Measure Is What You Get
B.F. Skinner told us fifty years ago that rats and people tend to perform those actions for which they are rewarded. It is still true today. In our world, as soon as testers find out that a metric is being used to evaluate them, they strive mightily to improve their performance relative to that metric-even if the resulting actions don't actually help the project. If your testers find out that you value finding bugs, you will end up with a team of bug-finders. If prevention is not valued, prevention will not be practiced.

For instance, I once knew a team where testers were rewarded solely for the number of bugs they found and not for delivering good products to the customer. As a result, if testers saw a possible ambiguity in the spec, they wouldn't point it out to the development team. They would quietly sit on that information until the code was delivered to test, and then they would pounce on the code and file bugs galore. The testers were rewarded for finding lots of bugs, but the project suffered deeply from all the late churn and bug-fixing.

That example sounds crazy, but it happened because the bug count metric supported it. On the flip side, I know of a similar project where testers worked collaboratively to deliver a high-quality product. They reviewed the spec and pointed out ambiguities, they helped prevent defects by performing code reviews, and they worked closely with development. As a result, very few bugs were found in the code that was officially delivered to test, and high-quality software was delivered to the customer.

Unfortunately, management was fixated on the bug count metrics found in the testing phase. Because the testers found few bugs during the official test phase, management decided that the developers must have done a great job, and they gave the developers a big bonus. The testing team didn't get a bonus. How many of those testers do you think supported prevention on the next project?

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.