I was discussing release criteria with a client recently. The development manager proudly arose and said, "We have no defects at level 0!" I asked what that meant, and he smirked, saying, "No bugs we absolutely have to fix before we ship." "Great!" I said. "How many defects do you have at the next level?" He sat down and said, "I don't actually know." "Oh?" "Well, the developers and testers are fighting it out down the hall."

Oh dear.

These folks were tying themselves into knots, trying to deal with the risk of releasing the product with risky defects. They had four levels of priority and five levels of severity. They used all combinations, but they really only had two levels: the ones they fixed and the ones they didn't fix. The problems they didn't fix during a given release, they didn't normally fix in a later release either. They weren't using the information in their problem reports to their benefit.

Instead of priority and severity, I use risk as a way to deal with problem reports, and how to know how to fix them. Here are the levels I choose:

• Critical: We have to fix this before we release. We will lose substantial customers or money if we don't.
• Important: We'd like to fix this before we release. It might perturb some customers, but we don't think they'll throw out the product or move to our competitors. If we don't fix it before we release, we either have to do something to that module or fix it in the next release.
• Minor: We'd like to fix this before we throw out this product.

Three simple levels. But it's not simple to transition to them. When I suggested these levels instead of their twenty combinations, the managers said, "But the developers won't know what to do first. And the testers won't know what to verify first." I asked if we could talk to a couple of developers and a couple of testers.

One developer thought the levels were a good idea. He couldn't tell the difference between a priority 2/severity 3 problem and a priority 3/severity 2 problem. To him, they were all the same. He thought if he could see all his critical problems, he could manage his time and approach his fixing better.Another developer was less interested. "I fix all the problems in severity order. I don't care about priority. If it's severe, I need to fix it. This critical/important/minor deal just doesn't cut it for me." The development manager said, "But why do you fix according to severity? Why don't you fix according to priority? That's why priority is there." The developer said, "Well, the priority is about the customer. The severity is about the product. You deal with the customer, I deal with the product."

By this time, both the developer and the development manager were rolling their eyes, exasperated with each other. Instead of helping the developers and managers organize their work, the priority and severity scheme caused them to make different tradeoffs.

One of the testers said, "I'll have too many critical problems. How will I know which fixes to verify first? Without priority, I won't know which ones to deal with first." I asked the tester how he knew which fixes to verify first now. He said, "Well, I always verify in priority order." His manager said, "Um, that's not true. We talk about which features we have to make sure work first. Then you verify fixes for those areas in priority order. Hmm, maybe we can do that here."

This tester was right to be concerned about the risk of not verifying fixes when the developers needed the information. If you use release criteria, and openly discuss which areas of the product are critical for product success, you can manage that risk.

A three-tiered ranking for problem reports isn't going to solve the promotion-demotion problem of which problem is assigned to which level, especially at the end of a release. But it will make the number and risk of your problems clearer to everyone concerned. Then maybe you can fix more defects before the product is obsolete.

Reference
Arguing Apples and Oranges, By Elisabeth Hendrickson