When you're testing software, you've got your priorities, right? Find as many of the worst bugs you can in the shortest possible time, of course. But what if the project manager has different priorities, such as reliability of a particular feature, or overall stability of the software under load? In this column, Elisabeth Hendrickson explains why it's best to ask a few questions before embarking on your bad-bug-hunting trek, or you might find yourself on a wild goose chase.
I was very pleased with myself. I'd just found a bug that, under certain circumstances, could result in data in a stored file becoming corrupt. I tried not to gloat as I explained the bug to the project manager.
His response floored me. "Oh, that. Yeah, we know. No time to fix it. How did the upgrade tests go?"
"You KNEW? And you won't be fixing it?!? Data gets corrupted!" Self-righteous anger bubbled up, blurring my vision.
"Whoa. Calm down. Yes, we got a report about that bug from one of the field engineers last week. We gave it a lower priority because it was easy to tell the file was messed up and there is an easy workaround. The bug has been there since 1.0 and fixing it now will require major changes. We're just about to release and can't delay the schedule to fix an old bug. So now tell me about the upgrade tests."
I fumed in silence, then turned to leave. The project manager stopped me. "What about the upgrade tests?"
I shot back over my shoulder: "I was so busy isolating this bug that I didn't finish them. I'll have the results tomorrow."
The project manager frowned. "I really need the results today. Last week you told me you'd have no problem getting them done. I don't think you understand how important these results are."
"I'll get them done before I leave today," I mumbled. As I left his office, I wondered, What went wrong? Why didn't he care about my news?
I realized that I hadn't clarified up front what kind of information was most important to the project manager. If I had, I would have understood the importance of those upgrade tests before spending half a day chasing down the file corruptor bug.
It all comes down to requirements. Tests have requirements. That incident with the project manager was a wake-up call for me. It was the first time I realized that my audience (managers) needs particular kinds of information. Like me, you could argue that the file corruptor bug was important. However, whether or not to fix a bug is a business decision. The project manager had the perspective and authority to make that kind of decision; I did not. At the same time, the project manager was relying on me to give him accurate information to support his business decisions.
So tests have requirements, but I wasn't sure how to discover those requirements for my tests. The laundry lists of features-often labeled "Requirements"-didn't help.
I started by asking, "Who uses the information I produce and for what purpose?"
In this case, the project manager wanted to know if the upgrade process worked as designed so he could make a release decision. He didn't want more bug reports unless the bugs were new to this release or interfered with the core functionality of the software. If I happened to encounter bugs, he expected me to file them. He just didn't want me to spend all my time digging for bugs at the expense of running the upgrade tests.
Different projects have different test requirements. Further, the nature of the test requirements may evolve as the project progresses.
For example, on one project, we needed to find as many bugs as possible early in the project. Later in the project, we needed to understand the end user's experience during typical use. Our test requirements changed mid-way through the project. As a result, we changed our testing strategy. We did a lot of bug hunting early and spent a great deal of time characterizing