The mood in the meeting was grim. All eyes were trained on Doris, the head of customer support. Doris surveyed the room as she spoke..."We have a problem in the field with the new release. Sixty-three users reported unrecoverable errors this week-a record high. An additional 152 people reported crashes, but the software recovered after reboot. This morning, I talked to an irate user who said he'd uninstalled our software after it crashed on him five times in a row. He wanted us to give him a full refund plus expenses. In short, the users are really angry. What do I tell them? When will we have a fix?"
Silence.
"What the heck is going on out there?" the CEO snapped.
The VP of Engineering clenched his jaw. "I'll get right on it," he replied.
This scene has been replayed in software development organizations since the dawn of the information age. How does bug-infested software escape into the real world, causing users and technical support representatives such grief?
A common response is to blame the Test group: "How did you guys not find this bug?"
This blaming stance puts testers on the defensive: "Maybe it's because the programmers gave us rotten code, management dictated the ship date, and no one listened to what we told them about the quality of the software!"
Let's suspend the attacking and defending for a moment and look at how bad software escapes. If there's a critical bug in released software, one of two things happened:
- No one found the bug before release.
- Someone found the bug but no one fixed it before release.
None of us likes it when bugs in the first category bite us, but it's the bugs in the second category that hurt the most.
Consider the history behind Doris's report. The testers reported that the software under test occasionally crashed for no apparent reason. The problem occurred very rarely, once or twice a week during peak testing times. The testers could not determine the circumstances under which the crash would occur. The programmers looked at the problem and couldn't figure it out either. The project team reluctantly deferred the issue to make the ship date.
Within a month after release, Doris was completely fed up with handling complaints from irate users and took her concerns to the executive staff.
Predictably, Doris's report spurred a great deal of finger pointing. The testers, who reported the problem, blamed the programmers for not investigating it thoroughly enough. The programmers blamed the testers for not being able to reproduce the problem in the lab. Everyone blamed Management for absurd schedules. None of the blaming led to better software. In fact, the blaming led to worse communication and schedule slips as team members scrambled to cover their backsides.
It doesn't have to be that way.
Everyone on the project can play a role in preventing bad software from escaping.
So what advice would I have for the testers at Doris's company for their next release?
- Stop the blame.
Blame gets in the way of learning. Don't accept blame and don't dole it out. - Find extreme cases .
Find ways to turn minor glitches and occasional anomalies into major problems or frequent errors-even if that means doing things "no user would ever do." Users have a funny way of doing what no one expects. Although the testers at Doris's company were never able to make the software crash
reproducibly, they later learned from field reports that the crashes seemed
to occur more often when the system was under load or the user did things at unexpected times. The testers then used this information to make the software crash consistently in twenty minutes instead of once a week. - Explain the impact of bugs.
Make sure the bug report describes the effect of the bug on the end user. A
bug titled "uninst.isu file missing" may be correct, but a bug titled "user cannot uninstall" is much more compelling. - Champion serious bugs.
If you believe that a bug will cause serious problems in the field, fight for its correction. Don't accept excuses like "too risky to fix," "no user will do that," and "there's a workaround" at face value. Ask questions. "But what's the risk if we
About the author
The founder and president of Quality Tree Software, Inc., Elisabeth Hendrickson wrote her first line of code in 1980. Moments later, she found her first bug. Since then Elisabeth has held positions as a tester, developer, manager, and quality engineering director in companies ranging from small startups to multi-national enterprises. A member of the agile community since 2003, Elisabeth has served on the board of directors of the Agile Alliance and is a co-organizer of the Agile Alliance Functional Testing Tools program. She now splits her time between teaching, speaking, writing, and working on agile teams with test-infected programmers who value her obsession with testing. Elisabeth blogs at testobsessed.com and can be found on Twitter as @testobsessed.
