What Do You Do when a Showstopper Escapes into Production?

Member Submitted
You are the QA manager of a company developing an enterprise application. Last week, your team released a product version, including features requested by new customers. But, it included a showstopper that already has affected about a third of customer installations. What would you and your team do in this situation?

Here’s a real life scenario. Let’s see how many of you can relate to it:

It is the middle of the day on a regular Tuesday afternoon. You are the QA Manager of a company developing an enterprise application, and last week your team released a minor version that included the fixes from all the patches of the past two months plus four minor features that were requested by product marketing in order to “close some pretty important deals."

All of a sudden, the phone rings. It is your R&D director “inviting” you to an urgent meeting in his room. You arrive to find the R&D director, together with his development team managers, the product marketing manager, and the support team leader in charge of your product, who is standing next to the whiteboard…

As you sit down, the support team leader tells all of you about an urgent showstopper that was released in last week’s version and that is expected to affect about a third of the companies who install this upgrade.

I want to stop this scenario (that happened to me about seven years ago) to ask you a simple question: What would you and your team do in this situation?

I believe that no two teams would react in the same way, and I don’t want to come up with best- and worst-case scenarios, but here are two contradictory approaches to serve as points in the possible behavioral continuum.

Scenario No. 1Blame and Panic
Step 1
The meeting turns into a witch hunt, where development blames testing for not finding the bug, then testing blames development for not documenting all the changes made in the system, then development and testing blame product marketing for pushing the teams to release even though not all the tests had been completed, etc.

Step 2
After the meeting dissolves without any clear action items, support starts telling customers not to install the new release, the programmers start working on a solution without fully understanding the problem, and you are left on the side wondering how you missed this bug and trying to find the person who should be responsible for it.

Step 3
Since the developers think this is absolutely urgent, they decide to send the fix directly to support, and only in parallel they send it to your team for validation and verification. They do this at 8:30 p.m., when no one is left in the office and you can only start testing it at 8:30 a.m. the next day. About thirty minutes into your testing cycle, you start finding bugs in their new version. The problem is that your support team already started delivering this solution to the initial set of customers who already complained about the bug.

Step 4
By midday, your team finishes the tests on the system. They find that the initial fix only works on about half the supported configurations and, more importantly, it also causes a regression bug on an area not directly related to the fix.

Within thirty minutes, you get a new version that is verified and released to the support team by 4 p.m.

Step 5
Customer support wants to kill both your testing team as well as the developers, because now they need to find every company that downloaded the first fix and call to ask them not to install it—or, even worse, to install yet another fix on top of it.

Product marketing is also mad at you, since they already started getting calls from customers, account managers, and even some of your company’s top executives complaining about the mess and bad publicity this fiasco is already creating in the field. They let you know that as a result of it your company will need to offer large discounts to all customers that complain, and they think this issue may cause a number of important deals to be delayed or lost.

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.