What Do You Do when a Showstopper Escapes into Production?

Member Submitted

Step 6
Step into your time-machine and fast-forward three to four months ahead. Everything is still the same. No one was fired after the fiasco, but the atmosphere was tense for about one week. After that, it became water under the bridge.

Your team is about to release another minor version, including all the patches of the past months plus another three features needed for important deals.

As always, product marketing is pushing for the release to go out on schedule even though your team got the final build a week late and you learned only yesterday that they included another feature that you were not even aware of.

As they say, "Nothing changes, if nothing changes."

Scenario No. 2Solve, Learn, and Improve
Step 0: Don’t Panic!

The last thing you want to do is to start looking for someone to blame. Chances are good that more than one person is “to blame” for making mistakes that led to the issue's having been released. It is almost certain that no one did this on purpose and, most importantly, blame will not help you to solve the issue!

So, try to stop your basic instinct to blame someone else for the issue. If another member of the team starts the blaming game, immediately ask him how this is contributing to solving the issue any faster. You can also state that there will be enough time after the issue is solved to understand what went wrong and why.

Step 1: Fix and Test
OK, so there is a bug out there and you had better fix it quickly! Put together the best team you can that will: analyze the issue; define the quickest, safest and most effective fix; create this fix; and, finally, recommend how to test it.

Which testing approach to take is not a simple decision, either. Depending on the bug and the product you are working on, you may choose to deliver the fix directly without doing any tests and verify only after the issue has been solved. (For example, if your application is web-based and the servers are down, it is better to get them back up and test once they are up). On other occasions, you may choose to run some or all possible tests in house before releasing the fix. This is usually the case if the bug is not really critical and the fix may cause bigger bugs, such as data loss or business disruption.

In short, the first step is to create the best solution and find the most appropriate approach to get it to your customers.

Step 2: Analyze
Once the critical part is over and the issue has been solved, but before people “move on” with their lives and tasks, it's better to make sure you understand what went wrong. The analysis process should never be a witch-hunt. It should be an opportunity where everybody feels safe to collaborate and bring forward all the factors—both internal and external—that contributed to the problem happening in the first place. This activity is fairly common and is called a retrospective or post-mortem.

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.