The $1 billion bug and why no one talks about peer code review.
It was only supposed to take an hour.
The bad news was that we had a stack of customer complaints. The latest release had a nasty bug that slipped through QA. The good news was that some of those complaints included descriptions of the problem - an unexpected error dialog box - and one report had an attached log file. We just had to reproduce the problem using the log and add this case to the unit tests. Turn around a quick release from the stable branch and we're golden.
Of course that's not how it turned out. We followed the steps from the log and everything worked fine. QA couldn't reproduce the problem either. Then it turned out the error dialog was a red herring - the real error happened long before the dialog popped up, somewhere deep in the code.
A week later with two developers on the task we finally discovered the cause of the problem. Once we saw the code it was painfully obvious - a certain subroutine didn't check for invalid input. By the time we got the fix out we had twenty more complaints. One potential customer that was trialing the product was never heard from again.
All over a simple bug. Even a cursory glance over the source code would have prevented the wasted time and lost customers.
The worst part is that this isn't an isolated incident. It happens in all development shops. The good news? A policy of peer code review can stop these problems at the earliest stages, before they reach the customer, before it gets expensive.
The case for review: Find bugs early & often
One of our customers set out to test exactly how much money the company would have saved had they used peer review in a certain three-month, 10,000-line project with 10 developers. They tracked how many bugs were found by QA and customers in the subsequent six months. Then they went back and had another group of developers peer-review the code in question. Using metrics from previous releases of this project they knew the average cost of fixing a defect at each phase of development, so they were able to measure directly how much money they would have saved.
The result: Code review would have saved half the cost of fixing the bugs. Plus they would have found 162 additional bugs.
Examine the following charts for the raw data. (Critical readers might take issue with the given dollar amounts. These numbers came from the customer and of course they are arguable and differ between organizations. However, as long as you agree that "bugs found later in the process cost more than bugs found earlier" the argument for code review still holds.)
Saving $150k: A real-world case study
Why is the effect of code review so dramatic? A lack of collaboration in the development phase may be the culprit.
With requirements and design you always have meetings. You bring in input from customers, managers, developers, and QA and synthesize a result. You do this because mistakes in requirements or architecture are expensive, possibly leading to lost sales. You debate the relative priorities, difficulty, and long-term merits of your choices.
Not so with when actually writing the source code. Individual developers type away at the tasks assigned to them. Collaboration is limited to occasional whiteboard drawings and a few shared interfaces. No one is catching the obvious bugs; no one is making sure the documentation matches the code.
Peer code review adds back the collaborative element to this phase of the software development process.
Consider this: Nothing is commercially published without corrections from several professional editors. Find the acknowledgments in any book and you'll find reviewers who helped "remove defects." No matter how smart or diligent the author, the review process is necessary to produce a high-quality work. (And even then, what author hasn't found five more errors after seeing the first edition?)
Why do we think it's any different in software development? Why should we expect our developers to write pages of detailed code (and prose) without mistakes?
We shouldn't. If review works with novels and