We all miss some bugs, but the bug you're most likely to miss is one that gives wrong results that might look right. Let's look at a famous and costly example, then we'll see what we can do during testing to avoid a similar disaster.
We all miss some bugs, but the bug you're most likely to miss is one that gives wrong results that might look right. Let's look at a famous and costly example, then we'll see what we can do during testing to avoid a similar disaster.
American Airlines took a $50 million hit a few years ago because of just one bug of this type. To understand the bug’s impact, it helps to review a few basic airplane economics. Airline seats are one of the world's most perishable commodities—worse than lobster or fresh strawberries. When the wheels leave the runway, the price of an empty airline seat instantly drops from $438.95 to zero. Airlines never want to fly an empty seat; they would rather put a discount fare in the seat than fly it empty. On the other hand, they don't want to sell a seat at a discount rate if there's a good chance they could sell it at full price.
So the airlines overbook flights and run complex "yield management programs" to determine how many seats of each type to make available on a given flight. These programs start running up to a year before the flight and continue until a few minutes before boarding. They attempt to maximize revenue from the flight by looking at the mix of passengers already booked, the rate at which the flight is filling up, the time of year, and so on. From these factors and many others, the number of available seats of each type is continually changed.
These programs have become critical for airline profitability, and they get fiddled with a lot. One such fiddle, right before the summer travel season, was catastrophic. After this revision, the program gave a reasonable—but wrong—result for the number of discount seats available. If the number for any given flight should have been, say, "seventeen," the program might output "twelve." That's the number that would show up on the screens of travel agents around the world, and when those twelve discount seats were gone, they’d direct customers to another airline. The other five seats would usually fly empty.
When you multiply hundreds of dollars a seat by the number of empty seats per flight, times the number of flights per day, times the number of days in the busy summer travel season, that's a lot of money—and it's gone.
In the case of American Airlines in 1988, it came to $50 million. The top brass at American had to go down to Wall Street to explain to analysts why there was such a revenue shortfall.
Here's a quote by American spokesman Al Becker you can probably use to good effect: "We're convinced that if we had done more thorough testing, we would have discovered the problem before the new software was ever brought online." Exactly what kind of "more thorough testing" would have caught this bug? The answer itself explains why it got by so many people in the first place.
What Went Wrong?
There were two causes: (1) the answers were reasonable and (2) they were difficult to verify because of the complicated algorithms used to compute them. Let's take these one at a time, and see what we can learn about our own testing.
First, there’s the problem of reasonableness. As a former physicist, the first thing I do with most answers is check them for reasonableness. This keeps me from forging ahead with mistakes that, if I thought about it, would imply that atoms were a foot across or some such thing. So reasonableness is good. But just because an answer
About the author
Bob Stahl is the president of The Testing Center in Bodega Bay, California. He has thirty-four years of experience designing and testing software. He can be reached at stahl@testingcenter.com.
