The Science of Catching Hidden Bugs


MyRand 2.0 succeeded in producing an even distribution of numbers. Number 9 was no longer the odds-on favorite. But then I realized I needed to ask another question. Would the results be different if I restarted MyRand before generating each number? I'd gathered the 10,000 integers without exiting MyRand. Perhaps exiting and restarting the program would change the results.

Experiment #3: Exit the program between each "Randomize!" while gathering the 10,000 numbers.

I discovered that the chances of getting a particular number changed dramatically depending on whether or not you had just started MyRand. If you left MyRand running, you had an even chance of getting any given number. However, if you'd just started MyRand, 9 was once again much more likely to come up. Another definitive "fail." Back to the drawing board.

Tips for Experimenting
The MyRand experiments demonstrate the power of asking, "What if...?" Adopting an experimenter's mindset adds a new dimension to testing. When I'm experimenting, I like to:

  • Gather lots of data. Notice that my sample for the MyRand tests was rather large: 10,000 numbers. Perform the same tests under different conditions. By restarting MyRand before each "Randomize!" I learned that a small change had a huge, unexpected impact on the test results.
  • Perform a series of slightly different tests and analyze the patterns of results. When I began using the malformed SQL query I mentioned earlier, inconsistencies in the data alerted me to a problem. What patterns or inconsistencies might alert you to problems in your system?
  • Finally, question the tests. If the system were producing incorrect results, would the tests make the failure obvious?

Scientists formulate hypotheses, design experiments, and compare predictions to observations. As testers, we also follow the scientific method. We hypothesize that we'll find a bug if we exercise the software in a particular way. By taking our method a step further, using test results to suggest new tests, we increase our chances of exposing camouflaged bugs and gaining a deeper understanding of the complex systems we test.

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.