The Science of Catching Hidden Bugs

[article]

MyRand 2.0 succeeded in producing an even distribution of numbers. Number 9 was no longer the odds-on favorite. But then I realized I needed to ask another question. Would the results be different if I restarted MyRand before generating each number? I'd gathered the 10,000 integers without exiting MyRand. Perhaps exiting and restarting the program would change the results.

Experiment #3: Exit the program between each "Randomize!" while gathering the 10,000 numbers.

I discovered that the chances of getting a particular number changed dramatically depending on whether or not you had just started MyRand. If you left MyRand running, you had an even chance of getting any given number. However, if you'd just started MyRand, 9 was once again much more likely to come up. Another definitive "fail." Back to the drawing board.

Tips for Experimenting
The MyRand experiments demonstrate the power of asking, "What if...?" Adopting an experimenter's mindset adds a new dimension to testing. When I'm experimenting, I like to:

  • Gather lots of data. Notice that my sample for the MyRand tests was rather large: 10,000 numbers. Perform the same tests under different conditions. By restarting MyRand before each "Randomize!" I learned that a small change had a huge, unexpected impact on the test results.
  • Perform a series of slightly different tests and analyze the patterns of results. When I began using the malformed SQL query I mentioned earlier, inconsistencies in the data alerted me to a problem. What patterns or inconsistencies might alert you to problems in your system?
  • Finally, question the tests. If the system were producing incorrect results, would the tests make the failure obvious?

Scientists formulate hypotheses, design experiments, and compare predictions to observations. As testers, we also follow the scientific method. We hypothesize that we'll find a bug if we exercise the software in a particular way. By taking our method a step further, using test results to suggest new tests, we increase our chances of exposing camouflaged bugs and gaining a deeper understanding of the complex systems we test.

About the author

Elisabeth Hendrickson's picture Elisabeth Hendrickson

The founder and president of Quality Tree Software, Inc., Elisabeth Hendrickson wrote her first line of code in 1980. Moments later, she found her first bug. Since then Elisabeth has held positions as a tester, developer, manager, and quality engineering director in companies ranging from small startups to multi-national enterprises. A member of the agile community since 2003, Elisabeth has served on the board of directors of the Agile Alliance and is a co-organizer of the Agile Alliance Functional Testing Tools program. She now splits her time between teaching, speaking, writing, and working on agile teams with test-infected programmers who value her obsession with testing. Elisabeth blogs at testobsessed.com and can be found on Twitter as @testobsessed.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!