Things that Find Bugs in the Night: Massive Automated Regression Testing

[article]
Summary:
Nighttime is the right time to run those extra tests you've always dreamed of. In this column, Harry Robinson explains how you can wake up to 40,000 additional tests a day at a cost that won't give you nightmares.

Testing Around the Clock
It's 10 PM. Do you know what your test system is doing? Probably nothing. And the same will be true at 11 PM, at midnight, and through the night. That's a shame. Wouldn't it be nice if you could pick up a few million extra tests in the off-hours? Well, you can, and for minimal cost and effort. Imagine you are testing a GUI where it takes a second to feed in an input and evaluate the result. Even at that conservative rate, a single machine could run through more than forty thousand tests in twelve hours! That's pretty respectable. If you are testing an API, you could churn through a billion tests without breaking much of a sweat. That's better than respectable! A few minutes of thinking and programming before leaving the office could pick up several hours of additional testing for free. And the results are well worth the effort. I love to hear a developer gasp "How on earth did you ever find this bug?"

Variety
Of course, it doesn't do much good to keep running the exact same tests over and over. Your machine must have a way to generate new inputs every time. Here are some ideas:

  • Use a random number generator to choose numbers within equivalence classes.
  • Feed the application random strings by mapping random numbers to a character set.
  • Generate every three-letter combination of printable ASCII characters. That's 884,736 strings!
  • Endlessly press all available buttons. (To find memory leaks, press every button except "Exit".)

Creativity
Oddly enough, providing fuel for round-the-clock testing requires creativity more than it requires programming ability. For instance, to generate random strings, I once tweaked an existing password generator program from the Internet. Within 10 minutes, I had generated millions of input strings that looked like "Mh7-ZyQr?9d3W".

As another example, I tested a system that processed natural language, so I needed to provide almost—sensible sounding phrases to the parser. I downloaded etexts such as Moby Dick and The Hacker's Dictionary from the Internet. This allowed me to test the application with evocative phrases such as "Come forth from behind your cotton bags!" and "Barfulation! Who wrote this, Quux?" respectively.

Elaborations
If you want to increase the effectiveness of these generated tests, you can go several routes:

  • Have your developers add assertions to their code. Assertions check the internal state of the software and make it possible to immediately detect more subtle bugs like data corruption and stack overflows.
  • Vary probabilities in your randomness. For instance, if your test program is exercising a dialog and has to choose between clicking buttons and typing characters into fields, favoring one over the other can dramatically change the behavior of your tests.
  • Use heuristic oracles to determine if results are reasonable. For instance, if you are testing a mortgage calculator, check that the calculated payment is never negative.

About the author

Harry Robinson's picture Harry Robinson

Harry Robinson is a Software Engineer in Test for Google. He coaches teams around the company in test generation techniques. His background includes ten years at AT&T Bell Labs, three years at Hewlett-Packard, and six years at Microsoft before joining Google in 2005. While at Bell Labs, he created a model-based testing system that won the 1995 AT&T Award for Outstanding Achievement in the Area of Quality. At Microsoft, he pioneered the test generation technology behind Test Model Toolkit, which won the Microsoft Best Practice Award in 2001. He holds two patents in software test automation methods, maintains the Web site Model-based Testing, and speaks and writes frequently on software testing and automation issues.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Oct 12
Oct 15
Nov 09
Nov 09