Things That Find Bugs in the Night: Massive Automated Regression Testing

Testing around the clock
It's 10 PM. Do you know what your test system is doing? Probably nothing. And the same will be true at 11 PM, at midnight, and through the night. That's a shame. Wouldn't it be nice if you could pick up a few million extra tests in the off-hours? Well, you can, and for minimal cost and effort. Imagine you are testing a GUI where it takes a second to feed in an input and evaluate the result. Even at that conservative rate, a single machine could run through more than forty thousand tests in twelve hours! That's pretty respectable. If you are testing an API, you could churn through a billion tests without breaking much of a sweat. That's better than respectable! A few minutes of thinking and programming before leaving the office could pick up several hours of additional testing for free. And the results are well worth the effort. I love to hear a developer gasp "How on earth did you ever find this bug?"

Variety
Of course, it doesn't do much good to keep running the exact same tests over and over. Your machine must have a way to generate new inputs every time. Here are some ideas:

• Use a random number generator to choose numbers within equivalence classes.
• Feed the application random strings by mapping random numbers to a character set.
• Generate every three-letter combination of printable ASCII characters. That's 884,736 strings!
• Endlessly press all available buttons. (To find memory leaks, press every button except "Exit".)

Creativity
Oddly enough, providing fuel for round-the-clock testing requires creativity more than it requires programming ability. For instance, to generate random strings, I once tweaked an existing password generator program from the Internet. Within 10 minutes, I had generated millions of input strings that looked like "Mh7-ZyQr?9d3W".

As another example, I tested a system that processed natural language, so I needed to provide almost-sensible sounding phrases to the parser. I downloaded etexts such as Moby Dick and The Hacker's Dictionary from the Internet. This allowed me to test the application with evocative phrases such as "Come forth from behind your cotton bags!" and "Barfulation! Who wrote this, Quux?" respectively.

Elaborations
If you want to increase the effectiveness of these generated tests, you can go several routes:

• Have your developers add assertions to their code. Assertions check the internal state of the software and make it possible to immediately detect more subtle bugs like data corruption and stack overflows.
• Vary probabilities in your randomness. For instance, if your test program is exercising a dialog and has to choose between clicking buttons and typing characters into fields, favoring one over the other can dramatically change the behavior of your tests.
• Use heuristic oracles to determine if results are reasonable. For instance, if you are testing a mortgage calculator, check that the calculated payment is never negative.

Costs
What are the costs involved with this approach? Well, other than stretching a few brain cells, there aren't many. That's a big advantage for those who are time-rich but resource-strapped. The economics are very tempting: A computer that is sitting there doing nothing is an idle resource.

It may even make sense to buy machines specifically for running tests around the clock. STQE's 2003 salary survey says that a typical tester earns around $50,000 per year, not including benefits, office, vacation, etc. You can buy a good computer system (Pentium 4, no monitor) for $700 that you can expect to last two years, or $350 per machine per year. Five computers that did nothing but run tests day and