own writing and teaching is full of advice on how to select the right set of inputs. But I also counsel testers to buck up and face infinity anyway. The method of doing this: massive-scale random testing. It is a tool that should be in every tester's toolkit and few, if any, testing projects should be without it.
Massive-scale random testing must be automated. Although it isn't easy to do the first time, it does get progressively easier with each project and eventually becomes rote. It may not find a large number of bugs, but it is an excellent sanity check on the rest of your testing: If you are outperformed by random testing, you may have a problem on your hands. And I am always pleased with the high quality (albeit few in number) bugs that random testing manages to find.
Another reason to apply massive-scale random testing is that setting up such tests requires a healthy knowledge about the input domain of the application under test. Testers must really get to know their inputs and the relationships that exist among the inputs. I almost always find bugs and get good testing ideas just from the act of planning massive-scale random testing.
2. Thou Shalt Covet Thy Neighbor's Apps
This commandment sounds a bit perverted, but I can assure you it has a "G" rating. The idea I am trying to get across here is not to test your application in isolation. Otherwise, you might run into the nightmare scenario of "application compatibility," or more specifically, lack thereof. Application compatibility, or "app compat" as it is widely abbreviated, means that one application does something to break another one. In case you are not sure, that is a bad thing.
One way to combat this problem is to keep a cache of apps (old ones, new ones, borrowed ones, blues ones-the more diverse the better) and make sure that you run your app at the same time each of these are running. Of course, we want to do this with operating systems as well. A user should not have to tell you that your application won't run with a specific service pack installed; that is something you should find out in your own testing.
So covet those apps and those service packs: The more the merrier!
- Massive-scale random testing is a subdiscipline of model-based testing. Learn more about both at www.model-based-testing.org
- Oracles, Failures, Models, and Sins: The Ten Commandments of Software Testing, Part 2
- Test-Physicians, Source Code, and BSODs: The Ten Commandments of Software Testing, Part 3