much simpler than automating a complete suite of GUI tests. The monkey test caused several crashes in the application, some of which I had never seen before. As a bonus, when I watched the monkey at work, I discovered a few obscure corners of the GUI interface I didn't even know were there.
Thrash It With Applescript
The application that endured the torture of my monkey script also had a Mac OS X version. I wanted to develop a monkey test for the Mac version too. I researched my options and found that Applescript would do GUI automation, and I could do similar things in Perl using the Mac's Open Scripting Architecture. I had to install a large number of additional modules to do this in Perl, which was compounded by a configuration problem with Perl's mechanism to automatically install modules on Mac OS. I fixed the configuration problem, but knew that other people trying to run my Perl code probably would have to apply the same fix.
I had to make a choice between using my favorite language and learning to do Applescript-like things without knowing Applescript, or biting the bullet and learning enough of the language to write the test directly in Applescript. Past experience told me I needed to first learn the native language before I could be productive in either one, so I started learning the basics of Applescript.
Within a few days I had a simple script. The application had a tendency to crash when I changed it from one username to another, so I decided to simply write a reliability test that would switch back and forth repeatedly between two usernames. The test would cause a crash within a few minutes every time I ran it--though I had not yet figured out how to synchronize properly with the application to know when it was ready to receive input. I just used fixed timing delays, which made the script slower, and not guaranteed to work every time. But I deemed it good enough, and sent it to the developer.
It turned out that our Mac developer was already familiar with Applescript, so he was happy with the route I had chosen. I haven't yet developed a complete monkey test on the Mac, because my simple script is keeping the developer busy with bug fixing for now.
Adjust When You Oversimplify
One of my earlier attempts at writing a simple robustness test turned out to be a bit too simple. I was testing a Unix-variant operating system with a C program that called randomly chosen system calls with random parameters. Shortly after I started the test for the first time, the system rebooted. "Yes!" I shouted, thinking I'd found a serious bug. But when I examined the test log, I found the test had called the "reboot" system call and was running as an administrative user that had permission to reboot. So the system did exactly what it was supposed to do.
I added a bit more infrastructure to the test to ensure that it didn't run as an administrative user. Even though this limited the test coverage, it was the simplest way to get my test running again. Later, my tests found problems in the operating system.
Keep it Simple
I like to follow this formula when I am testing reliability and robustness in a software product:
- Use the simplest approach that's likely to find the kind of bug you're looking for.
- Learn just enough about the technology involved to code the parts that need to be automated.
- Make the automation robust enough so that