default timeout, but some test drivers can overwrite defaults.
Test distribution between different machines can be supported, too. If you have a test lab with more than one machine, you can either distribute tests manually by dividing them into a few groups and running different test groups on different machines, or you can write scripts that distribute tests. In this test system, you have one test server and a few test clients. The main script is started on the server. A "client_ready" script is started on each client. The client_ready script does the original setup on the client computer and then takes "orders" from the server script. A script run on the server sends the test driver to the client. The client runs the test driver and passes the pass/fail (expected/unexpected) result to the server. The server script gets information from the client and reports the results. It also checks to see if the client machine is available to start a new test.
If the software you are testing is supported on multiple platforms, your test system should run on multiple platforms as well. Our synthesis compiler, for example, was working on the UNIX, Win95, Win98, WinNT, Win2000, and WinXP platforms.
Problems We Experienced Developing and Using Our Test System
- We didn't budget for the changes. We were just "sneaking the changes in." So when we hit release mode, the test system was not ready and was very unreliable. It began breaking in several places and it was a disaster. Management was very disappointed. It took us some additional time to clean things up.
- Only three people in our software group used the system. That was a huge mistake! More people could be familiar with the test system if we had better documentation sooner. And the system should have been available to developers so they could use it for their testing.
The system I described in the paper is very flexible. I have used this test system now for several years. It has demonstrated (after it became stable) significant advantages compared to other systems.
If you want to implement the system I described in the paper, learn about two tools:
- Software Testing Automation Framework (STAF) at staf.sourceforge.net. STAF is an open source, multi-platform, multilanguage framework designed around the idea of reusable components.
- askMaster, developed by my former coworker Phil Tomson.
Acknowledgements to Phil Tomson for important contribution to this work.