Getting Automated Testing Under Control

Testing of systems is probably the most difficult task there is in IT--especially if you want to execute the tests automatically. We make our living running test projects for customers, and have experienced many of the potential problems in testing and test automation ourselves: 

• Testing costs time and money (usually during the end phase of a development project, when such time and money is least available) 
• Manual execution of test cases is often a tedious and error-prone task (especially when tests have to be repeated on successive versions of a system) 
• Automated tests tend to be very sensitive to changes in the target systems interface 
• It can be difficult for outsiders to get a clear overview of what is being tested (especially when using automation scripts that non-IT people find hard to understand)

During the years we have evolved an approach to counter these issues. Although you shouldn't regard it as a "silver bullet," we think it can be very practical; we would like to use this article to share it with you. The approach is marketed under the name TestFrame™, but you can easily apply the techniques and principles outlined in this article yourself.

Test Clusters
Let us start with a look at what we have called the "test analysis process." The first step is to identify so-called test clusters -collections of tests that have more or less the same scope and level of detail. Even for big projects the number of test clusters should never be very large; typically there should be no more than two dozen of them for each project.

For a banking application, you might expect something like the following list of clusters:

• Tests of the User Interface   Are all screens, fields, and controls accessible? Do they work as they are supposed to? Are the proper help screens displayed? Does the tab bring the focus to the proper next fields? 
• Tests of the Entry Process   Are the mandatory fields really mandatory? Are the proper default values displayed? Do the right values appear in list boxes? 
• Tests of Relationship Management Functionality   Do the customer names you've entered really appear in the database? Can they be updated, removed, etc.? 
• Tests of Payments and Transfers   Are all possible forms of payments and other money transfers working properly? Is all data, like account balances, updated appropriately? 
• Tests of Interest Calculations   Are mathematical calculations processing data as you had expected?

The table below (see Table 1) is a list that might help you decide about the division of the testing task into clusters (don't see this as the law; your project may require different priorities). The rows are in descending order of importance.

You might begin creating test clusters by separating module tests, system tests, functional tests, and performance tests. Then you might further divide along lines-such as the part of the system that is being tested, or the department that has to be involved in the test development.

We always try to make at least a provisional list of test clusters one of the first actions in a project. If possible we do this together with the users and other involved parties in a joint session with a whiteboard. Listing intended test clusters is a high priority because it's a very good basis for planning and organizing the rest of the testing project. For each cluster, for example, we can decide on its priority and who will do the work. We can also track progress of development and execution by cluster.

Test Design for a Cluster
In the TestFrame approach all further test development is done at the level