Manual software testing can never catch all errors–so can automation help? David Norfolk looks at the pros and cons of automated testing and offers advice–and warnings–on its use.
Manual software testing can never catch all errors - so can automation help? David Norfolk looks at the pros and cons of automated testing and offers advice–and warnings–on its use
One thing you say for sure about software testing is that there is not enough time in the world to test manually all possible paths and combinations of paths through a realistically large–that is, useful–program. The tests will never be anything like exhaustive, so they must be made cost efficient: you must find the greatest number of defects with the resources available.
This has consequences. First, you cannot afford to run tests that do not find defects, so structured testing plans that maximise the chances are vital.
Second, automation, to maximise the use of resources, makes lots of sense–but only if the automation fits the demands of the structured test plan.
The basic principles of automated testing go back to earlier keystroke recording utilities. If you record the keystrokes made by someone entering a test case, you can rerun the test case at very little cost, perhaps to check that a defect that caused the test to fail on its initial run has now been fixed. In fact you now have a stored regression test that can be run after any code maintenance, to ensure that unexpected errors have not been introduced elsewhere in the program while part of it was being changed.
But you can do better than this. One automated test script can be cheaply cloned into dozens of scripts, each testing for a similar type of defect: for example exploring boundary conditions for an input string: too small, zero, too big, negative.
An important facility in any automated testing tool is a repository of test specifications. These can be developed early, before coding starts, and used to develop 'what if' scenarios for validating requirements specifications and so on. The most cost-effective time to find and remove defects is during requirements analysis–so requirements analysis tools can usefully be considered as part of the automated testing toolset, although they do more than just help validate requirements.
Test specifications in an automated repository are extremely cost-effective, because they can be used to generate test cases regardless of the maintenance state (database record formats and so on) of the system. Stored test cases can be rendered almost useless if the format of the records being processed changes. In addition, the test specification should make the business justification for performing a set of tests clear.
An interesting new application of automated testing is in extreme programming. Every functional requirement has its associated test case–prepared before coding–and these are run regularly, and the results published, to keep developers focused on the user requirements. Unit test cases are also prepared for all code and regression test cases for all bugs (to prevent their return) and code must pass all the tests before it is released. Managing and running all these tests would be impractical without automated test tools.
Automated testing is a real godsend to development quality in general, and although the basic principles still apply, the products have become increasingly sophisticated.
Nevertheless, automation has its limits and certainly has not de-skilled the process. It is easy for the inexperienced to confuse high volumes of test results–easily produced with automated tools–with effective testing.
The theories of testing talk of 'equivalence classes': test cases which all find the same general type of defect. If a program accepts integers between 1 and 9, for instance, integers between 2 and 8 form an equivalence class: if it processes one of them it will probably process the