How to Choose between Exploratory and Scripted Testing


your choices. Good test analysts may not necessarily be effective exploratory testers and vice versa. A nose for finding bugs quickly and efficiently is not a skill that is easily learned. Rigorous analysis and test design are skills that, while sharing similar characteristics, differ significantly in their application. Good documentation makes it easier for a test analyst to come up to speed during the design phase; however, good documentation may not help an exploratory tester become an expert user of a system during testing. So there are many considerations when evaluating the skill set you have available.

Coverage. Some systems and applications are such that the level of test coverage must be accurately measurable and demonstrable. Formal testing through a documented test design has a clear path from each requirement or specification to a test case that is created, reviewed, signed off, and executed (with a signed record of execution). The degree to which a function or process has been tested is therefore known. This detailed documentation of test coverage adds overhead that detracts from exploratory testing's efficiency advantage.

Verification. Verification requires something to compare against. Formal test scripts describe an expected outcome that is drawn from the requirements and specifications documents. As such, we can verify compliance. Exploratory testing is compared to the test engineer's expectations of how the application should work.

Acceptable risk levels. The degree of acceptable risk for a function is directly related to how critical it is to the business. When assessing the test approach (and the amount of test effort) for a particular requirement or area of the system, you must assess the level of risk. Exploratory testing carries a risk in that the coverage of testing cannot be guaranteed. However, this can be mitigated, as discussed later in this document.

Reproducibility. Test comprises essentially two components that can require reproduction: 1) the defects found must be able to be reproduced; and 2) the test itself may need to be reproduced such that the same sets of tests are conducted to give the same result. This second component of reproducibility is less critical but still important for some projects and is a requirement of standards such as ISO 17025.

Comparison of Scripted and Exploratory Testing
Table 1 assesses scripted and exploratory testing on each of the factors discussed above. Where an approach is weak, it is italicized.

Table 1 - Comparison of Scripted and Exploratory Testing

Mitigating the Downsides-A Hybrid Approach
From the above, we could conclude that applications suitable for exploratory testing are those that are simple and common (so domain knowledge is widespread), and which do not require high levels of verification or detailed coverage, and which are generally low risk. Systems that have not been well documented or have little or no lead-in time for testing are also candidates. Common examples would be multimedia CD-ROMs or informational Web sites with little or no unique or complex functionality.

Systems that require scripted testing are high risk (mission critical), have good documentation, and have lead-in time to testing. They also require significant time and skill in terms of resources and detailed coverage and verification. They are often complex systems such as Content Management Systems (CMS) or financial applications such as online banking or insurance systems.

However, not all applications are clear-cut and not every part of an application can be characterized into either category. A hybrid approach may be suitable, where certain functions are tested using scripting with full documentation and others are tested using exploratory testing. At other times, in particular where pressures such as timelines and budget force us toward

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.