Editor's Note: This article was published Feburary 25, 2003.
Exploratory testing has gained increased recognition as a valid testing methodology. As a test manager or engineer, you may be considering whether or not to use exploratory testing on your next project. However, as with other aspects of testing, such as automation, you must choose your method carefully to ensure a successful outcome. In this article we examine the factors you need to consider when making that choice.
In his article "What Is Exploratory Testing?" James Bach defines exploratory testing as "test design and test execution at the same time." In contrast, planned, or scripted, testing has a test planning and design phase, followed by test execution. The products of the test design phase are generally test plans, test specifications, and test cases and scripts. These products are then used during test execution to conduct the formal testing of a system.
In this document we will examine the factors influencing the effectiveness of both approaches, as well as combination and hybrid approaches, to select the most effective and efficient methods within the constraints of time and resources.
Factors to Consider When Choosing a Testing Approach
What are the factors we need to consider when choosing our approach? The main ones are listed below:
- tester domain knowledge
- system complexity
- level of documentation
- timeframes and deadlines
- available resources
- skills required
- coverage
- verification
- acceptable risk levels
- reproducibility
Let's examine each of these more closely.
Tester domain knowledge. Simply put, how well does an individual test engineer understand the operation of the system and the business processes that system is supposed to support? If a tester does not understand the system or the business processes, it would be very difficult for him to use, let alone test, the application without the aid of test scripts and cases. Likewise, in user acceptance testing (UAT), where business users are conducting testing, a new interface may require the use of formal scripts or training to orient the users before they can test the system. A simple system such as an information Web site, on the other hand, is well understood by a professional tester, making it a prime candidate for exploratory testing.
System complexity. The complexity of the system can also affect its suitability for exploratory testing. The user needs to understand how to use the system. Where there is a high level of dependency between functions for end-to-end testing (i.e., one process depends on the data created by another), detailed test planning is required. End-to-end testing can be accomplished with exploratory testing; however, the capabilities and skill sets required are typically those of a more experienced test engineer.
Level of documentation. Scripted testing generally flows from business requirements documents and functional specifications. When these documents do not exist or are deficient, it is very difficult to conduct scripted testing in a meaningful way. The shortcuts that would be required, such as scripting from the application as it is built, are accomplished as efficiently using exploratory testing. We will discuss later how some degree of formal planning can be accomplished.
Timeframes and deadlines. The lead-in time to test execution determines whether you can conduct test design before execution. When there is little or no time, you might at least need to start with exploratory testing while documented tests are prepared after the specifications become available.
Available resources. The total number of person-days of effort can determine which approach you should take. Formal test documentation has significant overhead that can be prohibitive where resources and budgets are tight.
Skills required. The skill sets of your team members can affect your choices. Good test analysts may not necessarily be effective exploratory testers and vice versa. A nose for finding bugs quickly and efficiently is not a skill that is easily learned. Rigorous analysis and test design are skills that, while sharing similar characteristics, differ significantly in their application. Good documentation makes it easier for a test analyst to come up to speed during the design phase; however, good documentation may not help an exploratory tester become an expert user of a system during testing. So there are many considerations when evaluating the skill set you have available.
Coverage. Some systems and applications are such that the level of test coverage must be accurately measurable and demonstrable. Formal testing through a documented test design has a clear path from each requirement or specification to a test case that is created, reviewed, signed off, and executed (with a signed record of execution). The degree to which a function or process has been tested is therefore known. This detailed documentation of test coverage adds overhead that detracts from exploratory testing's efficiency advantage.
Verification. Verification requires something to compare against. Formal test scripts describe an expected outcome that is drawn from the requirements and specifications documents. As such, we can verify compliance. Exploratory testing is compared to the test engineer's expectations of how the application should work.
Acceptable risk levels. The degree of acceptable risk for a function is directly related to how critical it is to the business. When assessing the test approach (and the amount of test effort) for a particular requirement or area of the system, you must assess the level of risk. Exploratory testing carries a risk in that the coverage of testing cannot be guaranteed. However, this can be mitigated, as discussed later in this document.
Reproducibility. Test comprises essentially two components that can require reproduction: 1) the defects found must be able to be reproduced; and 2) the test itself may need to be reproduced such that the same sets of tests are conducted to give the same result. This second component of reproducibility is less critical but still important for some projects and is a requirement of standards such as ISO 17025.
Comparison of Scripted and Exploratory Testing
Table 1 assesses scripted and exploratory testing on each of the factors discussed above. Where an approach is weak, it is italicized.
Table 1 - Comparison of Scripted and Exploratory Testing
Mitigating the Downsides-A Hybrid Approach
From the above, we could conclude that applications suitable for exploratory testing are those that are simple and common (so domain knowledge is widespread), and which do not require high levels of verification or detailed coverage, and which are generally low risk. Systems that have not been well documented or have little or no lead-in time for testing are also candidates. Common examples would be multimedia CD-ROMs or informational Web sites with little or no unique or complex functionality.
Systems that require scripted testing are high risk (mission critical), have good documentation, and have lead-in time to testing. They also require significant time and skill in terms of resources and detailed coverage and verification. They are often complex systems such as Content Management Systems (CMS) or financial applications such as online banking or insurance systems.
However, not all applications are clear-cut and not every part of an application can be characterized into either category. A hybrid approach may be suitable, where certain functions are tested using scripting with full documentation and others are tested using exploratory testing. At other times, in particular where pressures such as timelines and budget force us toward exploratory testing, we need to mitigate the weaknesses of the approach. Mitigation strategies include:
High-level scripts or use cases. Use cases or high-level scripts provide a set of actions for the tester where a guarantee of test coverage is necessary. The experience, skill, and knowledge of the tester is still relied on, but there is some level of comfort with this documentation.
Checklists. An even higher level of documentation is a checklist that specifies processes and sections of the system as a single item to be tested. The depth and manner they are tested are still open, but a level of assurance can be achieved.
Creating scripts during execution. Testers can create a test script as they execute it, though normally without the expected outcome. The actual outcome, however, may be documented for review. This gives a detailed and formal manner to demonstrate coverage, but it also increases the overhead for testing.
All of the above are simply about reducing the level of documentation, and hence overhead, associated with scripted testing while getting some of the benefits of the more formal approach. Use cases and checklists are a means for the business or test manager to nominate and ensure coverage of the higher risk areas of the application and provide a certain level of comfort. Use cases and checklists are particularly appropriate where little or no documentation exists and proper test analysis is difficult or impossible. This allows information and priorities to be passed directly through the testing process without having to reverse engineer missing information.
Conclusion
The benefits of exploratory testing are increasingly becoming recognized. When employing it, however, as with all testing, we need to examine the testability and risks associated with each area of the application and choose the most effective and efficient approach throughout the application. We also have to recognize the limitations in time and resources that constrain most test efforts. By analyzing the factors discussed in this article for each area of your system, you should be able to make an informed choice.
