Exploratory testing is a popular testing technique used to enhance a product’s test coverage by helping the team better understand the system and find issues when time is of the essence. Additionally, creative exploratory testing helps break the monotony of running regular scripted test cases and empowers testers to learn simultaneously as the tests are being run.
Despite these benefits, there are some situations in which exploratory testing does not work. Understanding these limitations is important in devising a holistic test strategy for the team.
What Is Exploratory Testing?
Exploratory testing is any testing where the tester is learning and designing tests on the fly. The tester then applies that learning to execute further tests and help improve the quality of the testing effort as well as the product’s coverage. Per this definition, exploratory testing can be done with scripted tests, but the learning element is what is most important. In one instance of exploratory testing, the tester is not guided by existing scripts and he uses his skills to explore the product and learn as he tests; he is not constrained by a pre-defined scope. This thought process aligns more with commonly used definitions of exploratory testing, including a very famous one by James Bach in his article “What is Exploratory Testing?” This article also touches upon the idea of exploratory versus scripted testing, which is what is of utmost importance for us here because it helps to decide when exploratory testing will be of value.
So, when do you really say no to exploratory testing?
Say No When Exploratory Tests Replace Scripted Tests
Given the value and return on investment (ROI) of exploratory testing several startups that often work within budget constraints and cannot allocate the required funds to software testing, resort to using this as their main testing technique to determine the readiness of the product to release. While this is certainly a good start and a better solution than not testing the product at all, these startups need to understand that this is not a perennial solution and exploratory testing cannot replace scripted testing. It is important to balance the two testing techniques so the right coverage is obtained within the cost and time constraints. Startups need to say no to exploratory testing—or, rather, say no to exploratory testing alone—once they have reached a few initial milestones. It’s important for all organizations to document and define the test coverage obtained through the overall execution effort in order to convince the product team and other stakeholders that the product is ready to ship. Organizations should have predefined measures and associated metrics that help confirm a product’s conformance with the exit criteria. This helps establish a larger knowledge base about the product and its quality rather than depending on individual testers for the coverage achieved through exploratory testing.
Say No While Dealing with Compliance Testing
In areas where scripted testing is of utmost significance and deviance from it is not acceptable, you must make the tough decision and say no to exploratory testing. This is especially the case with compliance-based testing, in which you must adhere to checklists, government mandates, and specific domain-based testing where obtaining legal certifications is a necessity. For example, in the field of accessibility testing, where Section 508 and other laws such as Web Content Accessibility Guidelines 1.0 and 2.0 are in place in the U.S., the tester has to go with a predefined checklist of tests and even look at using templates such as the Voluntary Product Accessibility Template that make it easier to test the product’s compliance with defined standards.