Exploratory testing (ET) "in lieu of/or complimenting" scripted testing continues to be a hot topic of discussion in the industry. If you have recently attended any testing conference or event, read any recent publications, or participated in any QA professional local association meeting, you have probably witness some type of "heated" debate about this topic, with strong proponents of one approach versus the other. I personally believe that, even thought the concept of exploratory testing has been around for a while, there is still a lot of confusion and misperceptions about what ET really is, and the value it can provide to any testing organization.
The purpose of this write-up is to take a closer look at exploratory testing and scripted testing and it's respective advantage and disadvantage (yes, there are benefits and drawbacks on both approaches), and provide some recommendations and guidelines about when you should consider using which type of testing approach. On an additional note, because Empirix offers tools to help you manage and organize your manual testing efforts, as well as automation and performance testing tools, we feel that we are well positioned to provide an unbiased and "neutral" point of view when looking at both types of testing types.
What is Exploratory Testing?
In his article, "Exploratory Testing Explained" 1, James Bach defines exploratory testing as, a "simultaneous learning, test design, and test execution". One could think about exploratory testing as a technique—or a sophisticated, thoughtful, and creative approach— to perform ad hoc testing without having a pre-defined set of test cases. Exploratory testing is not new; in fact, exploratory testing as defined by Cem Kaner 2 has been around for 23 years.
In "freestyle" exploratory testing, the only result of the testing activities is a set of bug reports. In "session-based test management" exploratory testing, a set of written notes for managing and measuring tests are produced at the end of each exploratory session, which are then debriefed by a test lead. And as Jon Bach mentions in our QAZone Interview, a session is a unit of time roughly 45 minutes to 3 hours long where the tester is guided in their exploratory by a charter or mission statement 3. In both "freestyle" and "session-based management" exploratory testing, no specific results are expected; the tester decides what needs to be verified, critically investigating the accuracy of results as needed. In addition, because there are not too many formal processes in place, and there are certainly less associated QA documents and overhead, exploratory testing can provide meaningful results quickly. (One common conviction among the exploratory testers community is their belief that writing down test scripts could disrupt the intellectual processes needed to find important bugs quickly 4).
Thinking of my time as a Software QA Engineer, I realized that we have all done exploratory testing at some point in our professional career, without realizing that we do it –unless you simply don’t create tests at all. Think about your first weeks as a new QA team member, and your tasks then…You were probably exploring product documentation and exercising the product to learn it. In fact, the majority of your written test procedures were probably created through a process of performing exploratory testing, while investigating the new features and functionality available on a new product release. So if exploratory testing is so common, why is it still the subject of many industry discussions and debates, and even a little bit of controversy?
I really think that the problem here is that folks try to generalize and evangelize one versus the other–exploratory testing OR