Documenting Exploratory Testing

I wrote this article while I was working on a team that was developing medical apps for smartphones and tablets. We were using Scrum and practicing exploratory testing in an FDA-regulated environment. We used the approaches I describe in the article and session-based testing to record what we actually tested. We coordinated with our internal auditor to make sure that what we did aligned with documentation requirements for various regulators.

With mobile, we were able to use the technology on the devices to record video, and we needed lightweight guidance documentation we could take with us on the move.

We had an intern on the project who did a fabulous job with our guidance docs, making sure they were regulation-compliant, auditable, and well maintained. She was able to do this in addition to some great exploratory testing on various devices, and she really enjoyed it.

She wanted to write a paper describing what we were doing for her internship program, but we couldn't find an article that would stand up to university-level citation requirements.

So, I wrote one.

–Jonathan Kohl, July 2012

Just because we are using an exploratory testing approach on our projects doesn’t mean that we don't document our work. With exploratory testing, you can document as much or as little as required by your stakeholders.

Determining documentation requirements on software projects isn’t difficult; it just takes a bit of investigation. For example, while working on a project in a regulated environment, I talked to the people in charge of regulatory issues for the company and asked them what documentation they needed. They said they needed a risk assessment, a test plan, test cases, and test results. I had everything but test cases, so I pressed further. They wanted test cases simply because that’s what they were used to. The regulatory requirements didn’t ask specifically for test case documents; there just needed to be enough documentation so that tests could be repeated by others.

Planning and Strategy
What does atest plan look like in exploratory testing?
Often, test plans that direct exploratory testing look very similar to their scripted testing counterparts. On any project I am leading, I go through analysis and planning to determine how to optimize the use of our people, tools, equipment, budgets, etc. A risk assessment (required by auditors) can help focus our testing.

To create a risk assessment, I do some research. I talk to marketing, sales, and product management and ask them about core functions of the product. What is the purpose of the software? What kinds of tasks would users expect to perform with the software? What would product success or failure look like? 

Next, I talk to the technical team. What features are new? Are there any technology changes that are required, such as new tools, libraries, or hardware? Have they discovered any challenging areas in their work? How do they define success and failure?

I then talk to product management about quality criteria. What characteristics are they expecting from the software? What about technical areas, such as security, accessibility, and performance?

The risk assessment can be a list of items describing how we would mitigate those risks through testing, a focused testing approach for a specific risk, or a recommended technique, such as performance or security testing. Figure 1 shows an example of a risk assessment.

After developing the risk assessment, I look at a test strategy. What is the purpose of our testing? Sometimes, project stakeholders have a clear objective for testing; other times, it isn't as clear. I provide options to stakeholders. "Do you want us to