You might notice that when we capture acceptance criteria in the form of concrete examples with expectations like this, they look a lot like scripted tests. We use them like scripted tests, too; we check that these expectations are met not just when we first release the feature but also in subsequent regression tests to verify that the system still meets these expectations. Because agile teams value fast feedback and need to run these tests so often, we automate these checks in order to keep the regression testing burden to a manageable level.
When a team takes this approach of capturing the acceptance criteria in the form of tests, it’s reframing Francine’s position even further: “Until we know how we’re going to check this feature, we don’t know enough to build it.” That’s a subtle but critical shift that puts tests at the front of the line instead of relegating them to the end of the cycle. The result of that shift is that tests are not just connected with requirements; they become an expression of the requirements. They're specifications—they specify the expected behavior of the system. When automated, they become executable specifications.
No matter how careful we are when we implement a given story, just checking the story against expectations is not sufficient to ensure we’ve covered everything. There is always the risk that we didn’t anticipate some condition or interaction that could result in bad behavior.
Exploratory testing—particularly session-based exploratory testing as defined by James and Jon Bach—gives us a structured way to investigate the system, looking for risks and vulnerabilities. As skilled testers, we use test design techniques and analytical skills to run one small experiment after another, using the information we discovered from the last little experiment to guide the next. As we exercise the system, we vary configurations, permissions, and data. We perform different actions, perform the same actions but in a different order, and perform actions at unexpected times.
As we go, we watch carefully for signs that we’ve stumbled onto a condition that the system does not handle correctly. In-your-face, unhandled exceptions, like an error message reading "ORA-01400: cannot insert NULL," would be one such sign. However, we're also looking for little hints of potential problems, such as lagging response times. Each small surprise might point to a potential risk that warrants further investigation.
For example, on one project, I discovered in my explorations that a field on a form that should have been read-only was actually editable. That's a small sign of something amiss, but it pointed to a much bigger problem. The field represented the groups to which a user belonged. By changing the value in the field, and without having system administration rights, I discovered that I could make myself an administrator. Obviously, this is a problem.
If it had remained uncaught, we would have had a big security hole in our application.
Tested = Checked and Explored
Agile approaches emphasize producing completed features during every iteration or sprint, at least monthly. Before we can call a story “done,” it has to be tested.
Remember the conversation that Francine and I had and consider our different approaches to testing. Francine emphasized checking; I emphasized exploring. Our approaches are the two sides of the testing equation:
- Checking: Does the system do what it's supposed to do?
- Exploring: Are there any other risks or vulnerabilities that we haven't thought about yet?
In hindsight, I can see that both Francine and I were right. And, both of us were wrong. Each of us had identified an important aspect of testing, but neither of us saw the whole picture. In order to have enough information to declare a story “done,” we have to check it and explore it. Both sides of testing—checking and exploring—are critical to releasing high-quality software.