The Two Sides of Software Testing: Checking and Exploring

[article]

You might notice that when we capture acceptance criteria in the form of concrete examples with expectations like this, they look a lot like scripted tests. We use them like scripted tests, too; we check that these expectations are met not just when we first release the feature but also in subsequent regression tests to verify that the system still meets these expectations. Because agile teams value fast feedback and need to run these tests so often, we automate these checks in order to keep the regression testing burden to a manageable level.

When a team takes this approach of capturing the acceptance criteria in the form of tests, it’s reframing Francine’s position even further: “Until we know how we’re going to check this feature, we don’t know enough to build it.” That’s a subtle but critical shift that puts tests at the front of the line instead of relegating them to the end of the cycle. The result of that shift is that tests are not just connected with requirements; they become an expression of the requirements. They're specifications—they specify the expected behavior of the system. When automated, they become executable specifications.

Exploring
No matter how careful we are when we implement a given story, just checking the story against expectations is not sufficient to ensure we’ve covered everything. There is always the risk that we didn’t anticipate some condition or interaction that could result in bad behavior.

Exploratory testing—particularly session-based exploratory testing as defined by James and Jon Bach—gives us a structured way to investigate the system, looking for risks and vulnerabilities. As skilled testers, we use test design techniques and analytical skills to run one small experiment after another, using the information we discovered from the last little experiment to guide the next. As we exercise the system, we vary configurations, permissions, and data. We perform different actions, perform the same actions but in a different order, and perform actions at unexpected times.

As we go, we watch carefully for signs that we’ve stumbled onto a condition that the system does not handle correctly. In-your-face, unhandled exceptions, like an error message reading "ORA-01400: cannot insert NULL," would be one such sign. However, we're also looking for little hints of potential problems, such as lagging response times. Each small surprise might point to a potential risk that warrants further investigation.

For example, on one project, I discovered in my explorations that a field on a form that should have been read-only was actually editable. That's a small sign of something amiss, but it pointed to a much bigger problem. The field represented the groups to which a user belonged. By changing the value in the field, and without having system administration rights, I discovered that I could make myself an administrator. Obviously, this is a problem.

If it had remained uncaught, we would have had a big security hole in our application.

Tested = Checked and Explored
Agile approaches emphasize producing completed features during every iteration or sprint, at least monthly. Before we can call a story “done,” it has to be tested.

Remember the conversation that Francine and I had and consider our different approaches to testing. Francine emphasized checking; I emphasized exploring. Our approaches are the two sides of the testing equation:

  • Checking: Does the system do what it's supposed to do?
  • Exploring: Are there any other risks or vulnerabilities that we haven't thought about yet?

In hindsight, I can see that both Francine and I were right. And, both of us were wrong. Each of us had identified an important aspect of testing, but neither of us saw the whole picture. In order to have enough information to declare a story “done,” we have to check it and explore it. Both sides of testing—checking and exploring—are critical to releasing high-quality software.

Additional Resources

User Comments

1 comment
Xiaomei Tai's picture
Xiaomei Tai

“ Checking: Does the system do what it's supposed to do?

Exploring: Are there any other risks or vulnerabilities that we haven't thought about yet?”

Checking, to me, may be done in a more formal way, compared to exploring. So I roughly divide testing into formal testing and informal testing.

And I think, generally, checking is done first so that we can verify the basic or normal examples of the software, and then exploring is done to explore some deeper hiden bugs. Based on this, I got a statement:"In general, formal testing is done before informal testing." Of course, sometimes, you can first explore before you even know anything about the product. That's the reason I put "In general" here.

I debate a lot with James Bach about this statement. James insists on that at all times informal testing occurs before any kind of formal testing.

After reading your artical, I know maybe I used wrong words in my statement. If I say "In general, checking is done before exploring", is it OK?

November 11, 2011 - 3:16am

About the author

Elisabeth Hendrickson's picture Elisabeth Hendrickson

The founder and president of Quality Tree Software, Inc., Elisabeth Hendrickson wrote her first line of code in 1980. Moments later, she found her first bug. Since then Elisabeth has held positions as a tester, developer, manager, and quality engineering director in companies ranging from small startups to multi-national enterprises. A member of the agile community since 2003, Elisabeth has served on the board of directors of the Agile Alliance and is a co-organizer of the Agile Alliance Functional Testing Tools program. She now splits her time between teaching, speaking, writing, and working on agile teams with test-infected programmers who value her obsession with testing. Elisabeth blogs at testobsessed.com and can be found on Twitter as @testobsessed.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Sep 24
Oct 12
Oct 15
Nov 09