The Magic 8 Ball of Testing

Over the years, there has been some progress on automating parts of the development and testing process. The nUnit family of test automation frameworks has caused a revolution in unit testing and has made the world safe for refactoring. FIT and FitNesse help with integration-level tests, and Web automation tools, such as Perl's WWW::Mechanize, Watir, and Selenium, show a lot of promise in aiding the testing effort at the highest levels of the application. All of these tools provide support for tests that we've thought of already, but none of them helps with the job of defining and refining requirements such that programs are automatically more testable.

To my surprise, this spring I was introduced to such a tool. "It's a little joke-can you spot the anagram?" asked Dr. Ralf Piolo. "It's called oClear. That's an anagram for 'oracle'-an oracle is a principle or mechanism for recognizing a problem, so a really clear requirement statement is one of the best oracles." Dr. Piolo had invited me to his lab at the University of Bala in Ontario, Canada, to see a demonstration of oClear. The program is a high-level requirements tool that he has been working on with several of his graduate students. The tool accepts requirements in natural language, parses them, and provides a complete analysis and a suite of improvement suggestions as output, also in natural language.

The interface of the prototype that Dr. Piolo showed me was very lean and clean-a single dialogue, with tabs for the various analysis features. The tool recognizes variables, ambiguities, equivalence classes and boundary values, and project-costing problems. It also suggests test script ideas and identifies expectations as to where bugs will be found.

Dr. Piolo encouraged me to submit a sample requirement. I proposed a requirement that I thought might fool the system: "Send a reminder notice to every customer who has not paid in thirty days." I entered it into the requirements parser. "Good," said Dr. Piolo. "Let's start by clicking the tab labeled Disambiguate." In the output window, the application reported:
Actor Ambiguity: oClear suggests that only those customers who owe money should receive reminder notices.

Rephrase Recommendation 1: "Send a reminder notice to every customer who has had an outstanding balance AND has made no payment for at least thirty days."

I was astonished. Not only had oClear identified an ambiguity but it also had proposed a reasonable solution for getting around it-and had put the solution into a well-structured sentence to boot. "Press Continue Analysis, and then press Disambiguate again," said Dr. Piolo. I did.

Actor Ambiguity: oClear suggests that the term "every" is ambiguous, as some customers will fall into exceptional categories, such as "privileged," "inactive," or "bankrupt."

Rephrase Recommendation 2: "Send a reminder notice to each active, regular-status customer who has had a negative balance AND has made no payments for at least thirty days."

Warning: Customer special-status categories must be defined.

This was really impressive. I wanted to press Disambiguate again, but Dr. Piolo wanted to show me more of the application. "Try pressing Boundary Analysis," he suggested.

Date Ambiguity: oClear suggests that assessment periods of more than twenty-eight days fail over February/March boundary.

Date Recommendation 1: Alter requirement to calculate payment due based on the [ ] th day of every month (or first business day thereafter).

Date Recommendation 2: Alter requirement to calculate payment due based on four-week period. (Choose day of week [Monday] and start date [yyyy/mm/dd]).

Date Recommendation 3: Alter requirement to calculate payment due based on [day of week] in 4/4/5 week intervals. (Choose start date: [yyyy/mm/dd]).

"I notice that