specialist translates the specification into a graphical model of the processing logic, inputs, and outputs. The graphical approach is an adaptation of Cause-Effect state model diagrams, with each cause and effect representing a component of the business processing logic that the software must execute. The modeling process immediately identifies any inconsistencies and ambiguities. Missing causes, missing effects, or unclear interactions between causes and effects as documented in the specifications are clearly exposed. For example, the specification may state:
"…Add input value A to value B. This number must be positive…"
This raises the following questions:
- Which number must be positive?
- What happens if the value is not positive?
The test modeling process is design-neutral; however, awkward user interfaces and other indicators of poor design tend to be exposed by virtue of their excessive complexity in the modeling.
All such questions about the exact requirements are documented as "specification ambiguities" and passed to the appropriate analyst or designer for resolution. The ambiguity review process has proven extremely effective in finding problems in specifications. It differs from normal specification walkthroughs because the review is done in the context of creating a rigorous logical model of the expected behavior. Ambiguities are identified and documented by the test engineer doing the modeling work. Modeling expertise on the part of the business analysis or interface designers is not required and, frequently, these groups are unaware of the test models being constructed. Thus the project receives the benefits of a rigorous software engineering process at a key leverage point in the lifecycle without requiring broad team knowledge of the process.
When the test model is complete, COTS (commercial, off-the-shelf) software is used to generate detailed test scripts directly from the model. In principle, any test design software that evaluates logic structures could be applied, but my company uses the Caliber-RBT product, which generates test scripts based on stuck-at-one, stuck-at-zero coverage criteria. A number of studies have shown that the stuck-at-one approach offers a high level of logical coverage. By running the test design tool directly against the completed test model, detailed test descriptions are immediately available, removing the requirement for a separate, manual test case design effort. Instead, the test scripts that will exercise the specified functionality are available at the same time the specification itself (cleansed of ambiguities) is published. These test scripts also confirm the expected behavior of the proposed system, either in additional walkthroughs or as a separate reference document. The scripts are also immediately available to QA teams for the purpose of creating appropriate test data and beginning test execution planning.
Model-Based Testing Results
The model-based testing process yields extremely high reliability production implementations, independent of the type of application or the platform on which it is implemented. Typically, applications will encounter few, if any, post-implementation defects in functional logic. This does not imply that the software is "proven" correct; black box testing of any kind will not necessarily detect defects in coding structures that are dependent on specific execution paths or sequences. However, as a practical matter, the process is highly effective for the following reasons:
- The modeling process forces a rigorous examination of the logical consistency of the specification, the largest root source of software defects.
- Automatic design of test scripts directly from the model removes the requirement for manual test case design, thus removing a manual link in the communication chain through which errors can be introduced.
- Automated test scripts always provide full functional coverage. This cannot be guaranteed when individual test analysts with varying experience and commitment levels design the tests.
- The process significantly shortens the feedback loop