- from requirements to test. This has a positive impact on the time and cost to fix problems.
Model-based testing delivers high reliability without adding cost to the requirements and testing phases. Once an organization achieves this level of reliability, it can address other aspects of the software development process without fear that software reliability will be compromised. For example, the organization can try new budgeting and project management techniques or pilot new software development tools. New users of model-based testing often address requirements specification formats and processes with a goal of reducing the number of ambiguities raised by the test modelers.
Several case studies have been conducted to quantify the value of the model-based testing approach.
E-Commerce Enterprise. In A
Case Study in Extreme Quality Assurance Jim Canter and Liz Derr provide an extensive analysis of the impact of an enterprise-wide implementation of model-based testing at an e-commerce firm. In their analysis, the number of post-implementation defects in software using traditional requirements analysis and testing techniques averaged eight to ten per implementation. This defect rate was independent of the number of people testing the application, which suggests that added test resources don't necessarily result in finding more defects. Application of model-based testing resulted in implementations with zero to three defects per release. In addition, the number of defects discovered during test execution also dropped, indicating that the model-based ambiguity reviews were effective in identifying and resolving requirements issues before they had the opportunity to become defects in the actual software.
Commercial Banking Project. Software Prototype Technologies undertook mplementation of model-based testing in a project for a major U.S. bank. The business objective was to create a software system that would allow business banking customers to be able to view and manage their accounts online via the Internet. The project structure for this effort was particularly complex and involved two business banking user organizations, five existing software systems, an off-shore software development vendor, QA staff from the corporate IT department, and a separate requirements management/test modeling team (creating both the specifications and the test model). Model-based testing processes were applied to the core application under development but not to interfacing systems being modified (these systems used their existing specification and testing procedures). The core application accounted for about 70 percent of the total functionality
Five weeks after production implementation, a breakdown of reported production issues was developed and is shown in the following chart:
In this chart, "non-defect reports" represent reported problems that were, in fact, correct as defined in the specification. "Data conversion" issues resulted from unanticipated production data configurations.
Hardware Configuration Engine. In another example, Sun Microsystems implemented an online rule-based software engine to assist their worldwide sales staff in configuring hardware suites for customers and subsequently generating price quotes. The hardware configuration options and rules are extremely complex and change frequently as new products are introduced and older products discontinued. Accepted quotes are transmitted directly to the factory floor for assembly, making software defects that allow invalid configurations very costly.
Organizationally, this project was another example of fragmented roles and responsibilities. A third-party vendor provided software development. Separate marketing groups and engineering specialists defined requirements for each product line. The resulting requirements documents frequently conflicted and always assumed that the audience was very knowledgeable of the products. Testing and configuration management were handled by a separate in-house QA group. Further, product releases and competitive pressures dictated that new software be released to the worldwide sales staff on a monthly basis.
The project transitioned to a rule-based testing approach after defect rates (including invalid configurations) reached