A Look at McCabe IQ: Metrics Analysis and Code Coverage

For the past eight months, I have been working with a Fortune 100 client to establish a QA department from the ground up. In addition to staffing and many other tasks necessary to get a new department underway, I have evaluated a wide variety of QA tools. One of the products that I have assessed, IQ 6, provides code metrics and code coverage capabilities. The opinions expressed here are mine alone, and they are neither an endorsement nor a critique of these products by YMS Associates.

Product Summary
McCabe & Associates produces a variety of tools for software development and QA under the umbrella title of McCabe IQ. While the product offers a range of component tools presented through a unified interface, individual components are also available in combinations to suit the needs of the customer. My review focuses on the McCabe IQ core product used in conjunction with the McCabe QA and McCabe Test components. This combination provides very powerful code analysis metrics, code coverage, and module-level unit and integration test case generation.

Metrics Analysis
Code metrics are static measurements of the code itself, and they are independent of software test execution. The simplest example of a code metric is the number of lines of code in an application. It is not necessary to understand the language of the code in order to leverage the power of code metrics.

The McCabe metrics originate from U.S. government-sponsored research by the company’s founder, Thomas J. McCabe. These metrics allow for quantitative measure of software code (currently McCabe supports Ada, C, C++, COBOL, Fortran, Java, and Visual Basic), and are available at the application or module level. Some of McCabe’s metrics quantify complexity (Cyclomatic Complexity), level of (un)structure (Essential Complexity—a.k.a. the "spaghetti factor"), and level of integration with other modules (Design Complexity). The tool also includes dozens of other metrics, including Halstead metrics, object-oriented metrics, lines of code, and lines of comment—as well as additional metrics originated by T. J. McCabe. These metrics can help a QA team objectively assess the relative complexities of several applications. Any QA department faced with more work than available resources can use these metrics to identify applications that are unstructured and more complex, and target those applications for more resources than the structured and less complex ones. With the McCabe QA component the user can track metrics trends over time and even create custom metrics based on custom criteria.

McCabe asserts that the Cyclomatic Complexity for a module is also the number of test cases necessary to perform a unit test on that module. (Although this review is not the appropriate venue to cover these phenomena in detail, it should be intuitive that a highly complex module will require a larger number of test cases than a simple one.) Similarly, the Design Complexity measure is equal to the number of integration test cases for that module. There are a number of ways in which these measurements can be used as predictors; outside research has correlated McCabe metrics to statistical likelihood and frequency of software defects in the code.

McCabe & Associates (www.mccabe.com) produces a variety of tools for software development and QA under the umbrella title of McCabe IQ. While the product offers a range of component tools presented through a unified interface, individual components are also available in combinations to suit the needs of the customer. My review focuses on the McCabe IQ core product used in conjunction with the McCabe QA and McCabe Test components. This combination provides very powerful code analysis metrics, code coverage, and module-level unit and integration test case generation.

Test Generation
The McCabe Test component generates module-level test cases for specific logical paths through the module (for unit testing), or test cases for specific logical paths in the module that interact with other modules (for integration testing). For each generated test case McCabe IQ presents a graphical display highlighting a specific path through the code. Next to the graphical representation appears the line (or lines) of code containing the logic necessary to traverse that specific path, along with the outcomes that must be achieved to follow the intended path. I feel that this functionality is more of a benefit to developers than to testers, because it is limited to module-level testing and requires a thorough knowledge of the application code to be able to implement the test cases.

Code Coverage
The code coverage portion of McCabe IQ is available in the McCabe Test component. This tool functions similarly to many code coverage tools. Hooks are placed in a copy of the source code (also known as code instrumentation), which is then compiled. During software testing the application under test generates a trace file which is read into McCabe IQ for analysis, and displays are generated identifying those logical paths which have been traversed during testing. Coverage metrics such as path coverage, branch coverage, code coverage, Boolean coverage, and class coverage are also generated. (Be aware that instrumented code should not be used for stress tests or load tests due to degraded performance.)

Interface
McCabe IQ's GUI interface is known as the "Battlemap." Each module of the code is represented individually, and is color-coded by configurable metrics thresholds (green for low complexity; yellow for medium complexity; red for high complexity, for example) using any McCabe or custom metric—either individually or in combinations. During code coverage, color-coding represents the level of coverage completed during testing. The Battlemap employs a challenging interface that is sometimes counter-intuitive. Access to information is often buried in hard-to-find locations. Similarly, the code of an average application creates a Battlemap so large that it is impossible to see the “big picture” unless you zoom out to an extent that makes the display meaningless. (For this reason, I recommend using a 21" monitor or larger.) Don’t attempt to print the Battlemap—you'll spend all day taping together a matrix of pages that could easily span twenty or more sheets.

Pricing & Support
The price of McCabe IQ is substantial. The list price for a typical setup including the core McCabe IQ product, QA Server and Test Server components, C++ parser, and three floating licenses is $45,000. Another issue, particularly with languages that do not have ANSI standards, is the McCabe IQ tool’s ability to "understand" your particular dialect of the language. Expect to spend time on the phone to their Technical Support with your compiler manual in hand. Consider establishing a nondisclosure agreement and sending Technical Support a copy of the code for one application; this will make getting started with the IQ tool much less painful. Once you have succeeded in using the product for one of your applications, the remainder of your code should not require nearly as much, if any, intervention from McCabe's support staff. McCabe & Associates' Technical Support is excellent and is a major asset to its product.

Final Analysis
From a QA perspective, I feel that the true strength of McCabe IQ is in code metrics analysis. It has more features than competing tools do. The ability to identify applications, or even specific modules, which are prone to failure is a tremendous advantage for any QA department. I encourage developers and development managers to use the McCabe IQ tool to enhance their development process and deliver better software to QA—because I have seen more than one development manager raise an eyebrow at modules that are highly complex and highly unstructured, and instructed their developers to review the code and ascertain the necessity of the current code design.

The code coverage and test case generation capabilities of McCabe IQ are useful. If you are only looking for a code coverage tool, consider evaluating other products with similar features that are available at a lower cost. McCabe IQ is a costly investment, but the potential payoff is worth every penny.