At a conference discussion on high-maturity processes that I attended, the most frequently asked question was "How many control charts do I need to be Level 4?" Perhaps a better question would have been "How can we use control charts to improve our business performance?" An even better question is, "How can Level 4 improve our business?"
Readers unfamiliar with the CMMI ® should scan other articles on StickyMinds.com or reference material at Software Engineering Institute . I will not talk about the techniques of various statistical methods; those desiring an in-depth explanation of statistical methods should look to books or articles that explain these topics in detail. The focus of this article will be to understand why statistical methods are useful, how to use them, and some of the common pitfalls I have seen in organizations that overlook the key question "How does this benefit our business?"
The phrase "business objectives" is used thirteen times in the CMMI Level 4 process area descriptions, providing clear direction for implementing Level 4. In a nutshell, Level 4 is about developing performance baselines-improvement objectives that are quantitative (numerical) improvements of the performance baseline measurements, analysis techniques that measure process performance (control), and the prediction of project results against estimates.
The term "quantitative project management" is one of the process areas at Level 4. In the most basic definition, it means that you can accurately estimate using past performance baselines; make commitments for cost, schedule, and quality with narrow variation from the desired values; and manage using techniques that provide early warning of deviation with the opportunity to take preventive or proactive action to keep on track.
Many companies first run into trouble when attempting to implement Level 4 because they have not established baselines for performance. For instance, what is the estimating accuracy for effort or schedule? If you do not know the current values, how can you set an objective for a 20 or 50 percent improvement in estimating accuracy? If you do not understand the current process and results for estimation, how can you change and improve estimating?
If you are working on projects with fixed-price contracts, the inability to estimate accurately will cause you to lose money (underestimating) or lose business (overestimating). If you develop software to sell or support internal operations, poor estimating will lead to committing to the wrong projects. This applies especially when there is large variation in accuracy between two projects; you will commit to a project with equal benefit and higher eventual costs, or fail to invest in the right project because it is overestimated. Reducing variation benefits the business, and industry reports show instances of cost/effort variation below 5 percent in high-maturity organizations (see Performance Results of CMMI-Based Process Improvement ).
Understanding the real aspects/attributes of process performance versus building control charts purporting to show stable processes is another misapplication of Level 4 methods. A control chart showing that you have a stable process is meaningless unless it contains a useful and valid relationship to the work being performed.
Two fundamental characteristics of the inspection process that can be evaluated with control charts are the preparation rate (number of pages per person per hour of effort spent in individual review) and defect density (the number of major defects per size inspected). Inspection process results usually are better when preparation rate is under control, and defect density is one measure of the value of inspections.
I recently saw an analysis of inspection data where an organization used control charts to show that the process was under control, but a fundamental oversight compromised the analysis.