7) Other Level 2 Measurements
Outside of the Project Planning and Tracking KPAs, there are additional Level 2 measures that are needed.
Requirements Management. Management requires that measurements be made to determine the status of the requirements management process. Typically, the status of each requirement is known, as well as the number of open, closed, and in-process modifications to requirements, as these changes directly affect the effort and schedule commitments the project has made. As an example, at the end of a project you would want a sum of new and changed requirements and their impact assessments to evaluate the original accuracy of the project estimate. Hitting your effort or schedule estimate does not mean much in terms of estimating accuracy if you have had a 50 percent increase (or decrease) in requirements that was not tracked and evaluated during the project's life. If you know the ratio of new or changed requirements to the original count, over time you develop a baseline that says we have identified nn requirements, and know this typically grows by 30 percent, so we should be estimating for this larger number.
Subcontract Management. Subcontractor cost, staffing, and schedule performance should be tracked, which suggests the same measures tracked in Project Tracking and oversight are tracked by the contracting organization. Critical computer resources, as allocated to the subcontractor, are tracked. The cost of managing the subcontractor and delivery dates should be compared to the estimates in the plan.
Software Quality Assurance. The cost and effort spent on quality assurance activities should be measured and compared to the plans. Numbers of audits or reviews performed and completion of SQA tasks should be compared to plans. These measures are one indicator of whether "real" SQA activity is occurring. The actual effort expended is one way to determine whether or not there is an effective SQA organization. A second measure would be the number of nonconformance reports and their status (e.g., days open, number open or in different stages of resolution).
Configuration Management. The cost and effort spent on configuration management activities should be measured and compared to plans. The number of change requests, trouble reports, change/problem summaries (including severity or priority of the change or problem), revision history, change in size of products, and results of the various configuration audits are all measures of configuration management activities.
I've presented a brief overview of the Software Capability Maturity Model measurement requirements. These measurements should be aligned with your business needs. If some of the items mentioned above truly have no business value to you, then don't collect them. However, "I don't have time," "It's too hard," or "We're different" are excuses, not business reasons. If you are not collecting and using the measures suggested by the CMM, there ought to be a reasonable case for justifying your position. Remember that the model was developed based on good development practices across a wide range of companies, and this collective wisdom shouldn't be arbitrarily ignored.
1 Paulk, et al. The Capability Maturity Model: Guidelines for Improving the Software Process. Addison Wesley, 1995. pp. 29-79.
2 Ibid. p. 359.
Copyright © 2001, E.F. Weller
® CMM, Capability Maturity Model, Capability Maturity Modeling, and Carnegie Mellon are registered in the U.S. Patent and Trademark Office