When it comes to measurement, the IT industry acts strangely. Even when it recognizes the merits of software measurement, the expectations for it are often unrealistic. Software practitioners want a silver-bullet metric that can answer any development question and do it to several-decimal-point accuracy. This article addresses the mistaken notion of measurement or a particular metric being a silver bullet - a notion that left untapped can impede your organization from ever getting started with measurement.
When it comes to measurement, the IT industry acts strangely. While other industries depend on measurement, tracking, and control as keys to profitability, the IT industry has yet to embrace measurement on a widespread basis. Even when it recognizes the merits of software measurement, the expectations for it are often unrealistic. Software practitioners want a silver-bullet metric that can answer any development question and do it to several-decimal-point accuracy. Predictably, software measurement doesn't match these expectations and, thus, is usually abandoned before it can deliver a return on investment. This doesn’t have to be the case–software measurement can deliver value even when the measures are subjective (as with customer satisfaction ratings) or when the measures are imperfect (as with defect tracking).
This article addresses the mistaken notion of measurement or a particular metric being a silver bullet–a notion that left untapped can impede your organization from ever getting started with measurement. Future articles will focus on other aspects of measurement and progress from setup through implementation and growth of a measurement program.
... No Silver Bullets
Those of us with statistical or engineering backgrounds will always attempt to make measurement into an exact science (recall college labs where data outliers on research graphs were too difficult to explain and therefore were “erased” ?) In the real world of Information Technology, however, measurement doesn’t always translate into predicable outcomes and not everything that can be measured should necessarily be.
Measurement consists of taking a series of observations about a process or product and analyzing the data to indicate where positive changes might be made. It is important to realize that just because something can be measured to the nth degree of accuracy does not make it valuable to measure–there needs to be a purpose and a method behind the measure before it will be useful.
The first step to creating a successful measurement program is to realign your and your company's expectations about software measurement:
1. Follow the Goal-Question-Metric (GQM) approach to software measurement introduced by Victor Basili of the University of Maryland. This approach forces companies to clearly identify their strategic goals for metrics and to pose questions that will track whether or not the goals are being met. Only then are the metrics needed to answer the questions identified and data collection mechanisms put into place. The resulting metrics necessarily depend on the specific goals and questions of the organization. Within the SEI Capability Maturity Model for software are a number of Level-3 key process areas that can form the basis of an organization's goals/questions/metrics. ( Note: A new book featuring a foreword by Victor Basili was recently published by McGrawHill: "The Goal/Question/Metric Method" by Rini van Solingen and Egon Berghout. ).
This is an area that is often glossed over in an organization's rush to establish a solid metrics program quickly. DON'T skip the proper planning–this is critical. In the same way that skipping software requirements leads to products that do not meet customer needs, skipping measurement program requirements (Goal/Question/Metric) will lead to a measurement program that does not meet its customer needs. There is a great deal more to say about this topic than can be provided here, suffice to say that GQM is not a scientific approach, but a rational planned approach that will lead to higher success rates for measurement programs.
2. Communicate early and often that there is no "silver-bullet" software metric, just as there is no silver-bullet accounting metric . Defects, functional size, project duration, and work effort all measure a different aspect of software development, and they are not interchangeable. No