While most people agree that metrics are important, choosing the right ones is a critical, but often very difficult process. This paper offers a set of guidelines and considerations when designing metrics for software, test, and documentation projects.
Software project metrics are measurements, formulas, or constructs that help form raw data into patterns that can be used to improve the project outcome or process, identify trends, and compare the project to other projects or to earlier versions of the same project. They are often used to report project information to management as they often sum up areas of the project in a clear and easy to follow way.
Without them, it is often impossible to determine if a project is on track to meet important goals or deadlines. Furthermore, the effectiveness of improvements made to the process cannot be measured or compared.
The process of continuous improvement, where improvements made to each cycle of a software project make it better than the last relies on accurate, clear, and impartial measurements.
Metrics fall into the following general categories:
Used to compare projects, or project elements, to previous versions of the project, or to other similar projects. The more similar these projects are to each other, the better and more accurate the comparison will be.
These are metrics that track the current progress of a project. For example, they might be graphs showing the amount of work completed or remaining, tables showing which components are complete, being worked on, or haven’t been started. This kind of metric is generally more immediate than other types, and shows the current state of the project in the present, past, or very near term future.
These are metrics that show the expected state of the project, or project components at a future time. For example, they might be graphs showing the projected defect counts during an ongoing project, or customer defects over time for a project that has shipped. While prediction metrics are typically very "fuzzy" and subjective by nature, they are very useful to compare against the current tracking metrics to determine if things are going as expected and, if not, how far off they are.
These are metrics that are show information about an area of the project, without an expected result. They are primarily used to generate discussion. For example, a pie chart that shows different categories for the current defects on a project can be used as an informational metric. By itself, it doesn't mean much, but it generates discussion about why there were more defects in one category or another. This kind of metric is distinguished from a tracking metric by the lack of an expected right or wrong result for it. Many times informational metrics can be combined with each other to become a higher-level tracking or prediction metric.
These metrics are used to show the conformance or deviation of a project to an established process or standards. For example, if the process for a project calls for all defects to be fixed and closed 30 days after they are opened, the process metric for this might be a graph or table that shows which defects were opened each month, when they were closed, and a color-code to indicate which ones fell outside the 30 day window. If the process metrics for a project consistently show nonconformance, this may point to a problem with the process itself rather than the project it is measuring.
Most projects require metrics from all of these categories.
Recording the metrics
For metrics to be useful, everyone involved with the data collection and metric generation must be using the same data, and getting the same results. For example, when counting defects across the various components in a project, it is