Building a Meaningful Metric Mousetrap

[article]

This approach looks at total effort and includes the set-up time and the on-going effort to maintenance the metric. Calculate the hours:

 


Task




Hours




Set-up Effort
(define and automate process)




25




On-going Effort
(monthly)




12 (1 monthly)




Effort per Year




37



 

Value
Determining value is critical factor. If the perceived benefit exceeds the total effort hours in a year (e.g., the value-rating is greater than 1 when perceived value is divided by hours), then the metric may be considered a value-add to the organization or application. In this case, if we divide the perceived benefit of 75 by the total effort hours in a year of 37, we get 2.03 value-rating (rounded up). When using this approach, you will often find that the effort is greater than the perceived benefit (e.g., value-rating is less than 1 and should therefore not be considered as a potential metric).

Comparing a Potential Metric
When embarking on a metrics program, it is important to look at a number of potential metrics in order to determine their value-rating in relation to other metrics. As stated in the early section, gather input by a) uncovering problem areas in the organization, the application team, or project and b) identifying what value-added metrics are used in other organizations. Discuss the metrics with those benefitting from it and ensure they would, in fact, use the metric for improvements.

Of those potential metrics that remain, go through the steps described above (e.g., understanding the benefit of the metric, determine effort to collect the metric, identify who benefits most from the metric, and assess value of the metric). Then compare the potential metrics together. Here is an example of a metrics comparison table that uses the value-rating as its gauge.

Note: it includes the "time to build the product" metric discussed above:

 


Potential Metric Name




Perceived Benefit




Effort




Value-Rating




Time to Build
the Product




75




37




2.03




Build Errors




85




45




1.89




Code Volatility




45




60




0.75




Change Control
Volatility




55




45




1.22



 

In this case, we see four potential metrics. Of the four, the SCM metrics program may only choose to pursue the "time to build the product" and the build errors metric since it has the highest metrics value-rating.


Note: there will be times when the value-rating is below 1 but the organization still wants to proceed with a metric. In these cases, it is important to communicate the risk of the metric being perceived as not adding value.

Monitoring the Metric
As time goes on, any metric that has been established and is produced on an on-going basis should be revisited on a periodic basis. The reasons are two-fold: First, it is important to periodically check if the metric is actually being used. If it is not being used, then the metric should be discontinued. Second, the value-rating of the metric should be re-evaluated. Is the benefit still perceived to be high as time moves forward? Is the effort more or less than what was initially calculated? In some cases, once a metric is used and drives a positive change in the organization, then the metric has done its job and may no longer need to be generated (or generated less frequently then on a monthly basis).

It is also important to allow the data to collect for a few cycles prior to prescribing a target for a metric. It is important to identify the range of scores a metric produces over a reasonable time period to understand what "average" is and what are variations. Once this is done, then a target for improvement

About the author

Mario  Moreira's picture Mario Moreira

Mario Moreira is a columnist for the CM Journal, a writer for the Agile Journal, an author, an agile and CM expert for CA. He has worked in the CM field since 1986 and in the agile field since 1998. Mario has experience with numerous CM technologies and processes and has implemented CM on over 150 applications/products, which include establishing global SCM infrastructures. He is a certified ScrumMaster in the agile arena having implemented Scrum and XP practices. Mario is the author of Adapting Configuration Management for Agile Teams  and Software Configuration Management Implementation Roadmap. Also consider visiting Mario’s blog on CM for Agile and Agile adoption at http://cmforagile.blogspot.com/.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Sep 22
Sep 24
Oct 12
Nov 09