The Next Generation: Software Metrics

3. Easily customized metrics with dashboard presentation
Given a wealth of data and a means of mining that data, the next step would be to provide an easy way to express and to customize the metrics I'm interested in tracking. This generally requires some calculation and presentation specification language that can use the data query language to mine the data and compute the desired metrics in a manner that can be presented for comparison over specific points in time or along a continuum.

Depending on my role in the project, I may want a specifically designed dashboard.  If my goal is to monitor product quality perhaps I want to look at metrics such as:

• the number of problems introduced per 100 changes within a product development stream
• the number of problems raised each week for a given product development stream
• test suite completion and success rates over time for a given product development stream

If I'm a project manager, my mix will be different.  The point is that I don't want a few pre-canned metrics - I want the ability to easily define which metrics are necessary for me at any one time.  Ideally I can sit down and invent a metric on the fly, interactively, as a result of a trend that I've noticed over the past few weeks - and I can turn it on or off as necessary.  Metrics are useful for evaluating product quality.  They are also useful for evaluating process quality.  What should happen when we improve the process in this way? Let's measure and make sure it does happen. 

Metrics are also useful for evaluating teams.  This is a scary one.  Nobody wants their weaknesses pointed out.  "Sure we have twice as many bugs, but we do three times as much work."  How do you establish fair team metrics?  This is where interpretation is tricky.  Do you want more changes from a team member or fewer - what does it indicate?   More functionality completed?  Additional rework frequently happening? Incremental feature development versus big bang?  A lot of the interpretation will depend on your
process guidelines and procedures.  But a lot will simply reflect the different work habits of your team members.

So perhaps you want to overload metrics here and over time prune out the ones that aren't
really useful or yield ambiguous interpretations.  When your metrics can be easily specified and collected into a dashboard or two, you'll start asking better questions and hopefully will improve your decision making. Maybe it will be as simple as noticing that 20% of the staff disappear at March break so that better be accounted for in the
planning phase.

4. CM processes and applications that can reliably collect data
Remember that one of the key problems in establishing metrics is interpretation.  For good interpretation, you need to collect meaningful and reliable data, and you need to have a yardstick against which the metrics can be interpreted.  This is where the CM tools and processes must come to the rescue.  A version control system that tracks all the file revisions and marks each one with the user, timestamp and description will not give me a lot of metrics, and the ones it does give me will not only be difficult to mine, but will be
difficult to interpret.