Mike: Probably the biggest misuse of metrics that I can think of is using them as a basis for performance evaluation. Too often, companies judge the worst testers by looking at the number of test cases they execute or the number of defects they logic. While that may seem a good way to motivate teams or individuals, it can actually have a number of unintended side effects. Long term, it can actually be demotivational and lead to an overall lack of faith in metrics and the value they provide.
Cameron: There's an old expression, an old adage out there, that numbers never lie. How does this apply to metrics?
Mike: Metrics don't lie, numbers don't lie, but saying that is a lot like saying, "Guns don't kill people, people kill people." When they're not used properly, metrics can be left very much open to interpretation. That also means that they're open to manipulation and abuse. I think that's part of the reason that more companies don't invest more heavily in metrics. Unfortunately, metrics, like a lot of other things, only provide real value when they're used properly. By the same token, you don't get all the thrills you're looking for out of a Lamborghini if you keep it in first gear the whole time, but that doesn't necessarily mean that all Lamborghinis are junk, now, does it?
Cameron: No. That's a good analogy. I like that. What is some advice you would give people who are unable to really change the method of metrics they have in place but can adapt how they use that method?
Mike: A big factor in successfully implementing a metrics program lies simply in how they are presented. Even when a lot of things aren't under your control, improvements can be made to how metrics are reported to stakeholders. The key is to make sure that the message behind the metrics is consistent and clear to everyone by providing the necessary context. Because metrics is so open to interpretation, there is a risk that people will draw the wrong conclusions even if there's no intent to deceive. This can have the effect of triggering false alarms or, even worse, hiding problem areas and giving a false sense of confidence when things are not going well. Situations like these can be avoided by making sure that metrics are always provided with context in one form or another.
Cameron: You have more than a decade of experience in software testing and you've worked with clients both big and small. Does the importance of having good metrics or being able to use metrics correctly increase the bigger the project or the bigger the team working on it is?
Mike: I think that metrics have an important role on all projects. With bigger projects, though, things like communication become more important but also more difficult. Since metrics are a communication mechanism at their core, they can be a real necessity on big projects where there isn't always time for everyone to talk to everyone else. The key is to make sure that they are used consistently so that everyone is effectively speaking the same language. At the same time, bigger projects offer more opportunities to accumulate historical data, which actually increases the potential payoff in the long run. Longer timelines means there are more opportunities to learn lessons from the metrics that are captured and make course corrections along the way.
Cameron: You'd like the attendees of your presentation to really step away with a number of different things, but what is one thing, or the main takeaway, you'd like the attendees to come away with?
Mike: More than anything, I want to stress the point that metrics are important and that there is the place for them on most if not all projects. The key, really, is finding the right approach that works for each individual in their organization.
Cameron: This is a broad hypothetical question and there's no wrong or right answer to this: How do you think the use of metrics will change in the future?
Mike: That's a bit of a hard one. I guess if there's more people like me out there, they will be a little more widely adopted, I suppose. That depends on how many people come to the presentation, but with the onset of agile methodology, the trend at the moment seems to be steering away from standardized documentation and standardized process and, unfortunately, at the same time, standardized metrics. We often see things like this as cyclical. I really do hope that it's not something that falls off entirely because, again, I do see there being value in the process.
Cameron: I agree. I agree. Is there anything you'd like to say to the delegates to STAREAST before they attend the conference and before they attend your presentation?
Mike: I'm really excited to be part of this year's STAREAST conference. Not only am I looking forward to the opportunity to present and share my own perspectives on metrics, but I'm also really looking forward to the opportunity to attend the other session and learn from them. Thanks so much for this great opportunity, and I'll see you all in Orlando, I suppose.
Cameron: All right, and if you haven't signed up yet, sign up for Mike Trites's presentation, which is "Making Numbers Count: Metrics That Matter." And he will speaking at STAREAST, which is May 4 through May 9. Thank you so much, Mike.
Mike: Thank you.
Senior test consultant with Professional Quality Assurance Ltd., Mike Trites has nearly a decade of experience in the software testing industry and holds multiple certifications, including an advanced-level test manager certification from the ISTQB. As a consultant, Mike has worked with clients, both small and large, in a wide range of environments. He has helped deliver projects from web applications and database management software to financial software and video lottery systems. As a test manager, Mike has managed all aspects of the testing lifecycle and works with clients to help define, implement, and improve their internal testing processes. Find Mike on LinkedIn at ca.linkedin.com/in/miketrites.