Metrics have their place and can be extremely useful. They can also drive dysfunction when used in the wrong way. The minute you assign a number to something, people begin to focus on the number itself and not on the story that it is telling or the information being provided. But the stories and the context behind the data often are far more valuable.
Velocity as a Measurement
I had a team with a fluctuating velocity. Over a period of eight sprints, they were not even vaguely consistent, and the product owner therefore found planning difficult.
I decided to have a look at the detail and keep track of the information behind the numbers. The team was consistently losing and gaining team members due to contractors moving backwards and forwards for visa renewals and requirements and team members going on leave. The team size was never consistent, so how could their velocity possibly be? This number was not entirely useful on its own for the team, however it helped determine that communication and interacting when team members were abroad were huge issues. It was very useful as a starting point for trying to solve the problem of difficult communication.
Another problem highlighted by looking deeper into the meaning of the metric was that the team had a distinct lack of direction. The backlog was inefficiently groomed and prioritized and occasionally almost nonexistent. You can see the effect this had in sprints 70 and 71, which were particularly bad (see figure 1). Once the team was aware of this, they put practices in place to ensure more visibility and encourage better stories and prioritization from the product owner.
Being exposed to this information was invaluable for the team. It helped them to ask questions about what was happening and what they needed to improve going forward. The individual numbers themselves were fairly worthless, but the patterns were hugely insightful.
How Does Velocity Become Dysfunctional?
I have heard in agile and not-so-agile environments things like “The team needs to double their velocity in n sprints.” What I do not understand is how anyone can fail to see the dysfunctional behavior waiting to happen when a team is tasked with this.
The behavior I have observed frequently in this instance is the team’s doubling their story points per story. Every three becomes a five, and every five becomes an eight. The only thing this achieves is turning an insightful metric into a number that adds little or no value. Most unfortunately, this can affect the team’s trust and behavior—both the trust that they feel from management and the trust that they have in management.
The lowest tier of Patrick Lencioni’s Five Dysfunctions of a Team is the absence of trust. In my experience, using the metric in this way can start a vicious, dysfunctional circle with a lack of trust at its core.
Bugs as Information
Information on bugs can be very useful for both testers and other teams. The status of a product’s health should include how many bugs have been found and which ones the testers and product owners feel are non-negotiable for launch.
When reviewing the number of new bugs in sprints, the sprints with more bugs might mean that the team was pushing too hard for speed and quality slipped. By looking at the kinds of bugs and the numbers of bugs—the team can determine the value of the ratio of quality to speed. If this ratio is something that the team, testers, and product owner are happy with, then all is good. If not, then a new negotiation needs to take place.
The average time it takes a critical bug fix to go live can also be valuable information for a team. If it is difficult, takes long, and the team has to release the last fifteen features and regression test for two weeks before they can release, then maybe there is a problem. If teams have this information and understand the impact, then they can work on solving the problem.
Knowing the overall health of the build or product is far more useful than knowing that we have ten unresolved bugs. Those ten bugs could be symptoms of something serious within the system, or they could be minor text and UI changes that need to be implemented.
What’s Wrong with Bug Counts?
I was speaking to the product owner of an interesting project. He was complaining about the number of bugs logged that were essentially duplicates. I asked how the testers were measured, and he said bug counts.
What scared me most was that the metric had encouraged behavior that put the focus completely on the wrong things. The testers were focused on easy-to-find, UI-visible bugs that could be logged and counted separately per screen or per event. They were not at all focused on getting information about product health as a whole. The testers were not looking for information about what we do not know—the tests that take time and patience and intricate thinking to invent and run. Those tests provide valuable information if they yield any issues.
The metric, especially when used as a measure in this way, was totally driving the wrong behavior.
Metrics come in all shapes: velocity; bug counts; code coverage; cycle time; live releases per sprint, week, or month; etc. These can all provide valuable information for creating visibility around things that team members do not know. If the focus of the metric is only on how much or how many or how often, then I believe the value of the metric for the team decreases. The wrong behaviors are rewarded and encouraged, and too often trust levels are affected. In my experience, it is far more useful—and far less dysfunctional—to pay attention to the meaning and the nuances of the metrics than the numbers themselves.