“The CTO’s measure of success is that all teams see at least a 50 percent increase in velocity by sprint six.”
This was once told to a team I was coaching. It took perhaps ten seconds before someone was grinning from ear to ear. Even the team manager saw the flaw in the directive he’d just passed on. “Okay, so in sprint five we just need to make all our five-point stories thirteen-point stories, right?” joked one of the team members.
Welcome to Goodhart’s law in action—essentially, when a measure becomes a target, it ceases to be a good measure.
This is true of people metrics, too. Setting performance goals based on team metrics can dampen morale and even set teammate against teammate. Instead of promoting collaboration, it leads to cross-team competition and gaming of metrics to look better.
With the transparency of agile and the granularity of team-based metrics, it is extremely important to be responsible in how you use your measurements. There are five principles I adhere to when dealing with metrics: start collecting early and often, be consistent, stay focused, measure the project and the teams separately, and—most importantly—measure responsibly.
1. Start Collecting Early and Often
Start collecting metrics from the very first sprint. If possible, you want to capture how your teams were performing even before the agile transformation. When you’re already six sprints into an agile transformation before you start tracking your metrics, you’ve lost the opportunity to see where you came from.
When I signed on to coach a twelve-team organization, they were already three months into a rolling agile adoption. I was lucky that they had been entering data into their agile lifecycle tool. I continued to collect data as I observed the teams, so by the time I was three months in, I had a really good picture of each team. This gave me a place to work from as I dove into engaging with the teams. Already having trends and areas to focus on meant I was able to get some early and fast wins.
2. Be Consistent
This isn't so much of an issue with the agile projects themselves as it is with the tools we use to track measurements. While physical task boards can become cluttered, inconsistency is most often exacerbated by online tracking tools due to their customization abilities, which often results in different features being used in different ways from team to team.
When your teams cross over multiple managers or organizational structures, it can quickly become unwieldy to track the various metrics each team or department wants to use. This is especially true when using any of the larger, enterprise-scale agile lifecycle tools that allow easy customization.
You need to have some common, agreed-upon measures for all teams and organizations. My recommendations are cycle time, escaped defect rate, planned-to-done ratio, and a team happiness metric. Even across a broad and diverse section, these are some consistently understood and measured criteria. A good team-level assessment tool is also helpful here.