A handful of years ago, some of my clients began to ask me, “How are we doing?” My replies were always something like, “You’re doing pretty well at pair programming, and I like how teams have shifted from writing requirements documents to talking about user stories. But teams really haven’t embraced the idea of automated testing yet, and that’s where we need to focus most.” But this wasn’t the type of answer they wanted; they wanted to know, “How are we doing compared to our competition?”
At first this question bothered me. It doesn’t matter how your competitors are doing at agile, I reasoned. If you’re not perfect yet, keep improving. It took me awhile, but I eventually realized the flaws in my thinking. A business does not need to be perfect; it needs only to be better than (and stay ahead of) its competition. Google is the dominant search engine today not because the results it shows are perfect but because its results are usually better than those shown by its competition.
This means that agile does not need a five-level maturity model similar to CMMI. Organizations are not striving for perfection against some idealized list of agile principles and practices. Rather, they are trying to be more agile than their competitors. This does not mean that becoming agile is itself the goal. Producing better products than the competition remains the goal. But being more agile than one’s competitors is indicative of the organization’s ability to deliver better products more quickly and cheaply.
With this in mind, Kenny Rubin, Laurie Williams and I created the Comparative Agility assessment (CA), which is available for free online. Like the Shodan Adherence Survey and Agile:EF, a CA assessment can be based on individual responses to survey questions. However, it was also designed to be completed by an experienced ScrumMaster, coach, or consultant on behalf of a team or company based on interviews or observation.
Survey responses for the organization are aggregated and may then be compared against the entire CA database. Responses can also be compared to a subset of the database. You can, for example, choose to compare your team to all other companies doing web development, all companies that are about six months into their agile adoption efforts, all companies in a specific industry, or a combination of such factors. You can also compare your team against its own data from a prior period, allowing you what improvements have been made since then.
At the highest level, the CA approach assesses agility on seven dimensions:
- Technical practices
- Knowledge creation
Partial results showing a team assessed on three dimensions are shown in Figure-1. This figure shows how one particular team compared to a population of other teams pulled from the CA database (in this case, other teams doing web development). Zero represents the mean value of all matching teams in the database. The vertical lines labeled from –2 to 2 each represent one standard deviation from the mean. From Figure-1 we can see that this team is doing much better than average at Planning, a little better than average at Requirements, and significantly worse than average at Quality.
Figure 1: The results of a Comparative Agility assessment show that this team is better than average at planning and requirements but worse at quality.
Each of the three dimensions shown in Figure-1 is composed of from three to six characteristics. A set of questions is asked to assess a team’s score on each characteristic. For example, characteristics of the planning dimension include
- When planning occurs
- Who is involved
- Whether both release and sprint planning occur
- Whether critical variables (such as scope, schedule and resources) are locked
- How progress is tracked
The questions for the “when does planning occur” characteristic are shown in Figure-2. As can be seen in that figure, questions are answered on a scale including true, more true than false, neither true nor false, more false than true, false, and not applicable.
Figure 2: One of the characteristics considered by the Comparative Agility assessment is when a team plans