Using Evaluation Frameworks for Quick Reflections


Are we Agile? If we ask a leader we'll get one perspective, but if we ask each person on the team we may be surprised by the variety of answers. Since 2002, IBM has used agile evaluation frameworks with dozens of teams to help them learn, improve, and share their experiences with agile practices. The metrics in the Extreme Programming Evaluation Framework, or (XP:EF) originally focused on XP, and similar instances covered other methods.[i] But because the framework can be used with a broader set of practices, it's simpler to say "the Agile Evaluation Framework" (Agile:EF). This article shares tips on using the framework in a lightweight manner to leverage "the simplest metrics that could possibly work."

Review of the Technique
Agile:EF contains four parts. Context factors, indicators, outcome measures, and a survey. The metrics in the first three parts describe what a team did. This article focuses on the last part - the survey, which can serve many purposes:

  1. Stimulate team discussion to seek improvements.
  2. Spot weak areas in the team's process, in time to address problems before bugs result.
  3. Find out what the quiet people in the team think.
  4. Describe to other teams how much of which practices were used.
  5. Teach and remind team members of the practices.

Agile Pulse
Think of this survey as an "Agile Pulse." By taking it periodically you can monitor the health of a team's agile adoption. This can be done as part of your iteration retrospectives, or every two to four weeks, perhaps with some variation in questions or team members to avoid burnout. The list of questions should be short (about fifteen). The Agile Pulse survey should be written so that it takes 15 minutes or less to complete. That will make it practical to take it periodically. It should include a spectrum of practices, not just the ones the team does. That reminds teams to consider using those practices. If they share their experience, it also helps other teams know which practices were not used.

An Example
Here is example data from a team who has used the Agile Pulse survey.. Each question has a short label for the practice, such as ‘Non solo', which means any style of code review, or Iterative, which means ‘stakeholder feedback on working software.' The questions in the survey provide a little explanation of each practice to introduce them or remind team members.

Figure 1 shows the average of how much each team member feels he used a practice. The scale is 1 through 10. One means a person feels their team never uses this practice. Ten means they always use it. Five indicates a partial implementation. An example question would be "Unit Test: Do developers frequently run automated unit tests?"


Figure 1: Sample survey results


The thin black bars illustrate the range between team members. It's the average plus or minus one standard deviation, so 70% of our scores will fall in that range. This team is beginning its agile journey, so we don't expect high numbers.

We use the chart to trigger discussion. The average for "Automated Unit Test" looks low. Is the team comfortable with that, or is that something team members would like to address? Why is there such a wide range of responses for "Reflections"? The team should discuss this. Maybe some felt that reflections were not held often enough. Maybe some felt left out. Perhaps some felt no actions resulted from the improvements identified. The chart can help spark discussion, and give quiet people a louder voice.

Successfully using the Framework

Now we'll share some observations from experience using Agile:EF that will spare you some pain.

{sidebar id=1} Focus time on actions related to notable scores.

Quickly celebrate high scores. Ignore unremarkable scores. Focus a little time on averages below 4, and standard deviations above 2.5 (an indicator of divergent opinions among the team). Also, discuss averages that have dropped since the last survey. Focus the bulk of your team's discussion time on identifying a couple of improvements so you can make retrospective meetings more efficient. People often love to talk about their job. The passion for their craft is good, but it can lead to some meandering meetings. Point them to a few numbers highlighted by the survey to be discussed.


About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.