Using Agile metrics to manage projects and strengthen organizational transformation.

Traditional program management offices (PMOs) are responsible for providing checks and balances to the development and IT organizations regarding budget and schedule. Oversight and management that comes from the PMO drives certain behaviors in the project managers (PMs) and therefore in the project staff. Similarly, the Agile PMO provides certain checks and balance, but principally focuses on the holistic well-being of the project. This difference in 'tone' not only drives different behaviors, but also can help to support dramatic transformation within the organization. The driving force becomes encapsulated in the difference in perspective between traditional earned value reporting and Agile's achieved value reporting.

Devolution of transformations
The collection of documented Agile success stories are swelling, driving the process into the mainstream of software development. [i] However, in the past several years, we have also begun to see a phenomenon whereby successful transformations begin to devolve fairly rapidly, to the point where projects begin to slide and even fail. Many companies have even begun contracting in for services to re-enable Agile transformation, trying to recapture their early successes and perhaps making the transformation stickier the second time around.

So, are all Agile transformations unstable? I think not. What I think is more likely is that there may be active resistance to change acting as the outside force.

This situation brings to mind the old Newtonian adage: "Things in motion tend to stay in motion unless acted upon by an outside force."

Typically, a process change is typically not deemed successful until there is consensus amongst the team members and positive results have been shown. This suggests that active resistance would have been weeded out prior to declaring the methodological shift successful. The alternative might be a passive resistance, or friction force that is slowing down or stopping the transformation. If we accept this, then in order to keep the transformation moving, we need to find something that can counter this resistance and encourage the teams to continue.

Consider, do the Agile projects in your organization get positive reinforcement, negative reinforcement, or neither? And, what form does this reinforcement take, if it exists? In most development and IT organizations, there is a responsibility that belongs to the PMO to provide checks and balances on each project. It offers an independent review of budget and schedule for senior management as well as a periodic critique for the project managers. It is this critique that serves as possible reinforcement for the transformation.

What is the tonal difference?
There is what I would call a tonal difference between the critique from a traditional PMO and an Agile one. Traditional PMOs focus on variation from plan to date and quantify this as an earned value, the implication being that by merely making progress against the plan, the project is earning value for the company. Conversely, an Agile PMOs focus is on variation from commitment and quantifies the amount of value being released to the business. These are actually two very different measures - one allows the team to see how it is doing against its expectations and the other allows it to understand how it is doing against the business' expectations. The later is quantified as an achieved value.

{sidebar id=1}The tonal difference is established as a project-centric view for the traditional approach versus product-centric for the Agile approach. Project-centric refers to the act of completing the plan for the product and product-centric obviously refers to the product itself. To better understand this difference we need to ensure we understand the motivation behind the actions and decisions each PMO takes, so we will look at some of the specific traits of each traditional and Agile PMOs.

Traits of a traditional PMO
In companies that use traditional ‘plan-driven' methodologies to execute their projects there is usually a PMO to provide a source of check and balance. In these organizations, the PMO becomes the most influential force in the organization with respect to both the execution of programs and the approach by which they are undertaken. In the classical sense of you-are-what-you-measure, when the PMO requests that the PMs come to the review prepared to discuss budget, progress, and work performed to date - the essential variables for Earned Value calculations - PMs begin to focus their work and their teams on these measures.

This course of measurement only represents lagging indicators, telling the team how well it has done; not leading indicators, suggesting where it will end up. Ultimately, the result of Earned Value calculations is that we can tell the business how well or poor we performed to date, but we can not accurately recommend to them how things will turn out. The unfortunate effect of this is that everyone worries more about where they are as a function of where they anticipated being, but not enough about what value they are providing for the business as they move forward.

This lack of leading indicators for the project forces us to continuously rely upon the original plan for guidance. As the project itself wears on, deviation from the plan tends to increase and the use of the original plan as a barometer becomes more and more invalid. Stepping aside from this deficiency to forecast, projects tend to get graded on how close to the plan their progress to date lies. In an Earned Value environment, the significant measure is the budgeted cost of work performed. As it states, this is a measure of how close to the planned cost you were able to execute the work completed.

The focus on lagging indicators produces a set of behaviors in the Project Manager and subsequently the team. Since the team wants to be seen as successful (even more so than it wants the project to succeed), there is a desire to meet the plan and to be able to correlate work completed to budgeted costs. Because these objectives are almost always ambiguous, the teams can claim success - producing another behavior, one of obfuscation of completion status. This is manifested as the popular problem of tasks being on schedule all the way until they are supposed to deliver, and then stalling at 90 percent complete.

An age old complaint is that the teams and the teams' management end up finding ways to ‘game' the system and even though the PMO methodically goes through these monthly evaluations to assess progress, it knows the progress is merely a sham. Only once delivery milestones are met or missed does the business really understand the entirety and impact of the situation. This ultimately ends up as to why we continuously hear of projects being canceled after significant funds have already been spent with no clear path to success.[ii]

Traits of an Agile PMO
The Agile PMO also provides a check and balance process for the project, but there are only two real measures used by an Agile PMO. These measures have become synonymous with the tools designed to help track them - the Burndown, which tracks how the team is doing on their commitments and the Burnup, which tracks the achieved value to date that these commitments are bringing the company (see Figures 1). Both of these graphs include trend lines as their most substantial metric. The graphs themselves are clearly plotting lagging indicators, but the trend lines represent leading indicators. When the Project Lead presents the graphs to the PMO, if the trend lines do not imply that the iteration and the release have a sufficient probability of success, then a dialog can ensue focusing on how the team is trying to address the deficiency.

Figure 1: Sample Burndown and Burnup Charts. Burndowns graphically display the how the amount of work the team has committed to complete changes over time (hopefully is lessens as the iteration progresses, but it does not have to be monotonic). Burnups graphically display progress against an estimated scope level in discrete terms of stories completed. Scope is allowed to fluctuate based upon the business' needs and is usually represented on a Release basis.

The role of the PMO here is to be a sounding board for the ideas the team has developed, as well as a periodic check to make sure that the team is indeed using their tools as they were intended. Though there are certainly ways to ‘game' this system as well, the constant juxtaposition of leading and lagging indicators make it much more difficult. Additionally, when maintained on a daily basis, these indicators allow the team to immediately see the results of their decisions and actions in terms of their productivity and schedule. This in turn reinforces the team's accountability and encourages them to actively participate in generating solutions to problems as they arise.

Why Agile projects are often managed using either no PMO (no support) or a traditional PMO (anti-support)
The majority of organizations implementing Agile in their project organizations do not implement a separate PMO for oversight of the projects. Mostly, this is because PMOs have not been included in the Agile literature and are overwhelmingly associated with ‘heavyweight' processes. In the attempt to offer something new, most practitioners subsequently reject all things old. The difficulty with this is that when projects get into trouble, most project managers are so deeply invested in them they frequently ignore the signs that support, guidance, or intervention is needed. They lose the independent council that the PMO offers with its critique and without this, projects in trouble tend to stay in trouble until acted upon by an outside force. And then it is often too late.

When Agile projects are perceived to be unmanaged because their releases are not meeting expectations (or for any other reason), the first step usually seems to be to put a traditional PMO back in place (or push Agile projects back under existing PMO jurisdiction in mixed methodology organizations).

Since it is well known that you can not improve what you can not measure, the traditional PMO attempts to inflict Earned Value reporting onto the projects. Unfortunately, this usually has a negative effect as the only way to report Earned Value metrics is to collect them, and the only way to collect them is to change the process or begin tracking the project with a second set of books. Even though well meaning, by applying emphasis on a non-complementary set of practices, we get a methodological phenomenon akin to two sets of waves meeting asynchronously - the waves and troughs cancel each other out and the water goes flat and the nice smooth forward motion stops abruptly.

How to set up an Agile PMO to best support the transformation
The instantiation of an Agile-centric PMO at the outset of an Agile transformation can help bolster the depth and breadth of the changes. Many practitioners reiterate that Agile is a very experiential methodology and that there is a fair need to suspend disbelief while trying the first Agile project. Accepting these sentiments, it makes sense that the creation of a support mechanism simultaneously with the initiation of the transformation would be of great value.

Therefore the best way to inculcate the team to the new methodology is to have one or more experienced mentors on the team, constantly able to bolster individuals when doubts arise. If this is appropriate for the team members, why not for the project managers too? By establishing a PMO made up of a mentor and some or all of the project managers working on Agile projects, both a support network and an oversight board are created. As the PMO, this group retains the responsibility to provide checks and balances to the Agile projects. My recommendation is to review monthly - a period that will guarantee data from at least one complete iteration (lagging indicators) and initial work and projections for the next (leading indicators), as well as reactions to the previous iteration's retrospective.

Establishing a peer group gives the PMs an informal network from which to seek help outside of the PMO reviews, while also helping them all mature and learn from each others' experiences. Once the peer group gets large enough, the actual PMO review responsibility can be rotated through the group with the mentor and several different PMs being responsible each time.

The actual content of the review becomes simply reviewing the trendlines for the Burndown and Burnup charts and asking for explanations on anomalous behavior. If the charts imply confidence of meeting the business' expectations, then the review is short and sweet. If adequate explanations cannot be put forward or leading indicators belie satisfactory conclusions, then a longer discussion is necessary to help the PM identify interventions that will change the course of the project. It still remains the PM's responsibility to steer the team, but as with any retrospective, having these independent viewpoints often help make the path more obvious. 

[i] Such as the 2004 report ‘The Total Economic Impact^TM Of Using ThoughtWorks' "Distributed Agile" Approach' by Forrester Consulting for ThoughtWorks Inc., and the 2006 paper ‘Distributed Scrum: Agile Project Management with Outsourced Development Teams' by Jeff Sutherland.

[ii] See the overly quoted Chaos Report from the Standish group. There is now a series of these, the original from 1996 and the most recent revision last year. The Standish group has revealed that we software developers are doing much better now, because 35% of projects can now be considered successful, up from 16% in '96. Though a 100% improvement, these are still what I consider baseball numbers - a batting average of .350 would probably get you consideration for a batting title, but imagine an airplane manufacture where only 35% of their planes would fly or a construction firm where only 35% of their bridges were traffic worthy.