Program planners in IT organizations have a dilemma: On one hand, their agile teams tell them that if requirements are defined up front, agile teams cannot operate; but on the other hand, the program’s budget and scope need to be defined so that resources can be allocated and contracts can be written for the work. How does one reconcile these conflicting demands?
Agile processes suffer greatly if requirements are tightly defined ahead of time. At its core, agile is built around having continual discussions with the customer to elicit requirements; those discussions transfer knowledge to the team. Rather than relying on requirements documents, which are the primary method of communicating to the developers in waterfall projects, agile projects rely on in-person communication. If you give an agile team a requirements document, they will have to spend months deconstructing it and learning it, and by then their entire release timeline will have expired. If you have a requirements document, you might as well not make your project agile at all.
But agile projects do need some level of up-front requirements—in the form of a release roadmap. A release roadmap is a high-level timeline that lists the business capabilities to be attempted for each release over time. When an agile team begins work on a release, the first step is to decompose the release’s planned capabilities into high-level “features” or “epics.” These are later broken down into more granular “stories” that can be completed during the iteration. This progressive elaboration of requirements is referred to as rolling wave planning.
Rolling wave planning is a challenge for program planners because traditional program planning rests on the assumption that one can create a detailed plan and cost estimate for the work to be done. For software development, this assumption is false.
The root issue here is not so much an agile issue as it is a software development program planning issue. It is mostly a settled matter that large IT projects are risky and challenging, and the best way to deal with large software development requirements is to decompose them into many smaller, more tractable—and largely independent—efforts. For example, a 2010 Department of Defense report to Congress stated (emphasis added):
A major change in the new process will be moving from large multi-year programs to portfolios of short-duration projects. This requires a new approach to project oversight. This approach will place more accountability on timely coordination, quicker decision making, and increased stakeholder involvement through more frequent performance-based in-process reviews.
Agile has been looked to as a solution for the challenge of large software projects. However, using agile on software teams does not magically fix large IT programs—as we saw with HealthCare.gov. According to an article in The New Yorker, “Healthcare.gov involved fifty-five different contractors that each delivered a piece of the final system to [the Centers for Medicare and Medicaid Services] for integration and testing. Whatever development processes those contractors used internally, the overall project was run according to waterfall principles.”
The problem is that agile methodologies like Scrum do not say how to manage a program. To figure that out, one has to go back to the first principles of the Agile Manifesto. If traditional program management is out, then what replaces it?