Agile processes such as scrum and xp change not only the way software is developed, but also the organization developing the software. Much has been written about how agile processes affect the customers, the developers, and even the testers in an agile process, but little exists about changes to the managers. Agile methods such as Scrum do profoundly affect management work. Most managers in software development are used to planning and controlling, figuring out what to do and then telling people to do it. In an agile environment, the focus shifts to aspects of the project with greater potential payoff, and different players take responsibility for key project elements.
Traditional development methodologies fully analyze and design a system before coding it. Testing usually follows the coding. It is not until the very end of the project that the system can be implemented. The opportunities for managing to optimize value are limited and are usually not even considered.
Agile processes introduce the concept of controlling both development of functionality and release dates to optimize the value of the system being developed. Agile methods call this workload management. This is different from traditional task management in which the specific tasks involved in building a system are planned and managed. Instead, the focus is on implementing in early iterations the functionality that gives the highest return on investment (ROI).
Workload management is possible because in agile methods development occurs in a series of short iterations, usually between two and four weeks in duration. An increment of functionality is ready by the end of each iteration, so analysis, design, testing, coding, and documentation can be performed in every iteration. (The term "ready" in agile developmen means potentially shippable or able to be implemented. "Ready" in traditional software development has always meant complete, including user documentation, and fully tested.)
Iterative development provides management with many opportunities to arrange the sequence in which functionality is developed, so that the most valuable functionality is built first. Management can continue to rearrange the sequence of functionality development as the project progresses and business priorities change. Agile processes make it feasible to group increments of functionality into more frequent releases, allowing the business to realize early and frequent benefits.
An additional benefit of workload management is inventory reduction. Inventory in software development is exemplified by requirements documents that go out of date as the business environment and requirements change. As in manufacturing, unfinished "raw goods" software inventory is an undesirable cost. Yet traditional development methodologies amass huge inventories of analysis, design, and coding artifacts even as business changes render them obsolete. The agile approach minimizes the extent to which an organization accumulates such artifacts. Only those artifacts that are necessary to build each iteration's increment of functionality—the highest-priority functionality—are built.
A key tool for managing workload is the Product Backlog. The Product Backlog is a simple list of requirements for the system. Each item on the list is a single line in length. Functional requirements, such as "the ability to calculate available credit," are listed along with nonfunctional requirements, such as "the capacity to handle up to 100,000 simultaneous transactions with subsecond response time." An estimate of how long it will take developers to turn the functionality into an increment of potentially shippable product is included in each backlog item.
The Product Backlog is a prioritized list, often maintained in spreadsheet format so that it can be easily manipulated and interpreted. Items at the top are those that will deliver the most business value. Business priorities can change over the course of the project, and consequently the order of the list can change as well. Dependent functionality, or functionality that is required to support highest-priority backlog, is an even higher priority.
Not all the details of every entry have to be specified in the Product Backlog. Instead, requirements can be extracted from the description of the system that was used to acquire the initial funding for the project, focusing on the highestpriority items in the Product Backlog first. At first, only as much information as is needed to drive the first probable release is listed in the Product Backlog. The lower-priority functionality can be itemized and delivered only when it is deemed to be the highest-priority available functionality. Even then, its development may be deferred if it costs more than it is worth.
The Self-Managed Team
Who manages the work during each iteration? The agile answer is "The development team"! Agile work management is a radical shift from traditional Project Management Institute (PMI) priorities. PMI practices call for a project manager to predict and plan all of the work, as well as to assign it to individuals, track its completion, and make any necessary adjustments along the way. In agile development, the development team identifies and organizes the workload into a potentially shippable product increment. Collaborating with the workload managers, the development team determines how much priority functionality it believes it can cover in the next iteration.
Sometimes the team does less than it has predicted it would be able to. Sometimes the team implements the selected requirements even more fully than it had expected it could. Regardless, the team does the best that it can. The teams manage themselves based on their skills and understanding of the technical and business domains, which are explained to the team at the start of the project and reinforced at the planning meeting that occurs at the beginning of each iteration. For one iteration, the team alone wrestles functionality from complex, sometimes unstable technology and from often-changing business requirements.
It may seem risky and even foolhardy to trust the team to plan and execute its own work. However, this type of agile development has been successfully used in literally thousands of projects. Two types of productivity occur. First, the project manager doesn't have to try to tell the team what to do and then keep the plan current as changes are required. Second, the team works more effectively without having to rely on external authority for any changes.
The U.S. Marine Corps uses an approach similar to agile processes for battle situations. In Corps Business, General Charles C. Krulak , the Thirty-first Commandant of the USMC, describes the new "three block war” that the Corps faces today—"Marines may confront the entire gamut of tactical challenges within the narrow confines of three continuous blocks." To prepare the marines, the actual fighters, for this situation, the USMC both trains everyone extensively in all potential skills and situations that can be conceived, and then advises the marines on the context, mission, goals, and risks of every situation before they are sent in. But, from then on, the marines are on their own, making their own decisions. Their officers provide as much tactical information as possible, but the ultimate decisions come from the soldiers. As General Krulak says, "On the complex, asymmetrical battlefields of the twenty-first century, effective decentralized control and execution will be essential to mission success."
This same type of decentralized control and execution is used by agile teams to successfully cope with the complex changing requirements and complex unstable technology required for today's successful systems.
The Agile Coach
In an agile world, the project manager is similar to a coach for a sports team. The coach doesn't establish the game schedule. The coach isn't allowed on the field during play. All the coach can do is ensure that everyone is as well prepared as possible and understands the rules and goals. Similarly, the agile project manager, or agile coach, is responsible for maximizing the productivity of everyone involved in the project. The agile coach doesn't actually do the project work, but is responsible for doing anything possible to make the actual workers as productive as possible.
A sports coach is responsible for fielding the best team possible for every game. The team members are the best that the coach can field, they are in the best condition possible, they are trained to work individually and as a unit, and they understand the rules of the game. The agile coach has the same responsibilities: Ensure that everyone on the project understands the process, practices, and rules and that everyone involved in the project, even if they’re not actually working on the project, understands how the process and rules apply to them. This usually means training the users, customers, and senior management. Each team member is the best possible person available and has been trained in their functional domain as much as possible; and the product owner represents the users and customers and has the authority to prioritize the workload.
Prior to the project, the agile coach is responsible for assessing the field of play. Is the development environment established? Are all of the engineering practices, standards, and conventions in place? Are the quality, testing, and production environments ready? If not, the agile coach ensures that work is staged and prioritized into the product backlog.
As the project gets underway, the agile coach is responsible for closely watching the development teams. Scrum affords a formal opportunity to do this at the daily status meeting, and at the beginning and end of each iteration. At the daily status meeting, Scrum's rules ask each team member to report if anything is impeding progress. The agile coach (called a Scrum-Master) is responsible for noting these impediments and doing anything possible to remove them. For example, if a needed software component is being held up due to delays in purchasing, the ScrumMaster is responsible for facilitating the purchasing process.
Informally, the agile coach is responsible for being aware of the team's condition, morale, equipment, and everything related to its productivity. For instance, if construction outside the building is disturbing the working environment and making thought impossible, the agile coach's job is to fix the problem however possible. I remember watching Jimy Williams, a recent coach for the Boston Red Sox, get ejected from a game. His team was dispirited, so Jimy picked a fight when the umpire made a questionable call. After he was ejected, the team was so embarrassed at their lack of spirit compared to Jimy's that they came back and won the game. Sometimes the agile coach is symbolically kicked out, and the team has to pick up the ball on its own. But because they have experience planning and managing their own work, they can do it!
The agile coach is also responsible for measuring project progress. If an iteration is falling behind, the agile coach works with the team and Product Owner to adjust the workload to still meet the goal of the iteration. If functionality isn't being built fast enough to meet a release date, the agile coach is responsible for working with the Product Owner and management to adjust goals, functionality, dates, or costs. The agile coach inspects the project on an ongoing basis and reports any variances from expectations, then works with everyone involved to make the best possible adjustments. The agile coach also produces regular reports that measure expectations against reality, and describes the adjustments that were selected, why they were selected, and how they were implemented.
The agile coach is the "go to" person on the project, responsible for everything but with authority over nothing. But who said that management was easy? The description of the various management responsibilities in Scrum presents a radically different picture of management from that found at many organizations. Scrum management is very active, constantly inspecting what is happening and making the most appropriate adaptations. This type of management is very committed to deriving business value from the expense of the project and collaborating throughout the project to make this happen. In the process, everyone is required to use intelligence and creativity. The result is satisfaction and pleasure in the workplace and an increased probability of success in the enterprise.
A Parting Thought
The last thing most organizations need is someone coming in announcing, "I've got a new process and you've got to change!” When I implement Scrum in an organization, I downplay the management changes implicit in such an implementation. I figure, let the organization realize the simplicity and benefits of Scrum, then it can figure out the degree to which it wants to change. Most organizations dramatically change themselves once they experience an agile process's practices and philosophies. They take the core concepts of iterative, incremental development, self-organization, emergence, and a focus on working functionality and spread them throughout the organization. The art of the possible becomes a guideline to organizational plans.