Agile software engineering and agile project management have become more mainstream in recent years with great success. But the benefits from agility should not have to stop there. Instead of initiating a project and letting the team run with it, progress reporting, planning and estimating should translate through all the channels, back to the corporate strategy. That way, executive management or the PMO can continuously balance the vision of the organization. This article will present these concepts for agile portfolio management.

Origins of Agile Software Development
The popularity of agile software engineering has been constantly increasing over the past decade. This was especially true after the formation and creation of the Agile Manifesto in 2001 which gave the agile movement a boost and foundation. This manifesto describes the core values of agile software engineering from a methodology-neutral perspective. That means the level of agility of existing or future methodologies can be determined by comparing them against these core values. Agile {sidebar id=1} software methodologies provide the foundation of the agile pyramid (see Figure 1), which we will discuss throughout this article.

The motivations for agile software engineering however root in the need to build software systems alongside ever-changing business environments and date back to the 1970's. As a matter of fact, many authors of the manifesto had worked with object-oriented software engineering and the Smalltalk^TM programming language many years before agility was formalized. These technologies led to methodologies like Scrum and eXtreme Programming, and it is not surprising that these industry leaders eventually initiated this manifesto.

Just to be clear, the ground-breaking concepts and technologies in the past were a reaction to the need of building more flexible IT systems. That led to technologies which enabled agility in the first place. Without going too much into the history, what is important is the fact that agile software engineering eventually brought us to where the business wanted project teams to be decades ago.

Building software in shorter and more frequent intervals, inviting feedback, adapting to change, and gaining better insight into projects, were among many other possible motivations. Until a few years ago, we did not had the diversity in technologies to accommodate this request.

The Need for Agile Project Management
What is surprising, though, is that the project management principles acquired decades ago are still applied today even when the underlying software engineering has changed to become agile. Even though agile software engineering had changed fundamentally how teams build software, the project management style stayed the same. How could a traditional project manager successfully lead an agile project?

That is quite a mismatch. For example, it is nearly impossible to plan a long-term project from the start to finish when the software engineering process invites and embraces change. With agile software engineering, it seems unnatural to plan tasks and activities in break-down structures and hand out work-orders.

As a reaction to this mismatch, industry leaders made an attempt similar to the agile manifesto. The Declaration of Interdependence for agile-adaptive management was released. The declaration highlights six core practices: value, teams, customers, individuals, uncertainty, and context. While each practice was identified as a contributor for success, it is the sum of all six and their interdependence which makes them complete. Because agile project management coordinates activities for agile development, it occupies the second layer in the pyramid (see Figure 2).

Since the release of the Declaration of Interdependence in 2005, the Agile Project Leadership Network (APLN) and its local chapters foster a community of agile project managers and duplicate these practices in the industry. Equipped with these principles, we can now close the gap between a naturally agile business environment and agile software engineering.[i] That means the needs for adapting to constant change in the business world translates directly into agile management practices, which are then engineered by an agile project team.

Challenge With Traditional Portfolio Management
In larger organizations, however, a major challenge remains: the alignment of the strategic corporate vision at an executive level and the transformation at a project level. Due to the size of the organization, political hierarchies, and geographical boundaries, the communication channels between levels are often ineffective. Written status reports, tools, and indirect communication channels can create additional layers of ambiguity.

Agile project teams need direct access to the folks creatingthe corporate strategy so that questions and issues can be resolved pragmatically. For example, executive management wants unbiased insights into strategic IT projects. Review meetings, however, are often executed with stakeholders at different levels of an organizations. Reports are created at many different organizational levels.

The higher the in the hierarchy the more condensed the status report, focusing on metrics rather than assessing the technical progress and issues. It is also very common, that project teams are not very open about the issues in written formal reports.[ii] Furthermore, the performance of one project may be difficult to compare with the performance of another project. Different metrics, review cycles and the inclusion of prose-style reports make it very difficult to compare individual projects with each other. Harvesting, administrating, and massaging the data could increase internal costs as compared to open communication in a review meeting with a clear agenda.

On the other hand, project team members (including the project manager) often cruise on their own towards the goals initially defined in vision documents, a.k.a. scope or mission documents Not knowing when and how the executive strategy has been adjusted, the project team could continue going into the wrong direction.

Sometimes the vision is re-communicated to projects that are in progress. This may challenge them with drastic changes to the goals of their project, especially if the project is near the end of the schedule. Smaller and more frequent changes to the corporate strategy will be easier to digest than one major shift late in the project.

Establishing Agile Portfolio Management
A portfolio which includes agile projects should be in-line with the Agile Manifesto and the Declaration of interdependence. Agile portfolio management must therefore foster and nurture an agile environment top-down. It seems logical, for example, that a portfolio manager should not request frequently endless amount of metrics from the project teams without violating the agile value of working software over comprehensive documentation.

Similarly, a portfolio manager should not ask the agile project manager to lay out complete detailed project plans before the project is initiated without interfering with the value of improving effectiveness and reliability through situationally specific strategies, processes, and practices.

On the other hand, an agile project depends on the PMO removing or changing non-agile, traditional and dysfunctional procedures. Only this will encourage a corporate-wide agileenvironment. In exchange, agility is translated basically from software code back to a corporate strategy from the bottom up. The agile pyramid only enables a bi-directional agile collaboration if all layers in the pyramid encourage agility. Without a layer, the pyramid is not effective and may collapse.

Agile portfolio management is therefore a two-way collaboration channel, simplistically balanced in a sense so that agile project teams are equipped to execute the corporate strategy while providing enough information back to portfolio management to support sound decisions (see Figure 3). [iii]

Let's take a look the concepts which enable agility in portfolio management.

Forward Management
One important force in any portfolio is the need to innovate and explore new areas for future competitive advantage. Technical innovation and feasibility, however, often emerge from the technical team commonly members at the base of the pyramid. Periodic iterative assessments and all-hands proposal submission processes provide great opportunities for agile portfolio managers to recognize and foster innovation.

Not only will they fill the funnel with great project ideas, they will allow portfolio managers to request funding and support for an idea for one of the future iterations. Agile software engineering with its iterative-incremental practices fits very elegantly with this approach. Resources can be allocated to one or two iterations, so the idea of the project is "tested" before more resources are designated and potentially wasted. An agile portfolio manager can actively influence the future of the portfolio, rather than comparing historical metrics and requesting detailed business cases.

Metrics
To compliment the assessment of a project, metrics in an agile portfolio management process should support the decision-making and selection process. These should not replace the periodic assessment. Similar to a financial portfolio, a metric must be reduced to the most fundamental but effective amount, for example a stock-quote. If buy or sell decisions are made, more scope can be made available, similar to quarterly summaries of a stock company.

The established metrics need to give executive management insight into the situational health of a project and compare it with other projects. The way metrics are derived must therefore be consistent among all participating projects. Equipped with the review and metric, executives can actively steer and lead the portfolio towards their evolving corporate strategy. A good example is an agile portfolio management process which emphasizes only on one value which allows executive management not only to view the state of the projects, but also of the entire portfolio.

Assessment
Recently I heard a doctor say "I believe it only when I see it twice." A slightly modified version of it for an agile portfolio manager could be "I believe it only when I see it at least once." Eye-witness observation of progress, first hand information from the project team, and comparisons of visions (corporate versus project) are integral parts of supporting sound decisions. The assessment is a social event between humans demonstrating progress and explaining decisions. As a periodic event, the assessment allows the project team to feed the corporate vision with new innovative ideas for the future. It could transform a command-based "please implement this specification" to an open-forum environment "let me exceed your expectations."

Project Selection
Instead of go/no-go decision points, agile portfolio management opens doors for a more detailed and effective selection process.[iv] Most important, the portfolio should be balanced and emphasize maximizing return of investment. Therefore, newer and more prominent project proposals might compete with running projects for resources. For that reason, running projects might be paused for a very short period of time, temporarily understaffed, and delayed to free-up resources, deployed with a subset of the features, or simply put back into the funnel of proposals for later revitalization. The world of an agile portfolio manager is not black or white.

Conclusions
The pyramid has been used to visualize the dependencies between agility and organizational needs. The four level of agility (corporate strategy, portfolio management, project management, and development) all need to be in place to gain the full benefits of for all stakeholders. Lacking in one or more will break the foundation of the pyramid and make the model incomplete.
With the wide-spread adoption of agile methodologies in larger organizations, portfolio management processes need to be revisited to reflect agile principles, including metrics, assessment, forward management and project selection.

---

About the Author
Jochen (Joe) Krebs (http://www.jochenkrebs.com) is a active member in the agile alliance and the agile project leadership network where he spearheads the local chapter in NYC. As a certified Scum master and co-author of the Rational Unified Process - Reference and Certification Guide (ISBN 0131562924), he publishes articles with a focus on project management and requirements engineering. He hold his MSc in Computing for Commerce and Industry at the Open University.

---

[i] Some might argue that agile software engineering is also natural.

[ii] See The Lean-Agile PMO: Using Lean Thinking to Accelerate Agile Project Delivery, Sanjiv Augustine and Roland Cuellar, Cutter Consortium Executive Report, 

[iii] For more on collaboration, see High Performance Agile Teams: An Overview of Collaboration

[iv] See Krebs, Jochen "Managing an Agile Portfolio", Agile Development Magazine, April 2007.