While many of us subscribe to agile methods and the ideals they represent, too often the organizations we work for force us to do things that are in direct conflict with agile principles. This is particularly the case for IT departments in larger organizations, where agility is evident as good intentions rather than being the reality on the ground.

Just over a year ago, the financial services company I was working with initiated an agile project with a lot of fanfare. It had management sponsorship, was given a catchy name, and the development team was led and trained by leading Scrum consultants. Today, the mention of that project, which resulted in the manager who led the initiative now maintaining a low profile, causes embarrassment among the remaining members of the development team. The project’s failure to deliver on expectations has meant that other agile initiatives have suffered and the crater it leaves ensures that it will be some time before another agile project is given the same level of support.

Constrained Agility
I’m sure this is not an isolated instance of well-intentioned agile projects going wrong; you may have your own stories to share. In this particular case, the reason team members gave for their failed attempt at agility is that the project did not have control over its own destiny: There were too many constraints and external dependencies to deal with. These constraints arose from the fact that the project had to work within the boundaries of the original business case that established the scope in advance and placed limits on the planning of each sprint. Furthermore, while the agile team members were “sprinting,” other teams that the solution relied upon were progressing at their usual, relatively glacial pace. In other words, while the business saw the project as a singular independent initiative, the solution architecture relied upon other components to achieve the desired outcome. In reality, project success was dependent upon other teams in the enterprise.

This example shows that to be successful in large organizations, agile principles must encompass all aspects of a project. You must be agile early in the lifecycle when the project’s business case is developed, rather than only being agile in the software construction phase alone—or worse, restricted to a single development team. Routinely creating such a business case calls for estimating the entire project scope and establishing the high-level solution design that often impacts multiple teams,

To assess an initiative’s (economic) feasibility and have the budget agreed upon, IT projects are forced to lock-in high-level requirements and make early design decisions, long before any software is written. As a result, specialist roles, such as business analysts and solution architects, are needed in the early stages of an IT project as they are responsible for vital, early-planning decisions. The people who work in these roles routinely work with the customer in order to scope the solution and liaise with the development teams responsible for its implementation, thereby establishing a project’s financial and technical envelope.

It’s in these ways the waterfall reasserts itself, even in so-called agile projects. Development teams are separated from their customers and relegated to working through proxies who communicate the project requirements. This behavior is exacerbated when a project is reliant on outsourced or offshored development teams for critical capabilities, thus resulting in IT projects operating in a more contractual environment where even agile teams must adhere to approval gates and review processes. While this certainly challenges the conventional notions of agility, it also represents the next frontier for scaling agility: Not only must you scale software development across the multiple teams constituting many IT projects, but you must also scale the coordination, planning, and design.

The Old “Divide and Conquer” Trick
Arguably, it is solution complexity that drives the described behaviors resulting in formal processes.  These processes are an attempt to reduce the customer’s financial risk since complex, multi-team solutions have a tendency to make project sponsors (and pretty much everyone else) nervous. So increasing project complexity usually demands greater “management visibility.” Formal handoffs between the analysts, designers, and developers involved can result in copious documentation being produced along the way. In attempting to “demonstrate” their understanding of what needs to be done, detailed use cases and system behaviors are described in words and models rather than working software, As a result, a project team can suffer information overload early in the lifecycle with the risk that they lose sight of the forest for the trees.

Of course, no project is intentionally set up to be slow and ineffective, or made to create impenetrable and expensive documentation instead of demonstrable capabilities. Yet, the loss of agility can appear to be inevitable when the problem domain is complex and the solution architecture grows correspondingly sophisticated. Traditional “divide and conquer” management patterns routinely see dedicated teams maintaining IT platforms or system component, focusing on a specific technology silo. However, even a simple regulatory change in the financial services industry can result in changes across multiple IT platforms, making it necessary for you to coordinate a multi-team change program and establish protocols for inter-team communication.

Sadly, it seems problems are more easily divided than they are conquered. While analysis and design can decompose complex programs and help us understand what needs to be done, to truly “conquer” we also need synthesis; this encompasses the merging of separate development streams and integrating them to ensure the teams work towards the one solution. It is in synthesis where many development lifecycles are weakest, as the emphasis is usually on the creation of components with the implicit assumption that they can be assembled when needed.

Continuous integration now provides powerful tools to bring multiple software components together, but in IT there are no equivalent tools to integrate the non-software aspects of a solution. In many business environments, such as the financial service industry, changes to distinct and independently managed IT platforms need to be integrated. These can be sophisticated systems in their own right, so it can be challenging to coordinate the design and development efforts of these separate teams (and sometimes whole organizations). Interfaces need to be agreed upon and in more devolved, outsourced, or offshored environments, their specifications become increasingly formalized and quickly morphs into that most insidious construct: the contract. 

These complex more scenarios demonstrate the need to scale agile methods, which arguably remains best suited to singular software teams rather than multi-team, multi-disciplinary enterprise projects. How do we attain the Holy Grail of agility in the face of growing complexity? Clearly, we cannot give up on larger enterprises that want and badly need to be agile; after all, they provide many of us with our day jobs. And, while many projects in such enterprises encompass non-software development scenarios, including business processes and hardware or networking infrastructure, they heavily rely on software and represent an opportunity to apply enterprise-scale agile techniques.

Balancing Change and Complexity
In facing the dilemma of agility when dealing with complex and pre-existing architectures—be they software, systems, business, or organizational—further analysis of our very own problem domain can provides useful insights. The conditions under which we attempt to use agile methods differ depending, for example, upon whether we are developing a new software product or working on an IT project.  

For agile projects to be successful across a range of settings, we need to make some sense of our collective experiences and better understand the challenges practitioners face in attempting to be agile in the enterprise. Clearly, current agile methods work exceedingly well in some environments but can struggle in others. While many of us appreciate the elegant simplicity of story cards and burn-down charts, applying such techniques can be a challenge in IT projects where multiple teams must working within contractual constraints.

While we may be geared to embrace change when being agile, we must also embrace complexity. The latter is inevitably challenging, because the more quickly we develop software and the greater the sophistication of the solutions we build, the more difficult it is to maintain agility. That’s the nice way of putting it. Sometimes, in attempting to shrink time to market, we may also create complexity by making poor design decisions and establishing architectures that are inappropriate or do not scale. For example, while we all may be able to understand and use basic mail-merge capabilities for our personal use, the same functionality in a banking system requires a highly secure, robust architecture when used to send out customers’ monthly financial statements. We must accept as a fact that sometimes complexity is just a reflection of the problem space, and there may be no option to remove it, whether starting incrementally or resorting to simpler development processes. Sometimes we are forced to build complex systems from day one.

While agile methods have mindshare, these are the instances where the plan-driven, waterfall processes stubbornly prevail, because it’s often that the solution is not entirely software-based. Instead, these solutions rely on people and business process that are less amenable to agility. Additionally, there can be hybrid situations, as in the example given earlier, where projects attempt to be agile even when they must work to reach predetermined planning milestones.

Not everyone may agree with this assessment that sees agility as desirable but not always attainable in organizations; this only underscores the point that there is a spectrum to agility. My research showed that change and complexity can be a useful way of framing our different experiences, as they represent the underlying factors affecting our ability to be agile. The degree to which these two fundamental forces are present helps determine the approach best-suited to a particular situation. We need to be more precise about what constitutes complexity, but for now—using the ubiquitous four-square grid in figure 1—we can view traditional plan-driven and agile methods in terms of where they sit on the change and complexity axes.

Figure 1. Sweet spots for agile and plan-driven methods

Figure 1 indicates the coverage of the two approaches and represents the agile and plan-driven “sweet spot” as separate ellipses. The different quadrants provide a simple means for categorizing the conditions a development team may find itself in. Agile methods are best suited to managing the change dimension, so its sweet spot spans quadrants one and two. Plan-driven approaches are appropriate for the management of large-scale, complex development programs, so they cater well to quadrants one and three where low change volatility allows more detailed planning. This grid suggests that we have a choice in quadrant one—where change and complexity are low and conditions are simple enough that we can choose to be either agile or traditionally plan-driven— and we clearly have a problem in quadrant four.

The Method Gap
The method gap we have in our process arsenal is exposed in quadrant four, where environments are characterized by both change and complexity. This gap would be less of a problem if current industry trends didn’t point to an increasing number of organizations being in the fourth quadrant [1]. Without the appropriate methods, fourth quadrant projects can exhibit schizophrenic, flip-flopping behaviors. These organizations want what agility promises, but too easily revert back to old plan-driven habits in the face of complexity.

Most people agree that what is needed is “agility in the large,” or what Scott Ambler calls “Agility@Scale” [2]. While current agile practices go a long way towards supporting self-managing teams, they need to go further in quadrant four. Here, we must move beyond processes that focus on internal, intra- team communications that typify agile methods and instead tackle inter-team coordination, addressing the interactions between different teams. If this takes us closer to plan-driven methods, then that may be the price we pay for operating in the fourth quadrant. Many teams find they are responsible for only one aspect of a larger solution and are necessarily dependent on other teams for success. Coordinating teams of teams, which some may be geographically remote or separated by contractual boundaries, is essential in large organizations, whether it is to execute an IT program or to manage the entire project portfolio [3].

Achieving this level of maturity requires us to embrace complexity and apply new, low-ceremony, cross-team techniques that leverage existing agile methods. Such process innovations are possible only when we are willing to “back up” from current notions of agility. Only by starting “back” earlier in the project lifecycle and stepping “up” to monitor how multiple developments streams contribute to a project or release can we achieve an “enterprise perspective.” And I would argue that such a perspective is the ultimate goal of agility: To move our currently used methods beyond development so they can become instrumental in guiding the evolution of the enterprise itself.

References
[1] Institute of Business Value, Capitalizing on Complexity, IBM Corporation, 2010

[2] Scott Ambler’s Blog, Agility@Scale: Strategies for Scaling Agile Software Development

[3] Johanna Rothman,Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects, Pragmatic Bookshelf, 2009

About the Author
Louis J. Taborda is managing consultant and founder of the Alinement Network, operating out of Sydney, Australia. He has been chief technologist formulating the ALM vision for a leading tool vendor and has consulted internationally, working across the project lifecycle to streamline processes and bridge the business and technology divide. He has over twenty-five years of software and systems experience and recently completed a PhD on managing change and architectural complexity. This work provided the foundation of the release-centric lifecycle described in Enterprise Release Management: Agile Delivery of a Strategic Change Portfolio. He is on LinkedIn and Twitter at @alinement.