Embracing Change and Complexity

For agile projects to be successful across a range of settings, we need to make some sense of our collective experiences and better understand the challenges practitioners face in attempting to be agile in the enterprise. Clearly, current agile methods work exceedingly well in some environments but can struggle in others. While many of us appreciate the elegant simplicity of story cards and burn-down charts, applying such techniques can be a challenge in IT projects where multiple teams must working within contractual constraints.

While we may be geared to embrace change when being agile, we must also embrace complexity. The latter is inevitably challenging, because the more quickly we develop software and the greater the sophistication of the solutions we build, the more difficult it is to maintain agility. That’s the nice way of putting it. Sometimes, in attempting to shrink time to market, we may also create complexity by making poor design decisions and establishing architectures that are inappropriate or do not scale. For example, while we all may be able to understand and use basic mail-merge capabilities for our personal use, the same functionality in a banking system requires a highly secure, robust architecture when used to send out customers’ monthly financial statements. We must accept as a fact that sometimes complexity is just a reflection of the problem space, and there may be no option to remove it, whether starting incrementally or resorting to simpler development processes. Sometimes we are forced to build complex systems from day one.

While agile methods have mindshare, these are the instances where the plan-driven, waterfall processes stubbornly prevail, because it’s often that the solution is not entirely software-based. Instead, these solutions rely on people and business process that are less amenable to agility. Additionally, there can be hybrid situations, as in the example given earlier, where projects attempt to be agile even when they must work to reach predetermined planning milestones.

Not everyone may agree with this assessment that sees agility as desirable but not always attainable in organizations; this only underscores the point that there is a spectrum to agility. My research showed that change and complexity can be a useful way of framing our different experiences, as they represent the underlying factors affecting our ability to be agile. The degree to which these two fundamental forces are present helps determine the approach best-suited to a particular situation. We need to be more precise about what constitutes complexity, but for now—using the ubiquitous four-square grid in figure 1—we can view traditional plan-driven and agile methods in terms of where they sit on the change and complexity axes.

Figure 1. Sweet spots for agile and plan-driven methods

Figure 1 indicates the coverage of the two approaches and represents the agile and plan-driven “sweet spot” as separate ellipses. The different quadrants provide a simple means for categorizing the conditions a development team may find itself in. Agile methods are best suited to managing the change dimension, so its sweet spot spans quadrants one and two. Plan-driven approaches are appropriate for the management of large-scale, complex development programs, so they cater well to quadrants one and three where low change volatility allows more detailed planning. This grid suggests that we have a choice in quadrant one—where change and complexity are low and conditions are simple enough that we can choose to be either agile or traditionally plan-driven— and we clearly have a problem in quadrant four.