Embracing Change and Complexity

The Old “Divide and Conquer” Trick
Arguably, it is solution complexity that drives the described behaviors resulting in formal processes.  These processes are an attempt to reduce the customer’s financial risk since complex, multi-team solutions have a tendency to make project sponsors (and pretty much everyone else) nervous. So increasing project complexity usually demands greater “management visibility.” Formal handoffs between the analysts, designers, and developers involved can result in copious documentation being produced along the way. In attempting to “demonstrate” their understanding of what needs to be done, detailed use cases and system behaviors are described in words and models rather than working software, As a result, a project team can suffer information overload early in the lifecycle with the risk that they lose sight of the forest for the trees.

Of course, no project is intentionally set up to be slow and ineffective, or made to create impenetrable and expensive documentation instead of demonstrable capabilities. Yet, the loss of agility can appear to be inevitable when the problem domain is complex and the solution architecture grows correspondingly sophisticated. Traditional “divide and conquer” management patterns routinely see dedicated teams maintaining IT platforms or system component, focusing on a specific technology silo. However, even a simple regulatory change in the financial services industry can result in changes across multiple IT platforms, making it necessary for you to coordinate a multi-team change program and establish protocols for inter-team communication.

Sadly, it seems problems are more easily divided than they are conquered. While analysis and design can decompose complex programs and help us understand what needs to be done, to truly “conquer” we also need synthesis; this encompasses the merging of separate development streams and integrating them to ensure the teams work towards the one solution. It is in synthesis where many development lifecycles are weakest, as the emphasis is usually on the creation of components with the implicit assumption that they can be assembled when needed.

Continuous integration now provides powerful tools to bring multiple software components together, but in IT there are no equivalent tools to integrate the non-software aspects of a solution. In many business environments, such as the financial service industry, changes to distinct and independently managed IT platforms need to be integrated. These can be sophisticated systems in their own right, so it can be challenging to coordinate the design and development efforts of these separate teams (and sometimes whole organizations). Interfaces need to be agreed upon and in more devolved, outsourced, or offshored environments, their specifications become increasingly formalized and quickly morphs into that most insidious construct: the contract. 

These complex more scenarios demonstrate the need to scale agile methods, which arguably remains best suited to singular software teams rather than multi-team, multi-disciplinary enterprise projects. How do we attain the Holy Grail of agility in the face of growing complexity? Clearly, we cannot give up on larger enterprises that want and badly need to be agile; after all, they provide many of us with our day jobs. And, while many projects in such enterprises encompass non-software development scenarios, including business processes and hardware or networking infrastructure, they heavily rely on software and represent an opportunity to apply enterprise-scale agile techniques.

Balancing Change and Complexity
In facing the dilemma of agility when dealing with complex and pre-existing architectures—be they software, systems, business, or organizational—further analysis of our very own problem domain can provides useful insights. The conditions under which we attempt to use agile methods differ depending, for example, upon whether we are developing a new software product or working on an IT project.