While many of us subscribe to agile methods and the ideals they represent, too often the organizations we work for force us to do things that are in direct conflict with agile principles. This is particularly the case for IT departments in larger organizations, where agility is evident as good intentions rather than being the reality on the ground.
Just over a year ago, the financial services company I was working with initiated an agile project with a lot of fanfare. It had management sponsorship, was given a catchy name, and the development team was led and trained by leading Scrum consultants. Today, the mention of that project, which resulted in the manager who led the initiative now maintaining a low profile, causes embarrassment among the remaining members of the development team. The project’s failure to deliver on expectations has meant that other agile initiatives have suffered and the crater it leaves ensures that it will be some time before another agile project is given the same level of support.
I’m sure this is not an isolated instance of well-intentioned agile projects going wrong; you may have your own stories to share. In this particular case, the reason team members gave for their failed attempt at agility is that the project did not have control over its own destiny: There were too many constraints and external dependencies to deal with. These constraints arose from the fact that the project had to work within the boundaries of the original business case that established the scope in advance and placed limits on the planning of each sprint. Furthermore, while the agile team members were “sprinting,” other teams that the solution relied upon were progressing at their usual, relatively glacial pace. In other words, while the business saw the project as a singular independent initiative, the solution architecture relied upon other components to achieve the desired outcome. In reality, project success was dependent upon other teams in the enterprise.
This example shows that to be successful in large organizations, agile principles must encompass all aspects of a project. You must be agile early in the lifecycle when the project’s business case is developed, rather than only being agile in the software construction phase alone—or worse, restricted to a single development team. Routinely creating such a business case calls for estimating the entire project scope and establishing the high-level solution design that often impacts multiple teams,
To assess an initiative’s (economic) feasibility and have the budget agreed upon, IT projects are forced to lock-in high-level requirements and make early design decisions, long before any software is written. As a result, specialist roles, such as business analysts and solution architects, are needed in the early stages of an IT project as they are responsible for vital, early-planning decisions. The people who work in these roles routinely work with the customer in order to scope the solution and liaise with the development teams responsible for its implementation, thereby establishing a project’s financial and technical envelope.
It’s in these ways the waterfall reasserts itself, even in so-called agile projects. Development teams are separated from their customers and relegated to working through proxies who communicate the project requirements. This behavior is exacerbated when a project is reliant on outsourced or offshored development teams for critical capabilities, thus resulting in IT projects operating in a more contractual environment where even agile teams must adhere to approval gates and review processes. While this certainly challenges the conventional notions of agility, it also represents the next frontier for scaling agility: Not only must you scale software development across the multiple teams constituting many IT projects, but you must also scale the coordination, planning, and design.