Over the course of the past decade, Agile software development has progressed from a grassroots, almost underground movement, to the mainstream. Early successes have paved the way for broader acceptance of Agile principles and practices, facilitating dialogue not only in IT back offices, but corporate boardrooms as well. With an ever-increasing focus on profitability, time-to-market, and customer satisfaction, the vigorous debate over Agile adoption appears to be shifting from a question of "why?" to one of "how?"

To maintain momentum, the Agile community must shift its focus from "spreading the gospel" to architecting effective real-world adoption strategies. Addressing institutional adoption issues requires a less rhetorical approach which recognizes that Agile methods and Agile leaders must themselves embrace change and seek compatibility if Agile methods are to find a persistent role in modern business vernacular, whether specifically in the context of enterprise software development or in settings further afield.

Many complexities continue to challenge Agile's ability to penetrate new situations and demonstrate successes. At Sapient, we see the same issues slowing Agile adoption repeatedly. These barriers are predominantly organizational, and in many cases, self-inflicted wounds resulting from common Agile misperceptions such as:

• Lack of executive support
• Company culture and geographic influences
• Inadequate tools and infrastructure
• Skill gaps
• Dependencies on third parties that do not do Agile
• Co-existence with governance programs and regulatory constraints

At Sapient, we have found that successfully deploying Agile throughout the enterprise is 80% organizational change management and 20% about selecting the right Agile processes and tools. This 80% of an organization's emphasis - the hidden part of the iceberg - deserves as much, if not more, attention.

Organizational Change Collapses in the Absence of Executive Support

When considering enterprise-wide Agile adoption, executive support is essential as so much of the change induced is organizational. Reporting relationships, points of accountability, and interaction patterns with the business must evolve to support Agile. However, such fundamental changes can unearth organizational politics and unpleasant turf battles, such as when a quality assurance department disbands as a means of breaking down silos within the IT organization to build cross-functional teams. Further, making the right decisions for the business requires contemplating factors beyond Agile processes and tools. Financial and people management issues, for example, may drive the partial or non adoption of certain Agile practices.

{sidebar id=1}In one instance, Sapient was tasked with developing the core business analytics and risk management platform for a leading North American power and gas distribution company. Client executives recognized early that the nature of what was being developed demanded an Agile approach, and that the program's success would establish a new delivery standard for their IT organization. However, key business stakeholders for this project were energy traders, which meant that any time they spent collaborating with the development team was time away from the trading floor - the heart of the client's business. Ultimately, assessing the opportunity cost associated with pulling subject-matter expertise from the trading floor, as well as deciding who would spend how much of their time with the development team, required executive direction.

The Cost of Emotional Immaturity

Agile methods, such as continuous integration, encourage early risk identification and mitigation. When combined with more transparent and incremental progress reporting, customers are often exposed to an unvarnished view of reality - good or bad - much earlier than when using a waterfall approach. As such, we refer to Agile methods as "fail fast" in that if a project is so fundamentally flawed that it will fail for any reason, its flaws are more likely to surface sooner when an Agile process is employed. However, it can be very uncomfortable when things do not go as planned, especially in an organization that is historically less accepting of "failure."

Such circumstances threatened to derail an entire Agile pilot program Sapient was leading at a major North American casino gaming company.[i] Poor estimation and test environment contention led one pilot team to monumentally under-deliver on its first three iterations. At this point, faced with a very significant commercial impact, the business became understandably agitated and applied the brakes. As the "new kid on the block," Agile was an obvious target for blame despite the fact that the employ of Agile methods had nothing to do with the root causes of the team's performance deficit; rather, latent organizational issues that would have doomed the project irrespective of the chosen development process had been brought into focus, and through the use of Agile methods these fundamental issues surfaced sooner than would have had a waterfall lifecycle been utilized. Without strong leadership commitment, this descent into the "blame game" would surely have resulted in Agile's rapid exit from the organization.

Overcoming such awkward moments, and recognizing them as integral to the transformation process and valuable learning opportunities, requires emotional maturity. This is perhaps why The Standish Group went so far as to rank emotional maturity on par with the use of Agile methods in its most recently published CHAOS 10 factors for success.[ii] Here, too, executive support can play a key role in establishing an affirmative culture wherein the strategic view is valued foremost and leaders demonstrate a willingness to stick with change despite interim challenges.

Different Strokes for Different Folks

Distributing work across the globe can raise cultural issues that make implementing Agile principles more or less difficult depending on geography. For example, many of Sapient's German clients fully understand the benefits of Agile, but still insist on creating extremely detailed specifications up front. Similarly, self-organizing team concepts have been trickier to implement in India, where a deeply internalized bureaucratic and hierarchical mindset (no doubt a vestige of the British colonial system) manifests more commonly than in North America. In other words, every culture will potentially interpret and apply Agile somewhat differently. If your team is distributed across geographies, you must be aware of cultural idiosyncrasies and address these through clearly established ground rules and expectations at the time of project initiation.

Enterprise Agile Means Enterprise Tools

As long as Agile is the exception rather than the rule, the use of team-oriented tools is a fine strategy. However, as you progress into program- and enterprise-level deployments, the need for tools increases along with the breadth of required capabilities. Skimping on the right tools and infrastructure, as Sapient found during the course of its own Agile journey, can seriously hinder the transformation process.

If some of the following are true, it may be time to give up your story cards and whiteboards:

• Do you have or want an organizational metrics program?
• Do you have large or distributed teams?
• Are stakeholders that you need to collaborate with distributed?
• Do you have or want a quality management and compliance program?
• Do your teams collaborate with third parties?

The need for tools, however, has exposed a substantial gap within the Application Lifecycle Management (ALM) tools arena. Many ALM vendors offer a robust and comprehensive feature set, but not all their tools are as usable or affordable as they should be, relegating these tools to shelf-ware status. With almost all ALM vendors racing to jump on the Agile bandwagon, it's important for enterprises to make sure their tools investments will stand the test of time, providing direct support for both their current IT infrastructure and their Agile goals. Many enterprises fail to consider how Agile tools must coexist alongside existing tools/processes and future business plans. Key questions to ask when making this evaluation include:

• How well does the ALM vendor's product roadmap align with your organization's Agile objectives and how do mis-alignments, if any, reflect the vendor's ALM vision and the priorities of the vendor's existing customer base?
• Can the ALM product's architecture scale effectively in line with your business plans?
• Does the ALM product support the needs of geographically distributed teams and is its licensing scheme aligned to a global, mobile delivery model?
• If a Software as a Service (SaaS) solution is an option for your organization, what service levels must the vendor commit to in order to meet your business continuity requirements?
• Does the ALM product treat each project as an "island" or does it provide an organizational view of your entire project portfolio?
• If your organization is still in the process of adopting Agile methods or must partner with third parties who employ a different process, can the ALM product support Agile, non-Agile, and hybrid projects alike?
• What support does the ALM product provide for traceability, risk management, IT governance, and other processes that are not explicitly Agile?

Throw Away the Playbook and Find the Right Help

Adopting an off-the-shelf Agile methodology, such as XP, Scrum, etc., across the enterprise without tailoring it to specific needs is almost always disastrous. This is attributable to the fact that most Agile methods are woefully lacking in guidance for critical pieces of the development lifecycle, notable examples being requirements elicitation, user-interface design, and dependency management. [iii]^,[iv] Agile methods also offer scarce guidance on organization-level functions, such as governance, quality management, risk management, and business continuity. Often, we see organizations trying to replace processes with popular Agile methods without updating other elements of their delivery model to complement Agile principles.

For instance, a leading electronics retailer recently admitted to us that its teams were using Agile methods, but that the organization had virtually lost its visibility into project priorities. How can this happen when teams are collaborating closely with the business and capturing requirements as stories? The answer, as it turns out, was that while the client's IT organization was aggressively adopting Agile methods, it had simultaneously abandoned its existing PMO and many of its portfolio management practices. Result: the classic problem of trying to fit a square peg into a round hole.

Beyond acknowledging that adherence to Agile "by the book" is less important than business results, it is also important to recognize that the people who taught you Agile, and are perhaps providing consulting support for your pilots, are usually the wrong people to help deploy Agile throughout the enterprise. This is because learning Agile and getting your feet wet with pilots is a knowledge, skills, and concepts problem, whereas, rolling out Agile broadly across the enterprise is a change management problem.

When identifying individuals to lead your Agile transformation effort, look for people who:

• Exhibit an overt focus on business results.

• Have a demonstrated track record of major, enterprise-wide change management initiatives.

• Are familiar with issues specific to enterprise IT environments and are fully versed in Agile and traditional methods.

Our Own Worst Enemy?

Now let us turn to the role the Agile community plays in all this. To date, passionate and often rhetorical arguments have dominated many of the discussions taking place at Agile conferences, on message boards, and other venues. This has led to a largely binary view - either you are Agile or you are not, with few shades of grey in between. While this purist approach can be effective in "winning hearts," it can also be alienating, particularly when complex organizational realities must be dealt with as a prelude to any real and sustainable change. Thus, the Agile movement is poised at a critical crossroads and we are forced to ask the question: to what degree are we willing to compromise our view in order to innovate and create value across the industry? To quote Forrester Research:

"Based on our interactions with enterprise IT organizations, Forrester estimates that most large enterprises have some projects using Agile but that very few large enterprises use Agile for most of their projects."[v]

Thus, adjusting our attitudes toward wide-scale Agile adoption starts by accepting that few organizations today begin their transformation with a purely waterfall process and that almost none will become fully Agile. The goal should not be to make Agile an all-or-nothing proposition. Rather, the focus should be to methodically and pragmatically look at an organization's realities and then answer the following questions:

1. How far can we go and how far should we go in adopting Agile?
2. How long will it take and are we committed to seeing it through?

In his book Crossing the Chasm, Geoff Moore presents his widely accepted model for explaining how innovations are cultivated and, when successful, take hold in the market.[vi] Moore's model (see Figure 1) is divided into segments that account for the innovators who forge the raw ideas, the early adopters who latch onto these exciting new concepts, and "the rest of us." The most significant feature of Moore's model is an inflection point in the graph, the "chasm", which separates the "early" and "mainstream" markets; bridging this gap is determined by the ability to deal with the scale and accompanying uncertainties of the mainstream market. Exploding into the mainstream requires momentum - something the Agile movement has in abundance - and pragmatism - something the Agile community needs to cultivate.

To successfully navigate the chasm, we must embrace the complexities of the mainstream market and accept that factors such as distributed teams and regulatory compliance are not inherently evil but realities within leading businesses - realities that are not going to simply disappear. Deploying Agile broadly across an enterprise requires striking a balance with some tough and unwanted compromises along the way.

However, we might find these compromises less objectionable if we were to leverage the complementary concepts of Agile and Lean. For those unfamiliar with Lean, it comprises a set of potentially competing principles whose overarching goal is cost reduction via the elimination of waste. Agile methods, with their focus on customer value, first-time quality, and visual control arguably implement many Lean principles. However, in many instances, we've seen Agile purists seek to optimize value solely at the level of individual teams to the detriment of the larger organization. By applying Agile/Lean thinking at only the team level, larger organizational issues such as those we described earlier fail to receive adequate attention, creating discord that ultimately threatens the longevity of Agile within the enterprise. The Lean crowd takes a broader view, underscoring the need to optimize the enterprise as a whole. In other words, Lean proponents may implement things that slow down a few teams, but optimize the throughput of the overall organization. Agile purists seem to rail against this kind of thinking, but it is the "missing ingredient" the Agile movement so desperately needs in order to break into the mainstream and stay there. Today, we stand on the cusp of Moore's chasm and our ability to take the next step depends on whether or not we can embrace the pragmatism needed to catalyze mainstream Agile adoption.

About the Authors

Erik Gottesman isa Senior Manager at Sapient. Erik is responsible for identifying and developing the tools that support Sapient|Approach, the distributed Agile delivery model used bySapient teams globally to deliver client success. In this role, Erikprovides product direction for ResultSpace, Sapient's Agileapplication lifecycle managementsolution. Previously, Erik provided leadership and technical oversight on delivery engagements withHilton, Nissan, and major financialinstitutions.

Andy Takats is a Director at Sapient. After many years helping clients deliver leading-edge software systems (including work for the Philadelphia Stock Exchange, European Space Agency, and then for United Airlines, Williams Energy, Merrill Lynch, and others while at Sapient), Andyturned inward at Sapient to lead an overhaul of the company's delivery approach. Currently he is back in the fire coaching Sapient teams and helping Sapient clients adopt Agile methods.

[i] Laman, Robert, et al., A Winning Bet: Communication Processes at the Core of Agile Pilot Projects, Agile Project Leadership Summit, London, UK, 2006.

[ii] The Standish Group, CHAOS 2007 Research Exchange, 2007.

[iii] Gottesman, Erik M., Life Beyond XP?, International Conference on Project Management Leadership, Bangalore, India, May 13-14, 2005.

[iv] Gottesman, Erik M., Growing Agility in a Large and Distributed Enterprise, Agile Project Leadership Summit, London, UK, 2006.

[v] Schwaber, Carey, et al., The Truth About Agile Processes: Frank Answers To Frequently Asked Questions, Forrester Research, Inc., August 29, 2007.

[vi] Source: Geoffrey A. Moore, Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers, Collins, 2002.