Agile adoption has begun to grow in industries such as insurance, telecom and financial services; but for agile development to become mainstream, it needs backing by major industry players, a unified rather than scattered agile landscape, and evolutionary rather than revolutionary agile transformations.

We need improved support for large-scale agile transformations, including knowledge transfer on a large scale, necessary HR policies, and scalable agile tooling architectures. Finally, we need to address development issues that large organizations deal with, including large-scale development projects, geographically distributed development, and compliance. Let's take a look at what promising work is happening in each of these areas.

Mainstream Companies Are Looking For Safe Bets
In Crossing the Chasm, [1] Geoffrey Moore points out that for a product to make the transition from “bleeding edge” early-adopter technology to wider marketplace acceptance, it needs to appeal to a growing majority of consumers. Mainstream organizations will go for what they deem are safe bets rather than venture into unknown territories. They will go with the market leader rather than hottest technology and seek continuous improvement rather than radical paradigm shifts.

Consider how Moore's observations relate to making agile development mainstream:

Backing by major industry players
A cottage industry of players have entered the agile market, including services companies such as Object Mentor, Mountain Goat Software, Net Objectives, Quadrus, and Industrial Logic, and product companies such as Rally Software Development and VersionOne. Most of these companies have small staffs of less than one hundred people. Until now, the presence of major services and product companies has been lacking. This is changing as we speak.

For example, the world's largest system integrators have recently put more focus on agile development. IBM Global Business Services just launched an Accelerated Solution Delivery Practice focusing on agile development and agile transformations, Capgemini is investing in an accelerated delivery platform for agile development, and even a few Indian system integrators, such as Cognizant and ITC Infotech India, are moving towards agile development.

Major vendors are investing in agile tools and processes. IBM has recently previewed Jazz, a next generation technology platform for, among other things, agile and collaborative development. Microsoft recently launched MSF for Agile, an agile process, and IBM, Telelogic, Capgemini, and 15 other large and small companies are co-developing the Eclipse Process Framework (EPF), [2] an open source process initiative focusing on agile processes.

Unifying a scattered agile process landscape
As agile development is growing in popularity so is the number of agile processes. These include XP, Scrum, OpenUP, AUP, DSDM, Lean Software Development, Adaptive Software Development, Rational Unified Process (RUP), MSF, FDD, Crystal Clear, EssUP, and Agile Modeling.

Many of these processes cover only certain aspects of the software development lifecycle. As an example, Scrum only covers project and requirements management aspects and needs to be integrated with other processes such as XP and Agile Modeling to address the full lifecycle. Integrating different processes is difficult and time consuming, and mainstream companies do not want to take on that investment.
The EPF project, however, addresses the process integration issue. EPF is an open source initiative providing an authoring, configuring, and deployment platform for software best practices. The project was launched earlier this year, and many leading agile processes are, or will soon be, available within EPF, including OpenUP, XP, Scrum, DSDM, and Agile Modeling.

Within EPF, we hope to leverage all of these agile methods to eventually develop reusable agile process components, which can be combined to produce different agile processes, such as OpenUP, XP, and Scrum, or to produce your own agile process. Organizations should be able to add or modify process components, use them internally, or make them available for free or sell them.

Evolutionary or revolutionary?
Agile development is often presented as an abrupt paradigm shift-a scary proposition for many companies. Agile transformations can be a continuous journey. You may transform additional projects every quarter, and each time, you learn and improve your transformation process further. You may introduce practices a few at a time. For example, you may start with iterative and test-driven development, and then follow up with pair programming and collocated teams.

Certain agile practices, such as self-organized teams, do represent true paradigm shifts. There is a big difference between allowing team members to influence decision making, and having team members truly control their own work and destiny. When a team is faced with a difficult choice, and team members turn to their manager for a decision and the manager replies, “I’m going out for a coffee, let me know what you decide,” it creates a radical shift in people's understanding of who is in the driver's seat. This can lead to a new level of collaboration and motivation, generating an often-radical productivity boost.

We also need to articulate what organizational factors are impacted by an agile transformation, and what commitments are required. If you are not prepared to revisit, for instance, how you approach testing, how you motivate and reward people, how your IT organization is collaborating with your line-of-businesses, and what tooling infrastructure you have, then full-scale agile transformations are not for you.

Dealing with large-scale organizational transformation
You can transform the way ten people work by having a skilled agile coach spoon-feed information to the right people, but transforming an organization of hundreds of people will also require the availability of a reliable infrastructure. Let's have a look at some of the issues that need to be addressed.

Knowledge transfer on a large scale
As consulting firm Industrial Logic was tasked to do an agile transformation of a 600-person development team at Kronos, it realized that just relying on a handful of coaches would not cut it. To help transfer basic knowledge about XP, the team built Web-based training courses, which allowed its coaches to focus on the higher-value activity of coaching teams on the practical application of XP. Large-scale adoption requires complementing traditional coaching models with electronically transferred knowledge, whether it is through Web-based training, process guidance, tutorials, or other means.

Changing HR policies
Agile development holds implications for traditional HR concerns, such as employee incentives and rewards, career paths, and responsibilities. We need early support from HR departments and managers in carrying out the necessary changes.

Agile development requires close collaboration among a set of people who regard each other’s as peers, which can change the dynamics associated with team leadership. Typically, a job promotion (from developer to architect, for example) is seen in larger organizations as a means to escape the dirty job of writing code. But these organizations need specialized and experienced people who serve as mentors in their team-lead roles, not as supervisors who have earned the right to sit on a pedestal.

A focus on collaboration also changes how you reward and measure people. Rather than measuring only individual productivity, you need to measure how much value an individual provides to the team. The most valuable team members often have lower individual productivity, since they spend most of their time mentoring other team members.

These HR-related notions reflect agile values, and it is crucial that an organization articulates the values it wants to live by in rewarding and promoting its people. [3] For agile development to become mainstream, we need to better articulate required HR changes and how to accomplish them.

Providing a scalable tool infrastructure
A key agile principle is to focus on individuals and collaboration over tools, and possibly as a result, many agile coaches are tool adverse. However, tool infrastructures can greatly facilitate collaboration and are crucial to making agile mainstream. True, some traditional tools may not be conducive to agile development and may inhibit meaningful collaboration. But there is a promise on the horizon-a next generation of tools focused on the collaborative aspects of software development is being developed.

VersionOne and Rally Software Development have launched agile project management tools, providing support for core agile concepts such as velocity, iteration planning, size and effort estimates, as well as breakdown of work into small units that can be accomplished in a couple of days or less. The IBM Rational organization, in collaboration with IBM Research, has recently previewed technology code-named Jazz, which goes beyond current agile project management solutions to include also agile development support, adding agile concepts such as team collaboration, transparent development, continuous integration, and test-driven development as first-class citizens. Jazz is also process-aware and can be tailored to support different processes.

Large-Scale and Corporate Development Issues
For it to become mainstream, agile needs to address the problems facing large organizations, such as support for geographically distributed development, large-scale development, and compliance. Current agile solutions do not clearly articulate solutions to these issues, and this needs to change.

Large teams and larger organizations
Many agile processes explicitly target smaller teams of a dozen or fewer highly skilled developers, but more experience and guidance needs to be provided for dealing with large teams. Support for larger teams is in development for processes like XP, Scrum, OpenUP, RUP, and Agile Modeling [4]. For example, IBM is currently refactoring RUP to be built as a set of extensions to OpenUP. This will add guidance for domains such as SOA, compliance management, geographically distributed development, and packaged application development to an agile core process.

Geographically distributed development
Geographically distributed development has become a matter of life in most larger development organizations. But face-to-face communication is a core principle of agile development. Organizations can compensate for the lack of shared physical project room through a combination of modified practices, such as more effective distribution of work across different teams, and improved supporting infrastructure. Jazz technology provides team awareness, allowing team members to see who is present and what they are working on. Various technologies such as instant messaging allow collaboration in the context of the work done. The goal is to leverage available technology to create a cohesive team, even if team members are miles apart.

Compliance
Compliance means Say what you do, Do what you say, and Show you did it. This is not inconsistent with agile development, but there are a few hurdles to overcome.
First, the agile community needs to overcome the inherent fear of documenting processes. Agile teams should continuously evolve the process to ensure it works for them, but the fear is that when you document the process, it becomes hard to change. EPF and RUP address this concern by separating a rich reference library of practices from a succinct description of how the team chooses to work in their specific project. There is no need for massive changes to elaborate process guidance; instead, teams change their short process description and associated pointers to reference material. In 2007, EPF will add Wiki technology to make these changes even easier to ensure team ownership of their process.

Second, teams need to avoid non-productive work to "show you did it." IBM Rational's experience shows that automation can allow necessary book keeping to be done so audits are passed with no or limited extra work.

Conclusions
Agile development will have to overcome many hurdles to become mainstream, but I expect tremendous progress in 2007. Leading systems integrators and vendors are investing in agile development, consolidation may happen in a scattered agile process landscape enabled by EPF, and agile transformations are hopefully increasingly presented as a journey rather than an “all-or-nothing" strategy. Tools specifically built for agile development are emerging. Much work is left to do in the areas of compliance, support for large-scale agile development, and geographically distributed development, but here also we see emerging practices and supporting infrastructures that will help making agile development mainstream. All in all, there is good hope that 2007 will prove to be a year where agile development is crossing the chasm and reaching mainstream development organizations.

[1] See Crossing the Chasm, Geoffrey A. Moore, Collins, 2002.
[2] Seewww.eclipse.org/epf/ for more on EPF.
[3] See Practice 12 in Agility and Discipline Made Easy, Per Kroll and Bruce MacIsaac, Addison-Wesley, 2006.
[4] See “Scaling Agile Development via Architecture”