In June 2002 I was hired as part of a turnaround team for a midsized company. As VP of engineering, my role was to transition a struggling engineering organization into a thriving team. By the time I left in October 2004, the organization—historically known for late delivery, poor quality, and low morale—was converted into an energized team that produced high-quality software on schedule. Here's the story.
I was responsible for five development teams that were working on five applications. The teams had duplicate and disparate infrastructures and development tools. My first step was to standardize all teams on the same infrastructure (bug tracking system, code control, etc.). The teams had been working with an extended waterfall process with no automated testing. At my previous job, a startup, we'd experimented with Extreme Programming (XP) and had good results with some of its practices, so I began introducing the teams to Agile development concepts:
The teams implemented continuous builds and automated testing, and switched to iterative development.
When forming a team for a new application or replacing engineers who had left the company, I hired highly skilled senior-level engineers with backgrounds in Agile concepts. They helped reinforce these concepts and brought enthusiasm for the changes. After some fits and starts, the teams started to realize the benefits of the new practices and continued improving in these areas.
Kick it up a Notch
Although the teams had improved, by the end of October 2003 it was clear more was needed. We still faced serious problems with getting requirements defined expeditiously and into the engineers' hands. I decided we needed to take our process to a more radical level: fullblown XP. I knew we needed to convince ourselves that this change could be successful. We started with a pilot project. A new team would try it first, using full XP to build a prototype of a new application. If that succeeded, we'd switch a team working on a legacy application. The idea was that with each success another team would switch until all teams were running with XP.
We had a typical cubicle environment, but for this pilot project we made an open workspace that we called The Pit. We moved everyone working on the project (engineers, QA, technical publications, technical project manager, and project manager) into The Pit. It significantly sped up and improved all communication.
I knew that I could not convert the teams without help. I brought in an outside consultant who was well known in the XP community. His task was to educate all of engineering on XP concepts and to help the first team moving to XP to get to a highly functioning level as quickly as possible. Because of his credentials he provided credibility to the effort and helped relieve the initial confusion. He returned each month to provide coaching and education.
The pilot project started with an existing high-level product requirements document (PRD). Normally, a technical product manager (TPM) would turn that into a detailed functional specification from which the engineers would design the technical implementation. Instead, we had the TPM sit down—first, with the marketing person who'd written the PRD and, later, with the engineers—to come up with a list of stories and estimates that represented the PRD. After that, the TPM ("Customer" in XP terms) gave the team members stories, one iteration's worth at a time, and explained whatever they needed to know. The team no longer had to wait on a completed and reviewed functional specification to get started, and the Customer was right there in The Pit with the team, ready to answer questions, give advice, and validate that the engineers were building what the requirements intended.
The pilot project was completed in December 2003 with a demonstration of the prototype to the executive team. They were impressed with the maturity of the application. Both the Customer and the engineers felt the XP experiment was a success. So in January 2004 we began rolling XP out to other teams (some of them had been experimenting earlier). It was tougher for the other teams because they were working with legacy code. It's hard to change legacy code safely and it's difficult to add the automated tests that make those changes safe. Even so, by March all the teams were working with XP at a fairly functional level and experiencing huge improvements.
Some of the engineers were skeptical, but all were willing to give it a chance. They had witnessed the success of the pilot, and each engineer knew in his gut that the problems that affected his ability to improve his products had to be addressed. Their willingness to give it a shot was crucial to our later success.
The action that symbolized our complete conversion was tearing down all cubicles and replacing them with a large open space for all the teams. Many of the engineers had been reluctant to work in an open, noisy space, but within a month, most of them came to like it. To provide private space we built a quiet area with couches and set up "hotel cubes" that people could use for private work or phone calls.
By the summer of 2004 we clearly had succeeded. All teams were entrenched in XP at varying levels of maturity. All teams recognized the changes we had made as the source of huge improvements in delivering on time with fewer defects. When there were new requirements, changed requirements, or unexpected problems in development, we could adjust scope to match reality. Because we were delivering working software all along the way, we were always reporting on the truth—not on what we thought the truth would be at the end. As a result, we were better able to help customer support, marketing, and sales prepare for what features were coming, train on those features, and set customer expectations. It was widely acknowledged that we had turned around an organization of more than 100 people.
Would I say that my organization was text book XP? No, but we were pretty close and the teams were focused on improving. The important thing for me was keeping the teams focused on the vision of the very best practices of XP and making sure they didn’t slide back into old habits. I wanted the teams to really try each practice in XP, not say, "that one doesn't seem right" before giving it a fair shake. I had to create a safe environment for them to attempt new things and make mistakes. Following XP helped us avoid going too far off course without time to adjust. If you're going to try a big change, pick an Agile process to follow (XP, Scrum, etc.), set the bar high, and go all the way. It worked for me.