Having troubles introducing Agile Development on your custom software development projects? Why not try moving to it gradually?

Implementing an agile development methodology on a custom software development project can be difficult. Many organizations new to the agile methodology struggle to adopt it. One of the big ‘problems’ with Scrum is that project-related issues come to the surface early because the team must deliver potentially release-able software within a month. Those issues in combination can seem insurmountable to those new to Scrum.

Typical issues/obstacles that arise include:

1. Lack of business ownership and the inability to make decisions,
2. Limited business buy-in into the concept of Agile,
3. Team communication, individual skills, and team fit,
4. Lack of trust in the team by the business.

In a traditional (waterfall) project, the team can spend months on design and when development begins they may start with the low complexity features, thereby not identifying issues until it is too late; forcing the project to run over-budget and/or delivering a solution that doesn’t meet the users’ needs.

When introducing Agile, organizations often attempt to tackle all of these issues head on and get overwhelmed with the new methodology, then choosing to revert back to what they are familiar with.

Why not introduce Scrum in stages to enable an organization to deal with issues one at a time and gain the benefits associated with solving each issue gradually?

Based on my experience as an Agile/Scrum Coach, I have found that introducing each stage below gradually over a number of months can work effectively for an organization that is struggling with Agile Development.

This article outlines a number of stages that a project team should go through in the introduction of Scrum. Each stage is summarized and then outlined in more detail below.

The first stage introduces prioritization of requirements, focusing only on the highest priority ones within a time-boxed two week sprint. Introducing prioritization can be difficult and issues with lack of business ownership and inability to make decisions must be overcome to enable you to complete this stage. The benefit gained is the ability to re-focus based on changing priorities.

At the second stage, you will overcome the obstacle of gaining business buy-in into the concept of Agile and ensure that the business agrees that quality and high priority business requirements are the top priority – over a detailed plan. You will have introduced estimation based on story points which will give the business improved control of the project in making decisions related to priority.

At the third stage, you will ensure the team is collaborating more closely; overcoming team communication and development skills deficiencies. You will have reduced the number of defects and the team will produce higher quality software quickly.

At the fourth and final stage, proper user stories and acceptance tests written by a Business Analyst will be introduced and the software will be frequently demonstrated to users so that their feedback can be incorporated. The business must learn to trust the team and agree to minimal documentation and sign-off. Once complete, teams are able to deliver software that is more closely aligned with what the user wants.

Each stage is outlined in more detail below:

Stage 1: Focus on the highest priority requirements within a time-boxed sprint

In my opinion, the biggest benefit to be gained by introducing Agile is the ability for a project team to re-focus based on changing priorities (and changing requirements). To gain this benefit, the first step is to focus on only the top priority requirements in the first two week iteration (called a sprint in Scrum).

Work closely with the business representative to ask the simple question 'what happens if we don't do that?' For teams that are less Agile, they will quickly realize they are working on a number of low priority requirements that do not necessarily provide much up-front benefit to the user.

The main goal of the initial prioritization should be to identify the minimum number of features required for the product to be released and demonstrated to the user. Teams can often think only of the end product that includes all features.

Once the first logical chunk of work has been identified, but before any development work begins, sit with the team to break down the requirements into tasks (referred to as sprint planning in Scrum). Get team members to commit to estimates so that everyone is aware of what they must do in that first time-boxed sprint. It is a good idea to use a board or software to track the tasks and ensure they are completed within the sprint.

The new focus for the business representative should be to document detailed requirements only for the next sprint (the next couple weeks). They must not spend a large amount of time writing specs for requirements that aren’t coming until the next release since this is often wasted effort as requirements and scope change.

Scrum features introduced at this stage

If you are familiar with Scrum, note that at this stage, we have not introduced many of the formal rules of Scrum. We have identified the Product Owner by asking the business to prioritize requirements using the Product Backlog. Prioritized requirements have been divided into a small chunk of work called a sprint and the team has performed sprint planning and committed to accomplishing their own tasks. In a future stage, they will commit to work as a team.

At the end of every sprint the software is released for testing. At this stage, I wouldn’t recommend attempting to release the software to testers multiple times throughout the sprint.

Issues overcome at this stage

Issues with decision making and business ownership must be overcome to enable you to complete this stage. It is often very difficult to identify the appropriate Product Owner who is respected by the business. In my experience with custom development projects, the best Product Owner is often a Senior Manager and therefore Business Analysts are required to represent that person at a detailed project level.

Developers always tend to under-estimate their work. When I start introducing Scrum I continually coach the developers to ‘over-estimate’ their tasks during sprint planning, but in the first few sprints individuals often over-commit as they get used to the concept of a time-boxed sprint.

Benefits gained at this stage

The business can see the return on their investment after a short period by gaining benefits from a scaled-down version of the product. After only a few weeks, the team has something they can release to the users. Because the team is only analyzing requirements coming within the next sprint, if those requirements change, the team can re-focus in a future sprint without the waste of business analysis on areas of the application that were not yet ready to be developed.

Stage 2: Communicate the benefits of Agile to the Project Sponsor

Now that the business requirements have been prioritized, and the team is working on the most important items using sprints, it is important to communicate this to the project sponsor.

Some coaches will suggest that this should be the first stage of introducing Scrum/Agile into an organization. I recommend that this is not done until after the first stage, because until the issue encountered in stage 1 related to decision making and business ownership is resolved, there is nothing to get agreement on from the project sponsor.

Once the requirements have been prioritized, it is important to ensure that the project sponsor agrees with the concept of Agile. The biggest advantage of Agile is that the team will provide the highest priority business requirements quickly and the solution will be of high quality. The biggest ‘disadvantage’ of Agile is that because quality is not negotiable, if issues arise, the lower priority requirements will be moved further out on the plan. In other words, they no longer have the Gantt chart that predicts (often incorrectly) exactly what features are scheduled when.

Instead of a detailed Gantt chart, communication with the Project Sponsor should include high-level goals (outlined in order of priority) planned for the next release. To be able to provide the goals planned for the next release, the developers must provide some high level estimates (story points – relative point system to estimate development time) for each requirement. Based on this the business is able to better prioritize and you should be able to come up with a rough idea of which requirements will make it in to the release.

The most important concept for the project sponsor to understand is the ‘Iron Triangle’, as depicted in Figure 1. One of the basics in Project Management is that with any project, if the scope (or the amount of features implemented) is increased then cost and/or time must also increase.

 

Figure 1 – The Iron Triangle

The key thing to focus on with Agile Development is that because you have outlined a fixed release date and have a defined team, time and cost will remain the same and scope is what changes. The Project Sponsor must understand that the goals for the release have been prioritized and if issues arise, the quality of the product does not suffer and we still meet the published release date – but features can be moved out into a future release.

Scrum features introduced at this stage

At this stage, we have introduced estimating the product backlog requirements using story points. I explain to the business the concept of high-level planning using Scrum in that a release is made up of a certain number of sprints each with prioritized features. You can also start measuring team velocity using the burn-down chart if the project sponsor is looking for proof of productivity improvements.

Issues overcome at this stage

The biggest issue to overcome at this stage is to gain business buy-in into the concept of Agile. If they don’t agree that quality and high priority business requirements are the top priority over a detailed (and unreliable) plan, then you should not proceed on to stage 3.

Benefits gained at this stage

The benefit gained from this stage is the Project Sponsor and Product Owner now have improved control of a project. It is much easier for them to prioritize requirements based not only on business priority but also on estimated story points. They understand that they will be delivered high quality software at regular intervals and it is completely up to them which requirements are included.

Stage 3: Improve Team Collaboration and Software Quality

The team now knows what high priority business requirements they should work on and the pressure of meeting an unrealistic project plan; which doesn’t accommodate for unforeseen issues and changing requirements, has been removed. They should already be feeling happier – so now is the time to get them working together more closely to help improve productivity.

Ideally the team should already be co-located, including Business Analysts, Testers and Developers. It is now time to introduce the ‘daily stand-up’, and the sprint retrospective to enable the team to talk about what they did well and where they can improve.

In the first stage, developers started committing to their own estimated tasks at the beginning of each sprint. Once that is working well, you can enable team members to select their own tasks from the planning board during the sprint. At the beginning of a sprint, rather than committing to individual tasks, they will be committing as a team to complete all of the user stories, which encourages self-management. The developers should help each other solve issues and collaborate on the best approach for design using pair programming where it makes sense.

Not only must the developers communicate more with each other, but the communication must spread to the Testers and Business Analysts as well. This is the only way we can define what should be built and tested.

When I first started coaching one team, someone told me that he felt like software testing was like going into an exam where they had no idea what to study since they weren’t sure what would be covered. This was because testers were ‘being creative’ and coming up with ‘new and crazy’ scenarios to test. There were so many ‘defects’ that every few weeks there would be a long defect prioritization session. This type of defect prioritization should never be required.

Instead, everyone should be very clear before development begins what will be tested so that the developers can write code that meets the requirements and defects are therefore minimized.

If the goal is high quality software or zero defects by the end of the release. The only way to accomplish this is to ensure that defects raised are ‘real defects’ i.e. there is a business requirement (or acceptance test) that is not currently being met. To ensure defects are ‘real’, testers must communicate regularly with the testers and developers. If in doubt, BEFORE raising a defect the tester must clarify that the issue they are raising really is a defect. If it is a new feature, it is instead added to the product backlog.

Scrum features introduced at this stage

The team now takes part in a daily standup for fifteen minutes or less at the same time every day. They are sharing tasks and becoming more cross-functional, working together to identify how they can improve through the introduction of sprint retrospectives.

The team is using acceptance tests to more clearly state the ‘definition of done’ so that both developers and testers know exactly what is expected. Software can be released to testing throughout the sprint rather than at the end of the sprint and defects should always be given top priority over new features.

Issues overcome at this stage

At this stage the team must overcome people issues. I’ve worked with teams where there were team members that either did not have the skills or the personality to work on a Scrum team. If they don’t have the skills, that is apparent very early on because they are not able to deliver the software within the sprint. Issues with personality I’ve seen include failure to communicate with the testers and business analysts due to the need to work independently. That doesn’t work with Scrum.

I have worked with a team that didn’t have any Business Analysts – just a Product Owner who was not available the majority of the time. In this case, we attempted to have the testers and developers write acceptance tests together at the beginning of a sprint. This was very difficult and productivity improved dramatically when Business Analysts were added to the team.

Benefits gained at this stage

Developers, testers and business analysts are all working towards the sam objectives in each sprint which will enable you to reach the goal of zero defects at the end of each release (and hopefully each sprint). This high-quality software will be provided more quickly due to increased team velocity due to sharing tasks and increased communication.

Stage 4: Ensure requirements are user stories that incorporate real feedback

Now that the team is developing high quality software and productivity is starting to improve it’s extremely important that you ensure that what they are building is exactly what the user wants.

In traditional projects, a lot of up-front analysis is done and requirements may be outlined either as technical or functional features for the developers to code, with no explanation of who needs it and why. Because of this, the team may still be building software that does not meet the user needs due to misinterpretation of the user (by the Business Analyst) or misinterpretation of the Business Analyst (by the Developer).

At this stage, it is important that the Business Analyst steps back and examines the high priority functional requirements scheduled for the next iteration to ensure that they fully understand the requirement and are communicating it clearly.

The best way to minimize misinterpretation is to write requirements as user stories. They should be worded in the following way: ‘As a’ user ‘I want to’ do the following ‘so that’ the following benefit can be met. The Business Analyst must make it clear what is expected at the end of a sprint – also referred to as ‘the definition of done’. To communicate this clearly acceptance tests should be written.

To ensure that the team has actually developed what the user wants, at the end of each sprint, demonstrate the solution and get feedback from the user! For software that is already in production, ensure there are frequent releases so that real user feedback can be incorporated. Note that the best way to enable your team to release software frequently is to introduce automated regression testing.

Scrum features introduced at this stage

Business Analysts must learn to write requirements in a new way and the business must agree to minimal documentation and sign-off. This is only possible if you have someone responsible for the product ownership that the business trusts.

The best way to demonstrate that limited documentation still works is to demonstrate the solution to the users regularly and incorporate their feedback – this is a key feature of Scrum. Ideally, the team has been producing working software at the end of every sprint, but if they haven’t been, now is the time to enforce this.

Issues overcome at this stage

It can be difficult to get organizational buy-in on limited documentation. Many large organizations are resistant to this type of change due to lack of trust in the team.

Another obstacle to overcome is the re-definition of team members’ roles. I’ve worked with teams who had an architect telling all the developers exactly what technical feature they should add. This not only impacts the morale of the team, since they are not able to be creative, but because the team did not know why they were developing a feature, they often built features that did not meet the users’ needs.

Benefits gained at this stage

The benefit gained at this final stage is that the software developed better meets the business needs. Expectations are appropriately set with the users and the software meets those expectations.

Conclusion

If your organization is having troubles introducing an Agile methodology on a custom software development project, try introducing the above stages gradually. By introducing only one stage, you can still gain the most important benefits of Scrum – the ability to re-focus based on changing priorities. Once your team has overcome the issues associated with that stage, you can move onto further stages and gain added benefits with each issue you work through.

In my experience, there are a large number of organizations that are adverse to change – especially large corporations and government organizations. In these types of organizational cultures, it is best to deal with one obstacle/issue at a time rather than getting overwhelmed with all issues at once.

I recommend that you first ensure the business is taking ownership and making decisions. I think it is best to focus on overcoming this obstacle first before moving onto the second stage.

Once a Product Owner has taken control of a prioritized product backlog and the team is working on only the top-priority requirements, it is then time to explain the benefits of Agile to the business. Once they have bought-in to Agile and are given more control over the project, it is then time to focus on improving software quality.

Improving team collaboration will ensure that developers, testers and business analysts are all working towards the same objectives in each sprint will enable you to reach the goal of zero defects at the end of each release.

The final goal is better met user expectations which I think is only possible through the introduction of user stories, acceptance tests and regular demonstrations with users. Getting the business to agree to new processes around documentation can be difficult as it requires that trust in the team is established – which again is difficult to do until the initial obstacles have been overcome.

---

About the Author
Lorraine Pauls Longhurst is a Certified Project Manager with expertise in the use of Agile software development methodology (SCRUM / ADM) with LPL Software Consulting based in Sydney Australia.