Calculating ROI on your Investment in Requirements Management Tools

Overview of ROI
Return On Investment (ROI) is a popular method of measuring the success of process improvements and IT investments. It is a measure of the dollars returned on dollars invested. As Jeffery E. Payne points out in his article "Quality Meets the CEO: How to Get Management Buy-in," ROI is an effective approach for arguing the need for, or demonstrating the success of, process improvements and IT investments.

Though there are a number of methods of calculating ROI, one straightforward, simple to understand method is the Benefit to Cost Ratio, which is simply dividing the benefits in dollars of process improvement or IT investment by the costs. Benefit to Cost Ratio = Benefits/Cost. So, a Benefits to Cost Ratio of say, three, would mean that for each $1 spent on the cost of process change and IT, $3 in benefits were realized.

In doing an ROI assessment, typical sources of cost include:

• IT investments, including ongoing maintenance
• Training staff in new processes and IT tools
• Consulting needed to assist process change and IT installation
• Recurring cost associated with new process and IT

Typical benefits that are considered in an ROI assessment include:

• Increased revenue, e.g. increased sales, or sales margins
• Retention of sales that would otherwise have been lost
• Reduction in operating expense, e.g. daily time savings, eliminated rework

Let's look at how to do a Benefit to Cost model for process change and IT investment of putting a requirements management tool in place at your company. Though this model was developed in conjunction with the rollout of a commercial tool, it should be readily adaptable to development and rollout of "home grown" tools.

Requirements Management Tools
Requirements Management tools are to requirements what Defect Tracking Tools are to defects. They provide an environment for, and database approach to, managing large numbers of requirements related in potentially complex ways (traceability). Requirements tools provide continuity through time, across projects, through staff changes, and through company re-orgs. They are a corporate memory for requirements.

In a project, setting a Requirements Management tool is used in a number of contexts:

• Planning the scope of a release, or a series of releases (key in iterative, incremental projects)
• Managing plan execution: who is responsible for what, when
• Tracking project status
• Change control of scope, particularly in evaluating the impact of proposed changes (this is where traceability of requirements is critical)

For a large company, dealing with thousands of requirements, moving to a database approach to managing requirements, allows a metrics-based management of requirements that is simply not practical with paper document-oriented approaches.

Calculating the ROI
No question that a requirements management tool is very useful, but can it pay for itself at your company? In his paper Calculating Your Return on Investment from More Effective Requirements Management, Dean Leffingwell advances this model for estimating ROI (numbers are from his example):

One can then make an argument that if the requirements management process was improved by say 10%, that is a cost savings of $504,000 x 10% = $50,400. If the cost of process improvement (e.g. training), plus tool purchase was a total of $19,900 (again, numbers from his example) then the Cost to Benefit Ratio is $50,400 / $19,900 = 2.53.

The model I will present here is a bit different from Leffingwell's; it will provide specific improvements one might expect from installing a requirements management tool. I will then try to quantify the benefits from that perspective. The model is based on one I developed for a company of about 500 R&D staff, 1000 employees overall. The tool, a commercially available product, was deployed in four cities. The ROI assessment was done about 1.5 years into the rollout.

Assumptions and Conventions
We begin by making a number of working assumptions and conventions that will be used throughout the ROI model. The model is developed in an Excel spreadsheet.

Cells with black text are ones for which you must input values that make sense for your company.
Cells with red text are ones with values computed via some formula.
Cells that are gray filled have values that are subjective.

These fields are the main source of uncertainty in the model. In interviewing staff as part of the rollout, the values for these fields varied widely.

We begin by calculating the cost of a fully burdened employee, per work day, per hour, and per minute. By "fully burdened" we simply mean the total cost to
the company: salary + total benefits. As a rule of thumb, the fully burdened cost is usually about 1.5 times the salary.

Next, we capture the actuals in terms of numbers of requirements that were entered into the requirements management tool as of the date of the ROI assessment; about 1.5 years after initial rollout. Of the approximately 5000 requirements entered, about 1000 had already been implemented and shipped as part of some project, and some 200 had been rejected, meaning a decision had been made that these requirements would never be addressed in any future release.

The Cost
An ROI assessment must be done for a fixed period of time; both the cost and the benefits must be calculated for the same fixed period. This study was done about 1.5 years into the rollout of the new requirements management process and tool.

Consulting cost was for onsite support beyond classes, i.e. additional support that was purchased to aid in the rollout of the tool and process. The Rollout Team was a core set of staff that spent a portion of their working time in support of the process and tool rollout. This number is probably a bit low as it doesn't account for travel expenses.

Another cost we account for is tool use overhead. Just as the proper entry of a defect into a defect tracking tool requires some time, the proper entry of requirements into a requirements management tool requires more time than the capture of requirements on the back of a napkin, or excel spreadsheet. As noted above, the values entered in the gray cells are highly subjective.

Finally, we account for the added rigor in requirements management that we are asking teams to make. For example, requirements that would otherwise not be recorded, or recorded on a white-board in an office, are now entered into a public record. This increase in easy-to-access requirements leads to additional review, discussion, test planning, and change control. The cost of doing the job right is, nevertheless, a cost, and is captured in the ROI model here.

In Ed Weller's Calculating the Economics of Inspections (StickyMinds, Jan 2002) he states the recommended rate for preparation and inspection of a requirements specification is about seven pages an hour. This seems a reasonable heuristic for estimating the average cost of additional review and rigor on requirements that were actually implemented. The requirements that were entered, but not implemented, receive much less scrutiny, here modeled as one-tenth the effort, or a factor of 10 increase in speed at which they are reviewed. For the model we will assume 1 requirement = 1 page.

Benefits
Let's move to calculating the benefits of the requirements management tool. Of the two broad categories of benefits discussed above, increased sales and reduced operating costs, this one will focus solely on reduced operating expenses for the company. This is fairly common for most IT investment ROI models. In the case of a requirements management tool one could probably make an argument that increased sales, or at least retention of sales are realized due to increased customer satisfaction resulting from better requirements management.

Cost Savings from Staff Working More Efficiently
We begin by calculating the savings realized from staff on projects having a readily available, always-up-to-date, common source of requirements upon which they can base their work. It is a cost reduction from staff working more efficiently to plan, develop, test, document, and develop training materials for a product.

Scenarios where inefficiencies occur when requirements are not readily available to staff include:

• Requirements exist only in Joan's head, so each and every staff member that needs that information to plan tests or write a user manual makes a trip to Joan's office.
• Requirements are recorded on Joe's laptop, but Joe is at the client site for the next two weeks.
• Bob writes the user manual based on the woefully outdated hardcopy requirements specification document he has on his desk, requiring significant rework later.
• Requirements kept on Sue's laptop are lost when laptop is stolen at the airport!

Note that this part of the model is not about doing a better job; it's about doing the same job more efficiently.

Avoiding the Cost of Lost Requirements
Staff churn is a common problem for projects. Staff quit the company; staffs move to other projects or are promoted; whole projects get re-staffed through company reorganization. When requirements are not recorded and managed in a companywide system, they are subject to loss due to staff churn. The loss means that the requirements must be "rediscovered" and re-engineered again and again. Human Resource groups report that 12 percent staff churn per year is not uncommon. Requirements kept on laptops are subject to loss when the laptop is stolen at the airport!

Avoiding Cost of Unnecessary Development
One of the benefits of a companywide requirements management tool is the increased visibility that requirements receive. On Projects employees are able to see what each other is doing; redundancy is spotted; priorities are better managed. The result: requirements get rejected. This leads to cost savings in avoiding unnecessary work.

Reducing the Cost of Requirements Related Defects
Finally, let's look at the cost savings in terms of fixing requirements-related defects. For this part of the model we'll use a few concepts also used in doing inspection ROI assessments, e.g. Ed Weller's "Calculating the Economics of Inspections" (StickyMinds, Jan 2002).

We'll do this by:

• Building a baseline model of what we believe was the cost of defect detection and removal before we added the new process and tool
• Recalibrate the baseline model with improved defect detection and removal, the cost of which we paid for as Added Review & Rigor above.
• Our cost saving is cost of baseline-1 subtract cost of baseline-2

It's almost an industry cliché that the later a defect is found in the development lifecycle, the higher the cost to fix. The relative cost in the increase varies from industry to industry. Here, I use a simple three-phase defect-removal model:

• removal of requirements defects anywhere before coding (peer review, analysis)
• removal of defects once they are committed to code (unit test, system test)
• removal of defects once they are released to the customer

The relative cost of fixing defects will be 1 staff day, 5 staff days, and 25 staff days; an increase of a factor of 5 from phase to phase, well within the industry averages that are cited in the literature.

Now let's look at the model. We start by estimating the number of requirements-related defects. Unless we are willing to acknowledge the possibility of a perfect requirement, it's safe to assume that all of the defects have at least one defect. If you are not comfortable with that assumption, adjust the percent as needed.

Next, we estimate the number of requirements related defects removed prior to commitment to code, and the associated cost. A removal effectiveness of 50% means that of the total population of defects, 50% were caught and fixed. We'll assume that on average a defect at this stage can be found and fixed in 1 staff day. Changes that don't involve code are simply cheaper to make.

Next, we estimate the number of requirements related defects removed from the code itself (e.g. unit, integration and system test) and the associated cost. The calculation starts with the number of defects that remain undetected and unfixed from the previous stage. Because we are now dealing with code, the cost of finding and fixing a defect rises from 1 staff day per defect to 5.

At this point let's review the total removal effectiveness of the first two stages.

Finally, the cost of defects shipped with the product to the customer. At this stage "finding" the bug is not so much a factor in the cost; the customer does that for you! Here, the cost of defects is determined by factors such as customer support calls, loss of sales from unhappy customers, and, of course the increased cost to patch software in the field. The cost of defects at this point will vary greatly depending on the industry, safety-critical products being an example where the cost can be very high.

Our baseline cost for finding, supporting, and fixing requirements-based defects is $2,608,696. Then we rerun the baseline, but with a 10% increase in defect detection and removal at each of the first two stages. This is the added review and rigor we paid for previously (see above under costs).

Our new cost is $2,024,348; a cost savings of $584,348.

Bottom Line
All that is left is to tally the Benefit to Cost Ratio, and, as they say, "Your mileage may vary."

Estimated savings: $1,884,565
Estimated cost: $427,657
Benefit to Cost Ratio: 4.4 to 1

Recall that cost at this point includes one-time expenses such as initial software purchase, and initial on-site consulting and training. The ratio should get even better once those one-time expenses are out of the way.

Summary
As noted previously, cells that are gray filled are ones where the values entered are highly subjective, and vary depending on your company and industry. I found that a useful approach to using this model in talking with people is to have them provide values for these cells, kind of a Wideband Delphi approach. In that way they can see if the values they are willing to buy into result in a good benefit to cost ratio. After you've talked with a number of people, use their responses to build a minimum and maximum version of the model. If you want to get a bit more sophisticated, you can even run the model with a Monte Carlo simulation tool. This would provide a probability distribution function of the Benefit to Cost Ratio.

And while this model was developed to demonstrate ROI after the fact, the ideal use of such a model is to help in selling the process change in the first place, and to forecast that point at which benefits would begin to outweigh the costs of the effort.

 Want to Learn More?

• David Gelperin, Maybe We Shouldn't Write Requirements, StickyMinds Original Article, October 2002. A brief discussion—and lots of reader comments—on how best to record and manage requirements: paper documents vs. spreadsheets vs. database, etc.
• Jeffery E. Payne, Quality Meets the CEO: How to Get Management Buy-in, STQE magazine, May/June 1999.
• David F. Rico, How to Estimate ROI for Inspections, PSPsm, TSPsm, SW-CMM®, ISO 9000, and CMMIsm, StickyMinds, Sept 2002.
• Ed Weller, Calculating the Economics of Inspections, StickyMinds, Jan 2002.
• Dean Leffingwell, Calculating Your Return on Investment from More Effective Requirements Management.