Measuring Process Improvement

There has been much talk in the software industry regarding software process improvement. The emphasis has often been on passing various "tests," such as the ISO certification and SEI CMM Levels. If you are taking one of these routes, you may be wondering if you are actually improving anything-or just learning to jump through new hoops for the purpose of an audit. Process improvement doesn't have to be academic, or solely focused on documentation. It can, and should, be used to solve real problems and make real gains.

This article discusses how to track progress in your process improvement program, and offers four questions to consider as you measure your progress: Are we achieving our project goals, solving our development problems, and making progress on our improvement action plans? What are our savings in time and money? Are we making headway on our chosen process model or standard? What lessons have we learned? Answering these questions lets you know how well your improvement program is going, provides visibility early for detecting problems, and gives you data to make your future plans more effective.

We'll also look at some examples of how companies track improvement; you can tailor these examples to fit your needs, or use them as a starting point to generate your own measurement ideas. Our intent is to stimulate you to think about the types of measures that would be useful in your process improvement environment.

Almost all process improvement programs-ISO, CMM, or a homegrown hybrid-are built from similar components. For the purposes of this discussion, we'll use the Shewhart four-step improvement cycle for planning, executing, and managing improvement programs. It divides the process into four phases: PLAN (planning the improvement effort), DO (executing the plan), CHECK (measuring improvements), and ACT (acting on the data from the CHECK phase). The DO and ACT phases are subjects for other articles; here our discussion will focus on the PLAN and CHECK components.

Plan: Defining Goals
To measure your progress in anything-a marathon, a weight-loss regimen, or a software process improvement program-you have to have a plan. That plan provides clear standards and goals against which to measure your advancement. In software processes, effective improvement planning is based on the business goals and problems of the organization. Example goals might include the delivery of a product, the completion of a software installation, or the upgrade of a database.

Examining both the goals and problems of an organization can provide an effective scope for any improvement program. This goal-problem approach starts with business goals and focuses on eliminating the problems that keep you from reaching those goals. The resulting goal-problem improvement program is compelling and practical. Here are some examples from one company:

1. Goal: Reduce product development cycle to six to nine months.
   Problem 1.1: We don't manage changing requirements.
   Problem 1.2: Difficult to find defects before test.
2.  Goal: Successfully deliver product X.
   Problem 2.1: Wrong files (e.g., DLLs) are put on CD-unsure of the correct ones.
   Problem 2.2: Defect repairs break essential product features.
3. Goal: The core software product performance is increased by 20 percent. (No identified problems)
4. Goal: Developers have the essential development tools (latest compiler, memory checker, debugger, and performance analyzer).
   (No identified problems)
5. Goal: Customer rating is nine out of ten on product evaluation form.
   Problem 5.1: Customers are unhappy. There are approximately 300 outstanding defects that have not been addressed.
6. Goal: Profit remains 15 percent (costs remain the same as last year).
   (No identified problems)

Measurable improvement plans should clearly describe the goals, the reasons behind those goals, and the actions required to achieve them. The