Most project managers want to reduce risk during a project. One way to reduce overall risk is to define and focus the project goals up front, and continually verify those goals and progress toward those goals during the project.

Bob Grady [1], claims there are three common goals for software product development: making a specific time to market, generating all the features, and providing a product of very low defects. In my experience, each project has one and only one of these goals as its top priority, and then trades off the other two goals within that context. This prioritization will determine the tradeoffs the project manager makes during the project.

The traditional definition of risk as "a potential problem that should it occur, will endanger or eliminate project success"[2] is different than the issue of changing project focus during the project. In the real world, project goals get set, and after some time, there is impetus to change them. This article addresses the issues of choosing a project focus, and recognizing when the focus needs to change.

Product Quality Model
To define and assess project completion risks, a project manager should first define the true measure of product quality. If we agree with Jerry Weinberg [4] that product quality is value to the customer, then we should be able to choose product goals that provide value to the customer. And, if we agree with Grady [1] that there are three common features of quality, then we can choose one of the three priorities below as the primary measure of product quality:

• Minimize time to market, by decreasing the calendar schedule
• Maximize customer satisfaction, by giving the customers the features they want
• Minimize number of known and shipped defects

The key to defining product quality is to choose which priority is most valuable to your customers. For any given product, one of these priorities is the foremost priority; the other two are traded off within that context.

Some people are not convinced that they only have one top priority. Here is a check to verify that choice: If it is three weeks before the scheduled ship date, and the defect count is higher than the developers prefer, and the last feature is not quite implemented, what does management choose to do? If management chooses to ship anyway, then time to market is the top priority. If management chooses to ship as soon as that feature is installed, then customer satisfaction is the top priority. If management chooses to hold shipment in order to fix the defects, then low defects is the primary definition of quality. As much as your customers force you into a particular quality definition, where the product is in its lifetime also has an effect on quality definition.

Maybe you're still not convinced you only have one top. Consider this example: you have a ranking of 0-100, where 0 is low priority and 100 is high priority. You rank the ship date at 100, defects at 97, and features at 93. At first glance, the rankings are so close as to be inseparable. Then there are other questions to ask: o all defects need to be fixed, or are there critical defects to be fixed before the ship date? Do all the proposed features need to be added, or are there critical ones that need to be fixed before the ship date? Or, is the ship date contingent on those particular features? When prioritization is unclear, it's time to ask more questions–when the overall project priorities shift, the product itself shifts, and product shipment is at risk.

Each product has a lifetime: it's created, revised, and retired. As much as your customers force you into a particular quality definition, where the product is in its lifetime also has an effect on quality definition. The key to project success (whether the product is actually sold commercially or used internally) is to recognize where in its lifetime the product is, and to use the natural drivers to help determine product quality.

Table 1 provides a view of the interaction between the definition of quality and the product lifetime. This is an application of Geoffrey Moore’s [3] high technology marketing model to software quality. The users (and potential users) have different perceptions of product quality over the lifetime of a product. If project management recognizes where the product is in its lifetime, then quality criteria can be determined and assessed during the project to manage risk.

For too long, product development groups (software development, software test engineering, project management) have focused only on low defects as the primary measure of product quality. Today, however, it is no longer adequate to define quality based on defect levels. Ship quality is truly the definition of what is "good enough" [5] for the customer to use.

Table 1: Quality Perceptions over Product Lifetime
Product Life/ Market Pressure	Introduction (Enthusiasts)	Early Adopters (Visionaries)	Mainstream (Pragmatists)	Late Majority (Conservatives)	Skeptics (Laggards)
Early Ship	High	High	Medium	Low	Low
Features	Low	Medium	Low	Medium*	Medium
Low Defects	Medium	Low	High	High	High
* Medium pressure for features does not mean pressure for new features. It means that the promised features must be in the product and must be working.

Introduction
Enthusiasts or technology nuts buy initial products. These people like the thrill of playing with new technology. They want to get their hands on it and play with it (High in the Early Ship dimension). As long as the product does somethin very well, they will be happy (Low in the Features dimension). As long as the defects do not get in the way of using the product in a narrow sense, the enthusiast is happy (Medium in the Low Defects dimension).

Early Adopter
Visionaries have a specific problem and their pain can be remedied by this product. They need it right away (High in the Early Ship) and since they plan to solve their problem, they have a number of demands for features (Medium on the Features dimension). The overall defect count is not particularly an issue, as long as the features they need the most work correctly (Low in the Low Defects Dimension). Visionaries willingly accept workarounds for defects or incomplete functionality.

Mainstream
Mainstream users are pragmatists. If they are going to buy a product, it better solve their entire problem in a robust manner (High on the Low Defects dimension). That is, the existing defects must have a low impact on their day-to-day use of the product. If you announce a new release, they will wait for you to ship it (Medium on the Early Ship dimension), but not too long. Since these people are pragmatic, they are not looking for any features above and beyond what you have implemented and advertised (Low in the Features Dimension).

Late Majority
The Conservative users wait for products to live up to their advertised reputations. They look for low defect levels, and low impact of existing defects, to improve their success in using the product (High on the Low Defect dimension). They want the features working as you have promised, but additional features are not high on their list. They don't particularly care when you ship the next release–they may not even upgrade to it (Low on the Early Ship dimension).

Skeptics
The Laggards are not interested in software that has defects (High on the Low Defect dimension). In fact, the only reason they might buy your software is to get a particular feature they cannot get anywhere else (Medium on the Features dimension). Since Laggards are so concerned with defects, they are willing to wait for product shipment (Low on the Early Ship dimension). These people want software producers to guarantee there will significantly fewer defects than in previous software releases.

Determine Project Focus and Ship Goals
With the product quality model, it is easier to define and prioritize project ship goals. It is easier because management can now just focus on the current project goals, not all the product goals over the product's entire lifetime. Project shipment goals take the form of ship criteria.

The project goals are the same as the project requirements for those organizations that write requirements documents before the product ships. In my experience, many commercial software companies do not specify product requirements, or at least not in sufficient detail to use as ship criteria.

It is crucial to set limits on this project's ship goals: goals for this release, not requirements for the final, to-be-retired, product. One way a project manager can focus the ship goals is to work with the product manager to help define birth-to-death product requirements, and determine this project’s focus based on that thinking.

Middleware Product Example
The following is an example of choosing appropriate beta and ship criteria based on product quality. In this example, a middleware communications product was in the Early Adopter phase of its lifetime: there were a small number of customers with money to pay for product development, because the customers had a huge need that this product promised to fulfill. The product had to be delivered when it was promised, and it had to have the features for the customer to prove to his or her senior management that this product was the right tool for the job. There was an overriding ship criterion—the product had to go to beta in April.

Beta Criteria
The following list is an example of beta Shipment criteria for this middleware product:

1. All code must compile and build for all platforms.
2. All developer tests must pass.
3. All available tests for beta customer (client side part of the product) must run and pass.
4. All current bugs are entered into the bug-tracking system.
5. First draft documentation is available and shippable to customers.
6. The code is frozen.
7. Technical support training plan is in place, and the people are in place.
8. There are less than 36 open high priority bugs.

The beta criteria are based around getting to a specific ship date, not necessarily reducing bugs or adding features. This particular organization started beta as early as possible to maximize their ability to work with their customers on product features. Up until this release, this organization had not formally counted or tracked defects. At the start of this project, Management estimated there were no more than 80 defects in the product. The project manager was sure there were significantly more, but she was willing to draw a line and choose a number out of thin air— 36 defects (less than half of the current management estimate).

Note that even when discussing the same product there is a contrast between the beta criteria above and the following Product Shipment criteria below. The overriding criterion was to ship the product by July 15.

Shipment Criteria

1. All code must compile and build for all platforms.
2. Zero high priority bugs
3. For all open bugs, documentation in release notes with workarounds.
4. All planned SQA tests run, minimum 90 percent pass.
5. Number of open bugs decreasing for last three weeks.
6. All beta site reports obtained and evaluation documented.
7. Reliability criterion: Simulate one week of usage by sending a minimum of 200 messages of varying sizes to and from varying platforms with varying classes of service.
8. Final draft documentation available, complete and submitted to corporate organization.
9. A working demo runs on previous release.
10. Verify that tokens reduce on-air time by 25 percent from previous release.
11. At least two referenceable beta sites (i.e. customers who were sufficiently happy with the product that they would agree to be contacted by potential customers).

The middleware communications product shipment criteria focus on getting to the ship date with reasonable software, but not particularly low defect software. The features were concentrated in the performance and reliability areas in order to make the customers successful when deploying applications.

Machine Vision Product Example
Contrast these product criteria with a machine vision product in its mainstream phase:

Shipment Criteria

1. All system tests executed (>90 percent passed).
2. Successful execution of any "Getting Started" sequence.
3. Results of executed tests must be discussed with product management team.
4. Successful generation of executable images for all appropriate platforms.
5. Code is completely frozen.
6. Documentation review is complete.
7. There are zero showstopper bugs.
8. There are fewer than four major bugs, and 15 minor bugs.

These defect-focused criteria (especially numbers 7 and 8) are focused on defects as a primary definition of quality. These numbers were chosen based on previous product releases, and the assumption that the severity of each the existing defects would be assessed. The ship date is still important, otherwise the defect levels would be lower and system test pass results would be higher.

Using Criteria to Manage Product Shipment Risk
Risk-aware project managers take these product ship criteria, and measure progress towards completion of the criteria during the project. Having focused product ship criteria makes it easier to make project tradeoffs as the issues arise.

As an example, consider the Apple Newton and the US Robotics Palm Pilot. The Newton clearly made an early ship date, but the handwriting recognition was insufficient for the product’s intended use. The Palm Pilot finessed the idea of handwriting recognition (the user writes a small, clearly defined set of symbols), and the product, from this author’s perspective, and is well on its way to product mainstream success. The Newton never made it into the mainstream, because of the defects in its handwriting recognition at initial product shipment.

Summary
Products evolve during their lifetime. The most effective product releases are those that further the company's goals while meeting the customer needs. Defining product shipment criteria for a given project makes the company clarify current product and specific project goals. Sometimes this causes a company to agree on long-range product goals. This in turn makes it easier for the project team to buy into current project goals, because they can see where the product is headed.

Once everyone has agreed on what the company needs from the product and what the customers need from the product, the quality criteria are clear. Project managers must choose the correct quality and ship criteria for their projects, define those criteria in terms of measureable goals, and then measure progress towards those goals throughout the project.

It is necessary to decide at the beginning of the project what quality is for the product, at the product’s current stage in it's lifetime. A project manager must use what is of value to customers to set shipment requirements including features and functionality, promised shipment dates, and product defect levels. Once the project manager has focused project goals, it is possible to manage project progress to those goals on an ongoing basis.

Acknowledgments
I thank the following reviewers for their substantive and welcome suggestions to this paper: James Bach, Robert King, Brian Lawrence, Tim Lister, Barbara Purchia, and Jerry Weinberg.

References

1. Grady, Robert. Practical Software Metrics for Project Management and Process Improvemen. Englewood Cliffs, NJ: Prentice Hall, 1992.
2. Lister, Tim and Tom DeMarco, Leading Successful Projects seminar, 1997.
3. Moore, Geoffrey. Inside the Tornado, New York: Harper Collins, 1995.
4. Weinberg, Gerald M. Quality Software Management, vol. 1. , New York: Dorset House Publishing, 1992.
5. Yourdon, Ed. "Good Enough" Software, Guerrilla Programme (April 1995).