What Does Success Look Like?

time, it had fewer obvious defects than its competitors, and it made its release date ahead of the competition. After Word 5.1, the other companies still released their products, but Microsoft became what Moore calls the market "gorilla." Now, with the most recent versions of Office, Word is firmly in the late majority. Those of us who use Word don't want to upgrade any more than necessary. When we upgrade, we discover new problems from the ones we're used to, and most of us prefer to stick with our known problems.

When you decide what's important to your project, define where your product is in its lifecycle. Are you selling to Early Adopters or the Late Majority? Is it some combination of markets? Use those customer definitions to determine your project goals. Even when you define what's most important for your customers, your company will still have corporate goals it needs to accomplish with a particular release.

Many project managers (PMs) talk about the tradeoffs between cost, schedule, and quality, but that's too simplistic for what PMs actually do. In reality, PMs trade off among the things the customers value (time to market, the feature set, and the defect levels) and the things the company values (cost to market, the number of people working on the project, and the working environment of the project). Release criteria help make these tradeoffs obvious.

As testers, and especially as test leads or test managers, we're more frequently aware of potential risks than other people-that's our job. You can't change the corporate needs that drive the release. After all, if you need revenue this quarter, you still need revenue. However, you can help illuminate the tradeoff decisions by discussing potential risks with the project manager. You can use the tradeoffs between the six project attributes (time to market, feature set, defect levels, cost to market, people, working environment) to best employ the test people and best use the test time to the project's advantage.

How do you ferret out this information? I interview the various project stakeholders: the project sponsor or senior management, marketing, support, training, whoever has a stake in the project's "doneness."

Use Context-Free Questions
I use context-free questions [Gause and Weinberg] as my primary interviewing technique for discovering what success means. These are some context-free questions:

1. hat does success look like?
2. Why are these results desirable?
3. What is the solution worth to you?
4. What problems does this system solve?
5. What problems could this system create?

These questions all get at why we're doing the project, without actually asking why. Sometimes, when we use the word "why," we sound as if we're interrogating others, or asking them to defend their positions. Interrogation or putting people on the defensive is rarely a good way to gather information about the project, and retain a good working relationship with other people.

You'll notice that the questions also don't ask "how." Sometimes, when we ask how, we start designing things. You don't want to design the project or the product; you want information about the product, to know how to evaluate the product's state of completion throughout the project.

Once you've collected information about the product and the project, you can start to draft release criteria. As a nice side benefit, you'll learn where to put your scarce testing resources and time.

References
 

1. Moore, Geoffrey. Crossing the Chasm . Harperbusiness, New York: 1995.
2. Grady, Robert. Practical Software Metrics for Project Management and
   Process Improvement . Prentice Hall, New York: 1992.
3. If you are interested in this article you would also benefit from the article "Release