Key Risks to System Quality

half second, completely displaying each page within 30 seconds, and allowing the typical user to complete an e-commerce sales within two minutes. As with the informal quality risks analysis process, once the quality subcharacteristics are identified, stakeholders should determine the priority levels associated with testing each area.

On very structured development projects, you can formalize the quality risks analysis process by using Failure Mode and Effect Analysis (FMEA). Figure 1 shows an example of a FMEA chart for a hypothetical word processor. Ideally, all the stakeholders perform the FMEA in a group meeting. Once again, you should prioritize the myriad quality risks to identify the most important areas for testing. In a FMEA chart, the Risk Priority Number is the product of three factors, Severity, Priority, and Likelihood, as describe above.

Any of these techniques can become time consuming, confusing, and cumbersome if you try to go into too much detail in the failure modes or quality subcharacteristics. Focus on relatively high-level descriptions. "Misspelled user messages," is good. Trying to list every possible misspelling that could occur in each error message would be bad. The objective is not to try to document bugs you haven't even found yet, but rather to focus test development and execution activities.

Not every quality risk can be a high priority. When discussing risks to system quality, I don't ask people, "Do you want us to make sure this area works?" In the absence of trade-offs, everyone wants better quality. Setting the standard for quality higher requires more money spent on testing, pushes out the release date, and can distract from more important priorities—like focusing the team on the next release. To determine the real priority of a potential problem, ask people, "How much money, time, and attention would you be willing to give to problems in this area? Would you pay for an extra tester to look for bugs in this area, and would you delay shipping the product if that tester succeed in finding bugs?" While achieving better quality generates a positive return on investment in the long run, as with the stock market, you get a better return on investment where the risk is higher. Happily, unlike the stock market, the risk of your test effort failing does not increase when you take on the most important risks to system quality, but rather your chances of test success increase.

Whatever risk analysis process you chose, at the end you should have a ranking of risks (or quality subcharacteristics) by priority. These risk priorities will help you winnow down the test effort from everything you conceivably could test to some manageable, agreed-upon list of what you should test . You should test the high priority risks items extensively, the medium priority risk items broadly, and the low priority risks items in a cursory fashion. The items that pose little if any risk to the quality of the system should receive none of your scare time, resources, and attention at all. For more information on these quality risk analysis techniques, see my first book, Managing the Testing Process , my new book, Critical Testing Processes Volume I, or D.H. Stamatis’ book, Failure Mode and Effect Analysis .

Building on Your Foundation
A well-defined, properly prioritized list of quality risks is the foundation of your test effort. Understanding the importance of each quality risks to your customers allows you to build a high-fidelity test system, predictive of the customer's experience of quality. In construction, sound building techniques are required to construct a trus tworthy house on a good foundation. Likewise, you now need to apply