A Strategy for Risk-Based Testing


The fact that you test an application extensively does not itself render the application more stable. However, NOT testing an application increases the risk that the software may not comply with the requirements and won't necessarily provide the expected business value. Here is a method for approaching risk-based testing.

In the ten years that I have worked in the software industry, I have come to realize that there is a widespread, but essentially wrong, belief amongst most software companies that software quality is a function of testing. The fact that you test an application extensively does not render it more stable, nor does it provide added value to the customer. How many times have you heard, "This software is not stable enough, we should have had more QA."? Unfortunately, testing has nothing to do with stability. You test to ensure that the software functions as it was initially requested by those who dictate the requirements. Therefore, not testing software increases the risk that the software may not comply with the requirements and won't necessarily provide the expected added business value.

This leads us to risk-based testing, which I would describe as Steve Wakeland defines IT risk, as 'the likelihood that a program fault will result in an impact on the business'. Further, I would specify a NEGATIVE impact on the business. Let me explain.

I classify bugs with severity ranking, as do most people, and define severity by measuring the negative impact a bug has on the business. The severity ranking is usually high, medium or low. High meaning the customer cannot work with the software and wastes time and money, medium meaning the user has to use some alternative workaround to achieve the aimed goal and low meaning the customer can STILL work with the software (feel free to specify additional levels for a more granular ranking). Thus, if I classify bugs found by QA in terms of severity by measuring (or guessing) the negative impact it has on the business, then I should focus my testing using the same mindset, i.e. negative impact.

Thus, testing and risk (as defined by Steve Wakeland) are related. How? Is it a straight line as shown in Figure 1? In that case, the only strategy is: Test anything; it's sure to lower the risk.

Strategy for Risk-based Testing1

Figure 1

Ideal Relationship
Pareto was probably right when guessing that 20% of the total feature set allows a user to do 80% of his/her work. What if the relationship between the testing effort and risk was more like Figure 2?

Strategy for Risk-based Testing2

Figure 2

We can define a better testing strategy that optimizes the QA effort to minimize risk. It is important to note that no matter how much testing you invest, there is always a risk. Therefore, releasing the software is directly related to the level of risk you are willing to accept.

However, if you see testing as an investment, the strategy becomes: What is the minimum testing effort that I should invest in order to maximize risk reduction? In other words, what should I test first to be in Zone A of Figure 2?

I control my testing effort with a simple methodology based on usage. Although users have their own way of using software, you can usually predict the top-ten features list that allows them to do most of their work.

Step 1: Identify the 'vital' functionalities that could prevent the user from using the software if a bug was found. (This bug would be a high severity.) A good example would be a login page for a Web application that does not work. An efficient way to gather this list of functionalities is to survey the end-user community, ask a domain expert, or assemble statistics from the log of a previous version of the application. Since the risk increases with the frequency of use, you should look at the most used feature

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.