“But it was just a tiny little change! How could we have known it would cause such big problems?” Regression (going backward) is a fact of life in software systems. Even though something worked before, there is no guarantee that it will work after the latest "minor" change. Yes, modular design and sound system architecture can limit the likelihood of unintended effects, but they won't eliminate them all together.
Regression testing will always be necessary. With the very limited amount of time we have for testing a minor change, though, how can we do sufficient regression testing? How can we know where to look? How can we reduce the risk that there will be problems?
The Regression Problem
The regression problem has its roots in the inherent complexity of software systems. As the complexity of a system increases, so does the likelihood that changes to it will have impacts that are difficult to foresee. This it true even with newly crafted systems whose developers used the latest techniques and are still available to make any necessary changes.
The regression problem grows exponentially as a system ages. After several years, it will change many times, usually by people who were not involved in the original development. Even if these people did their best to understand the underlying design and structure, it is unlikely that the changes they made fit cleanly with the original design theme. The more changes like that this that have been made, the more complex the system grows until it becomes brittle.
Brittle software is just like brittle metal: it has been bent and twisted so many times that anything you do to it is likely to cause it to break. When a software system becomes brittle, people actually fear changing it. They know that anything they do is going to cause more problems than it the changes will fix. Though the term is rarely used, being brittle (un-maintainable) is one of the main reasons that old software systems are replaced. The Regression Testing Squeeze
Because any system is subject to regressions, regression testing of any and all changes is important. Who has time to fully re-test a system to which a small change was made, though? When the development only took a week or so, we certainly cannot afford to spend months fully re-testing the entire system. We would be lucky to have a week for testing. More likely, only a few of days will be allowed.
Since full re-testing is out of the question, we must then decide how to spend the little time that we do have available for testing. How can we know where to look, though? How can we anticipate these unanticipated problems? It's kind of like the joke where the boss demands a list of all of the unexpected problems that will come up!
In reality, we always have a testing squeeze, even when testing a brand new system. There is never enough time to do all of the testing that should be done, so we must spend the time we have available for testing in the best way possible. The method we use in those situations, "risk-Based testing" is the same method we must apply in regression testing.
The essence of risk-based testing is that we assess the risks involved in the different parts of the system, and focus our testing where the risks are highest. This method may allow some parts of the system to remain only marginally (or possibly totally) un-tested, but it guarantees that the risks involved in doing so are low.
· "Risk," as it applies to testing, is the same as risk in any other situation. In order to evaluate risk, we must recognize that it has two distinct dimensions: Likelihood and Impact.
· "Likelihood" is the probability that something will go wrong. Without considering Impact, simply consider how likely it is that there will be a problem.
· "Impact" is the effect if something does go wrong.