If you aren’t measuring the coverage your regression tests provide, you may be spending too much time for little benefit. Consider the value of your regression tests as you create and manage them. You need to be smart about the regression tests you maintain in order to gain the maximum value from the work put into creating, running, and analyzing their results.
Do you know how many regression tests you have? Do you know what the coverage of your regression tests is? Do you know the value of your regression tests?
These may all be pretty simple questions, but your team needs to be asking them; knowing the answers is very important to your test plan. You can't manage what you don't measure!
As I conduct more testing and agile reviews with organizations, irrelevant regression tests seem to be an increasingly common problem. In many instances, I see a good portion of tests that are either redundant, repetitive, or not exercising any different code—i.e., not expanding code coverage. This is becoming even worse with the increase in automation. Due to the perception that automation tests require almost no time to run because they are running out of hours or in parallel with the operator doing other tasks, people think it’s not as necessary to manage the volume of regression tests, and the associated tasks disappear.
You have to be smart when designing your regressions tests. That’s why I use that acronym to remember what’s important to consider when creating and managing them:
- Specific: Tests have to be clear about what requirement is being tested
- Measurable: Tests must define the coverage of the requirement in order to measure the actual value
- Achievable: Every regression test has to have a standard value defined for it. This could be measured in the time required to create it, run it, or maintain it, or as when it last found a defect
- Relevant: The regression test must give more insight in the continuing function of the software, proving its fitness for purpose as well as just working correctly.
- Time: It is important to express the value of the regression test in time. Every test only has a meaning if one knows the time dimension in which it is realised. This can both be from the aspect of the time it actually takes to run the test, and the time lapse as to the value that the regression test still adds value.
Let’s dig deeper into what you should consider when creating, managing, and evaluating regression tests.
Know the Value of Your Regression Tests
The first thing we need to consider for each individual regression test is its return on investment. If the effort to create, run, and maintain a test is not either giving us confidence in the system under test or finding regression defects, then we have to question why we are spending this time.
I want to highlight in particular the “run” part of the equation. I have heard people say so many times that they think there is no cost to run regression tests after they’re automated. While a good automation framework may allow regression runs to be kicked off by the click of a button and run unaided, there is still a requirement to analyze the results, and that does incur a cost—even if those results are summarized in a dashboard.
This also takes into consideration that the scripts have been created within the framework to deal with unexpected events, such as pop-up messages, which could potentially stop a regression run. If they require a human to analyze the results, they are not fully automated. Maybe it is worth questioning if these tests are truly automated.