Get Smart about Your Regression Tests’ Value

[article]
Summary:

If you aren’t measuring the coverage your regression tests provide, you may be spending too much time for little benefit. Consider the value of your regression tests as you create and manage them. You need to be smart about the regression tests you maintain in order to gain the maximum value from the work put into creating, running, and analyzing their results.

Do you know how many regression tests you have? Do you know what the coverage of your regression tests is? Do you know the value of your regression tests?

These may all be pretty simple questions, but your team needs to be asking them; knowing the answers is very important to your test plan. You can't manage what you don't measure!

As I conduct more testing and agile reviews with organizations, irrelevant regression tests seem to be an increasingly common problem. In many instances, I see a good portion of tests that are either redundant, repetitive, or not exercising any different code—i.e., not expanding code coverage. This is becoming even worse with the increase in automation. Due to the perception that automation tests require almost no time to run because they are running out of hours or in parallel with the operator doing other tasks, people think it’s not as necessary to manage the volume of regression tests, and the associated tasks disappear.

You have to be smart when designing your regressions tests. That’s why I use that acronym to remember what’s important to consider when creating and managing them:

  • Specific: Tests have to be clear about what requirement is being tested
  • Measurable: Tests must define the coverage of the requirement in order to measure the actual value
  • Achievable: Every regression test has to have a standard value defined for it. This could be measured in the time required to create it, run it, or maintain it, or as when it last found a defect
  • Relevant: The regression test must give more insight in the continuing function of the software, proving its fitness for purpose as well as just working correctly.
  • Time: It is important to express the value of the regression test in time. Every test only has a meaning if one knows the time dimension in which it is realised. This can both be from the aspect of the time it actually takes to run the test, and the time lapse as to the value that the regression test still adds value.

Let’s dig deeper into what you should consider when creating, managing, and evaluating regression tests.

Know the Value of Your Regression Tests

The first thing we need to consider for each individual regression test is its return on investment. If the effort to create, run, and maintain a test is not either giving us confidence in the system under test or finding regression defects, then we have to question why we are spending this time.

I want to highlight in particular the “run” part of the equation. I have heard people say so many times that they think there is no cost to run regression tests after they’re automated. While a good automation framework may allow regression runs to be kicked off by the click of a button and run unaided, there is still a requirement to analyze the results, and that does incur a cost—even if those results are summarized in a dashboard.

This also takes into consideration that the scripts have been created within the framework to deal with unexpected events, such as pop-up messages, which could potentially stop a regression run. If they require a human to analyze the results, they are not fully automated. Maybe it is worth questioning if these tests are truly automated.

User Comments

9 comments
Robby Comstock's picture

Thank you for the great article Leanne.  We are currently working on a project to go through our regression tests and finding where the gaps are and what we can elminate.  

July 20, 2016 - 10:31am
Leanne Howard's picture

Hi Robby, let me know how you go.  I would be interested particularly in how many tests you retire/archive when you do the reveiw.

July 25, 2016 - 7:54pm
Tim Thompson's picture

Good advice, but I already stumble at the "S": "Tests have to be clear about what requirement is being tested" While I agree that this is a necessity the destructive rampage of Agile on QA made it so that QA no longer gets and firm or often even lose requirements. I se more and more a total disconnect between what the business needs (but does not tell Dev/QA), what developers implement (which is often what developers want because it is cool...but dysfunctional), and what QA tests (without any guidance of any kind we are forced to make stuff up to test for).

In a more waterfallish approach there was more time spent upfront to determine and document requirements. While I no terms advocate to return to a strict waterfall approach, the Agile teams need to move away from the anarchy and more towards putting pressure on the business to detail exactly what the needs are. Otherwise we end up in endless debates and get stuck in refactor and redesign cycles that waste a lot of resources. Quoting Mike Holmes "If you do it, do it right the first time!"

July 25, 2016 - 7:58am
Leanne Howard's picture

Hi Tim,  This is something that I see all too often with agile teams.  If the user stories and acceptance criteria are not well written, then it makes everything difficult for the team.  Try putting something like "All user stories must pass INVEST checks" into your DoR.

July 25, 2016 - 7:56pm
Mike Coughlin's picture

Tim, $.02.

I was recently part of a project team that ran a Proof of Concept on Agile Testing (migrating from traditional Waterfall). While not without its bumps and bruises, the entire process hinges on communication: clearly defined, well-estimated user stories with enough understanding to execute (dev / test) and a clear definition of done. If you have these aspects, there should be far fewer questions for what Testers should test. Everything should mature in the sprint ("final" design documents, solutions, test scripts, etc) but it should never be chaos and it all starts in Backlog Grooming with the information gathering and the estimation (Three Amigos involved). Sprint 0 and Backlog grooming, IMO, is so critical to the ultimate success of Agile (and carving time out to continue it DURING SPRINTS).

Agile is not an excuse to embrace anarchy. It is a clearly defined set of principles that govern a software development methodology. When we start a sprint, our testers execute to the definition of done and, because of the daily stand-up meetings and other related ad-hoc meetings, understand the intended functionality / flows that should be implemented (because the BSA is not a Programmer and things can morph in implementation). I would leverage your Retrospective as an opportuntiy to raise your concerns and then take a hard look at your Backlog grooming process. If Testing isn't involved early and communication isn't flowing then you're never going to get away from the anarchy.

July 26, 2016 - 2:44pm
Diane Coleman's picture

Great article. I am about to embark on a project that will automate our Oracle regression test scripts. This system will become decommissioned in about 2-3 yrs so I don't want to waste to much time creating a bunch of regression scripts that will be obsolete in a year. We are basically automating them in order to go into maintenance mode with only business critical changes, if necessary. I like using the SMART option for planning out the direction we should take but are there any other suggestions you would make in this situation? 

July 26, 2016 - 3:12pm
Leanne Howard's picture

Diane, I think that the key for me would be to make sure that you calculate the ROI for each test that you automate taking into consideration the short timeframe.  Obviously the value of the functions and product risks, as I mention in the artilce, are key factors.

Happy to discuss in more detail offline.  You can contact me via Linked-in.

 

 

July 26, 2016 - 8:09pm
Leanne Howard's picture

Some great suggestions Mike, all of which I would agree with.  I would have testers in teh Grooming sessions as well.  I use the Three Amigos a lot and it is a sign of a good team, if you see people huddles together throughout the day.

July 26, 2016 - 8:05pm
Gallop Solutions's picture

Nice Post. Considering each regression test as a return on its investment is awesome.

October 27, 2016 - 3:27am

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.