In recent years, many organizations have been impacted by DevOps.

Some have implemented DevOps end to end by changing people’s mindsets, automating deployment and build processes by implementing appropriate tools and processes, increasing test automation, breaking up silos between development and operations, and automating monitoring and reporting. Most organizations, though, have just started on their DevOps journey or are somewhere in the middle of the transformation process.

Part of the path to DevOps requires adoption of agile methodologies. What does it mean for testing when you switch from the traditional waterfall model, with a few long release cycles per year, to the agile model, with changes occurring every two weeks, or possibly even more quickly?

In most organizations, the world of legacy systems collides with the world of modern applications. However, the two need to coexist. In the past, we were executing regression tests for legacy systems over several weeks—manually as well as automated—but we don’t have the same amount of time for testing anymore.

Switching from a traditional waterfall model to an agile one requires new testing approaches.

Optimizing test automation is an excellent way to bridge this gap. With automated tests in place, it is possible to allocate scarce testing resources to high-value activities, reduce time spent on test execution, and increase the number of test cycles possible in a shorter amount of time. The impact of these changes can be realized immediately with reduced efforts, cost savings, and dramatically improved time-to-market readiness.

My organization has implemented a lot of automation in the last few years—and made many mistakes in the process! But we also learned a lot and are now able to adapt better to a world that’s becoming more and more agile. Automation only works if you have robust tests that can run unattended from the beginning to end.

Here are five key factors to achieve agile software testing in DevOps.

1. Test data management

Having the right test data is the first important step in automation. If your test data is not stable, you will never succeed. According to software industry statistics, nearly 30 percent of test execution failures are due to improper test data.

For each of your test cases, define what data is needed in order to execute it. Keep it flexible, describing the attributes your test data needs to fulfill rather than the test data itself. For example, “John Doe from UK” is the test data itself, and the correct definition would be “Male natural person with nationality UK.”

Once you have defined the required attributes, where is the best place to find test data? To be completely flexbile, and with the new GDPR legislation especially, it makes sense to build up required test data synthetically. This is a good option if you don’t need historical data (even though that is also possible to build) and if your data setup is not too complicated or distributed across too many different systems.

If you require more complex data, you might use existing customer data that has been anonymized. In this case, queries are needed to find data in the databases matching the exact combinations of attributes.

Usually, a combination of both approaches works best. Creating synthetic test data and running queries on a database to attain highly suitable test data can and should be automated.

Proper test data management is not only a must for automation; it’s required for manual testing as well. According to some surveys and statistics, manual testers spend 50 percent to 75 percent of their effort on finding and preparing appropriate test data. Test data must be independent from people maintaining it for you to get a return on investment.

2. Flexible steering

When you design your test cases, use steering parameters instead of copying the same test case several times to fulfill certain criteria.

Let’s assume you need to run your test case first in a development environment and later on in a production-like environment. You would need to log in with role X and role Y. It’s inefficient to copy the test case four times for each combination of environment and role when you can enter these parameters only once in the test case as a steering element. This reduces the overall maintenance of this test case by 75 percent.

Whenever possible, it’s advisable to work with reusable test step blocks. Frequently and repetitively used test steps can be defined once and reused in different test cases. For example, with a login for an application or generation of a new client, which are used over and over again for subsequent test cases, there will no longer be the need for repetition, which saves valuable time and effort.

A failing test case often results in an end to the automated execution; therefore, the next test case cannot run, because the application was left in an “undefined” state. Imagine starting the automated execution in the evening to run the test cases overnight. The next morning you would find your test cases were not executed because the first test case that had failed and prevented all the others from running.

This can be avoided by defining recovery scenarios for each test case or making them completely independent of each other. Instruct the test case what to do in case of failure.

3. Test environment and service virtualization

Next to missing test data, unstable, unavailable, or incomplete test environments are one of the biggest time-consumers in testing.

Consider this scenario: Your tests are planned, testers organized and blocked their time on a certain day, and, finally, when you want to start testing the application, it is not available. Or the application itself is available, but another dependent application or dependent service delivering the data you needed is not.

The more people and applications involved, the more complex the test environment becomes, and the risk of unavailability increases.

However, in our modern agile world, the days of waiting to test everything until all components are developed and all systems and services are completed are gone. Instead, you need to simulate those components or systems that are not yet connected or developed, so testing can be started earlier. This is possible with service virtualization. You need to know the parameters a service expects and the data it returns.

A good service virtualization testing tool can react to changing circumstances and switch from virtualized services to the real service if it becomes available, then back to virtualized services if the service is not available for whatever reason.

Bear in mind you can’t do all your testing based on virtualized services; at a certain point, you’ll have to start real, end-to-end testing. But many critical defects will be found before your end-to-end tests if you start testing much earlier. This saves a significant amount of rework and debugging costs later.

Additionally, a well-managed test environment helps to improve stability. This can be achieved with organizational measures (e.g., a centralized test environment management team that coordinates and controls the test environments) or defined test environment management processes.

Many organizations do not want too many processes put in place because they believe agile development doesn’t need processes. However, certain processes—or, rather, rules of engagement—are necessary to improve test environment stability.

These processes don’t need to be very time-consuming, at least for more advanced test stages, such as end-to-end tests or business verifications:

• If several applications and services from different teams are involved, define time slots when deployments should happen—testers will then know when everyone is done and they can start testing
• Actions like restarts of applications and services, database refreshes, or other outages should be planned during time slots where no testers will test
• If testers communicate when they plan to test—either manual or automated test cases—such as by calendar, a test environment management team can better coordinate and communicate
• The more detailed new changes are communicated to the testers, the better they can adapt their test cases

If scripted properly, DevOps automation can also help manage environments.

4. Test users

When automating test cases, it’s best to never include your personal credentials in the automation. If you’re out sick, your colleague won’t be able to execute the test cases you’ve designed. Also, using another person’s personal user account is typically a security breach in most companies.

The correct approach is to set up special test users that cover the different roles you require for testing your applications. Test users will persist if someone leaves the company or changes their role. Just make sure you take into account roadblocks like single sign-on, remote desktop connections, firewalls, cross-border access, or other barriers.

5. Continuous testing and continuous integration

Integrate your automated test cases into the continuous software development cycle. If the test cases are stable, why not run them every night or after every build to quickly get results about the status of the software? Ensure long-running tests are only integrated in test automation that runs during night builds, when enough time is available to finish execution before the next build starts.

Some automation tools integrate with standard development tools that provide continuous integration capabilities. Use distributed execution to run automated test cases on several machines in parallel to reduce testing time and get the results as early as possible.