Test Automation: The Promise Versus The Reality

Member Submitted

Test automation has emerged within the software quality assurance (QA) and testing process as a way to save time and provide significant return on investment. However, many organizations are finding that a productive test automation environment is often fraught with challenges. This article compares the challenges against the expectations of what test automation should bring to a project.

For organizations testing numerous releases of an application throughout the year, it's a good bet that automation tools have been considered if not already purchased. After all, test automation is supposed to excel at quickly testing the ever-expanding regression baseline as it becomes exceedingly costly to test manually.

As IT professionals, we all do our best to test the new functionality; everyone's eyes are on it. However, in spite of our best efforts, regression testing gets short shrift.

Sometimes we get lucky and everything works. More often we aren’t so lucky. Our new functionality works fine but our users discover that pieces of existing functionality are now broken. As IT professionals, we want to catch those problems before our clients do.

Does utilization of test automation tools really allow you to tame the wild regression beast? The answer is a resounding, "Yes, but..." Test automation will save time and provide a significant return on investment if the tools are implemented correctly. This is a big "IF" and many organizations find that implementing a productive test automation environment is fraught with challenges.

The Trouble with the Path of Least Resistance
It is easy to understand why many companies purchase test automaton tools and then find themselves disappointed with the results when we analyze the buying process and examine our initial expectations.

Tool vendors want to sell tools and who can blame them? It's how they stay in business. When we buy their toolset and plan to use it for many years, we want them to be successful and to be there for us in the future.

If tool vendors want to keep the sales cycle as short as possible, they must sell the prospect on the tool's ease-of-use, i.e., Record & Playback. Watch any automated tool demo and the sales engineer will walk you through hitting the record button, executing the manual test and then playing it back. It does not elicit the "oohs" and "ahhs" it once did, but many competent managers still get sucked into the fallacy that Record & Playback will give them the automated regression baseline they need. They want it developed quickly so they suspend disbelief to that end.

The fundamental problem with Record & Playback is that it's analogous to hard-coding values in application programs. Seasoned developers know that it's inefficient to hard-code values that may change often. So why would you want to "hard-code," via Record & Playback, in automation scripts? The truth is you don't, especially if you detest spending inordinate amounts of time maintaining these scripts for future application releases. Yet, spending huge amounts of maintenance time is exactly what many companies do when Record & Playback is used to automate regression tests. Eventually, managers start asking, "Is it really worth it?" To ensure automation delivers the value your organization expected when purchasing the tools, you must have a more robust strategy than just Record & Playback.

Spend Time Now or Spend Much More Time Later A.K.A. Pay Now, or Pay More Later
If Record & Playback is not an effective way to automate, why do so many organizations use it and what are alternative options? There are several reasons why a company might use Record & Playback, but most are variations of these four:

  • Lack of knowledge of alternative approaches.
  • Vendors relying heavily on the Record & Playback function in tool demonstrations.
  • Managers rushing to create automation baselines. Using Record & Playback takes less time than developing a manageable automation framework.
  • Managers tasking non-technical testers with creating the automation suite.

Number three (3) is deceiving as Record & Playback takes the

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.