Automation Déjà Vu—Again!


Don't get me wrong; there's nothing wrong with asking these and other questions, particularly if you are new to IT, software testing, or even test automation. The problem, however, comes when these questions linger, seriously delaying the effective implementation of test automation and, even worse, leading many down the wrong path with regards to test automation. In addition, the over-preoccupation with these questions that have been asked over and over again for more than ten years—despite the fact that some relatively widely accepted answers are available—is one of the major symptoms of test automation's stunted growth. This stunted growth is also evident both in the fact that shelfware remains prevalent and in the missed opportunity to address more pressing concerns. Over the years, we have failed to come up with comprehensive solutions to several important automation issues, such as:

  • Detailed calculations for framework selection
  • Detailed Calculations for Automated Test Development and Maintenance Times
  • Making Risk-based (Quality-based) ROI Calculations More Acceptable
  • Moving to a Fourth Generation Automation Framework
  • Devising a Good Answer for an Acceptable Percentage of Tests that are Automated

Detailed Calculations for Framework Selection
Below is a formula that I often use to help define the level of complexity an automated test framework should have:

= AN + VN + BN + TN + CN
+ EN + Ti + P + R

  • AF = Automation Framework Definition
  • AN = Number of applications expected to be tested by your organization
  • VN = Number of versions/releases that each application is expected to have
  • BN = Number of builds/nature of application changes that each application build is expected to have
  • TN = Number of tests that you're expecting to automate for each application
  • CN = Number of configurations that an application may have to test
  • EN = Number of environments in which tests will be executed
  • Ti = Time period over which the tests will need to be maintained
  • P = Organizational process maturity
  • R = Resource technical level

About the author

Dion Johnson's picture Dion Johnson

As a senior test consultant and managing partner for DiJohn IC, Inc. and advisor to the Automated Testing Institute, Dion Johnson provides IT consulting services that focus on the overall system development lifecycle, with particular focus on the quality assurance, quality control, requirements analysis, and automated testing. He has presented at numerous SQE conferences and contributed to and Better Software magazine. In addition he is an editor for the Automated Software Testing magazine. Email Dion at or

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, is the place to go for what is happening in software development and delivery.  Join the conversation now!