The formula is not meant for literally plugging in numbers to get an answer for AF, but rather simply to illustrate the relationship of the framework choice to the automation scope—a relationship that may be used as guidance for selecting an automation framework. However, the past ten years could've been used to develop consensus within the industry about how this type of formula could be used literally to plug in values for an actual answer to the type of framework that might work best for the identified scope.
Detailed Calculations for Automated Test Development and Maintenance Times
How long should one expect to develop and maintain automated tests? There is a factor that is widely accepted among industry leaders that it will take three to ten times as long to automate a test as it does to execute it once manually. That means if it takes you one minute to execute a test manually once, it will probably take somewhere between three to ten minutes to automate that test. This is still a fairly broad range because there is no industry standard regarding the time it should take to maintain an automated test suite given a certain scope. We need to focus on further defining this information.
Making Risk-based (Quality-based) ROI Calculations More Acceptable
The three basic approaches for measuring ROI are:
- Simple ROI: Quantifying automation benefits in terms of the monetary savings automated test execution provides over manual test execution of the same set of tests
- Efficiency ROI: Quantifying automation benefits in terms of the time savings automated test execution provides over manual test execution of the same set of tests
- Quality ROI: Quantifying automation benefits in terms of the potential monetary savings automated test execution provides via increased test coverage and reduced risk of application failures
Each measure has its pros and cons, but there is one measure that has been largely neglected in practice—Quality ROI. Quality ROI provides a broader view of the benefits of test automation that can get lost if you only focus on the Simple and Efficiency ROI measures. Neglecting the broader view may hurt an automation effort that is already riddled with unrealistic expectations about an immediate, high-monetary ROI. The past decade has seen lost opportunity in effectively moving the industry to a more balanced approach for measuring ROI.
Moving to a Fourth Generation Automation Framework
Automation frameworks may be discussed in terms of three generations:
- Generation 1 (Linear): Framework that is composed of automated tests where all components that are executed by a given tests are mostly contained within that automated test
- Generation 2 (Data-driven, Functional Decomposition): Framework that introduces more modularization, reuse of common components among several scripts, and greater separation of data from automation code
- Generation 3 (Keyword, Model Based): Framework that adds a new level of abstraction that separates code logic from test logic, allowing automated tests to be built and maintained in a less technical way
Each generation has its pros and cons and should be evaluated for use in terms of the automation scope of the organization. There is an opening, however, for a new generation to be introduced. I've dabbled with the concept of building detailed manual test procedures that could be interpreted by an engine and therefore double as automated tests. In addition, there have been some tools that have sought to offer this capability through business components that may be used to build manual test procedures that may also double as automated tests. So while there has been some movement in this direction, it could and should be more pronounced.