specified below. The reader should note that it might be necessary to purchase more than one software solution from a vendor or vendors to meet the various testing needs of the project. The criteria is enumerated below.
- Automated testing tools should have version control capability
- Automated testing tools should have workflow capability for the reporting and closing of defects.
- The recording testing tool should recognize the custom controls, objects, GUI, and generic controls (i.e Active-x controls) for the application under test to allow playback of the recorded scripts.
- Automated testing tools should allow for the sequencing of test sets with dependencies for the execution of the automated test scripts. For instance execute test script B only after test script C has successfully completed.
- The recording testing tool should have a scripting language that is widely accepted, robust and recognized (i.e. Visual Basic).
- The recording testing tool should produce reports to verify the execution of the scripts and provide a means to store the execution reports.
- The automated testing tool should be compatible and integrated with standard word processors and spreadsheets
- The vendor of the automated testing tool should offer online support to allow the customer to report identified problems or bugs with the automated testing software.
- The recording testing tool should work with external/internal data sheets, to allow for the creation of parameterized data driven scripts.
- The automated testing tool should have the capability for report generation to track and collect metrics for the number of scripts that passed, failed, number of defects open/closes, number of test cases that have been developed, etc.
- The automated testing tool should have email notification for the reporting and closing of defects.
- The automated testing tool should have an open architecture that makes it flexible enough to modify it to extend the tool’s functionality.
- The automated testing tool should have the capability to allow storage of automated and manual scripts and to serve as a repository for test artifacts.
Again the reader should be aware that the delineated criteria 1-14 above is by no means an all exhaustive list of attributes that the automated testing tools should possess before acquisition but rather the aforementioned criteria should serve as a baseline for the minimum features that the automated testing tools should possess.
1. Straightjacket effect:
Companies that fall under this category purchase all their automated testing tools from a single vendor and make a significant commitment of time, money and resources to the automated testing solutions from a single vendor. The company is highly dependent on a single vendor for all their automated testing needs. Although, this approach may be cost effective if a single vendor is capable of meeting all of a company’s testing needs, in actuality this is highly unlikely due to a company's bevy of heterogeneous IT applications. The test manager and the company may instead opt for a hybrid solution of automated testing tools where automated solutions are procured from 2 vendors or more to avoid the straightjacket effect of working with a single vendor and its potential limitations. Below is an example from an actual project where the straightjacket effect hampered one of my client's ability to automate test cases.
Example: My client, a consumer products company for which I did consulting for had spent over 500,000 dollars in automated testing software from a single vendor. The company bought a bolt on to their existing ERP system running in production and wanted to test the ERP bolt-on with their automated testing tools. The vendor offered an add-in program at a cost of an extra $10,000 dollars that supposedly could recognize