You Make the Call

Navigating the Tool Selection Process

When several different test automation vendors provide similar services, it is sometimes difficult to choose the right test automation software. Clinton Sprauve illustrates how to research various vendors, establish your testing needs, and create a solid plan of attack for the test tool selection process.

By now, you have probably seen the Computer Associates television commercial with the cardboard sales people "pedaling their wares." The cardboard sales rep asks the prospect how many copies he would like to purchase. The prospect says, "We'll need about twenty copies of your soft…." But the sales rep interrupts him and says "Great! Five-hundred it is!" As the customer tries to respond, the sales rep continually interrupts with "Great! Five-hundred it is!"

This frustration is especially true for those organizations in the process of selecting test automation software. It is as if software vendors do not listen to your needs. I find that most companies have a difficult time selecting the software that will actually meet their test automation needs. The problem seems to be not in finding out which tool is best, but how to put a process in place to actually make a sound decision. This article is to help organizations identify and choose the best test automation software based on their needs and not the vendors' needs. This process should include evaluation of all products: commercial, open source, and freeware test tools. How does a company go about choosing a test automation package? Here are a few of the most common flawed approaches made by various organizations.

Vendor Bake-Off
Most companies will usually contact several vendors of choice and have them all come in and do what is known as a proof-of-concept (POC). The POC is when the vendor validates his tool(s) against your application in your test environment. The one that works the best with the customer's application (and has the best price) usually wins the business. However, this is only a very small piece of the evaluation process. The key ingredient missing in this process is the criteria needed to properly evaluate a vendor and its software.

Dive Right In
Another method used is when the customer requests an evaluation copy of the software. Although this is not necessarily a bad method for tool selection, it can be frustrating. For example, a user downloads the software and decides to jump in and begin testing with the software right out of the box. People often install the software and start using it without reading instructions, online help, or product FAQs. Then the user may encounter problems and assume the product does not work. The user will spend a significant amount of time on the phone with technical support. Test tools are not always so easy to figure out. This usually ends in frustration. In addition, the user has the task of evaluating the software and doing his actual job (imagine that). Time is lost, and the user must request an extension on the evaluation license. Imagine if this is done with five different test tools.

The Official Test Tool Guru
The next method some companies use is to have someone within their company, who is an "expert" or who has familiarity with a particular brand of test tools, lead the evaluation process. This person may have extensive knowledge with a single tool, and may have never worked with any other tool on the market. They may have used this tool successfully with a previous employer. This can lead to a biased proof-of-concept rather than an objective one. The tester will naturally gravitate toward the tool with which they are most familiar. This could lead to the company purchasing a tool that does not meet the criteria of the organization. On the other hand, extensive tool knowledge is definitely an advantage. Test tool knowledge helps companies extract the "vendor fodder" and save time in the evaluation


About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.