CM: THE NEXT GENERATION - Evaluating and Selecting a CM/ALM Tool

and the ability to export data from their tool. These may be significant requirements that you just missed the first time through. Don't be afraid to issue a secondary RFI.

V. In-house Evaluation
The rubber meets the road when it's time to evaluate. You will find a very wide range of evaluation requirements - perhaps wider than you were led to believe by your interpretation of the response results. Maybe it can be installed in minutes, ...after a two-week course. Perhaps the response time is sub-second, ...on the top of the line quad-core system with 16GB of memory. Did you realize that you needed such a machine to do a "proper" evaluation once all of your data was loaded?

Most likely, you'll find vendors have worked hard at making evaluation easy. It's you're first real view of the tool and they want a successful outcome. Some vendors will offer to come in and help you to evaluate. My recommendation: take them up on the offer, BUT only after you have had a chance to do the evaluation yourself. If you have a difficult time without them there, perhaps they can show you why when they arrive. But if they need to hand-hold all the way through the evaluation, you have a clear indication of how much consulting support and training you'll require.

Some tools will install in minutes and be fully up and running. Other tools will install in minutes and be partially up and running. You need to ensure that you can evaluate all of the major components: all of the ALM functions, their global development solution, upgrade procedures, basic customization, etc. You want a feel for how much training and how much consulting you're going to need. You also want to be able to measure the level of administration required. And, of course, you'll want to see how easy each user role is to perform with the tool.

Make sure that your evaluation is taking place with your data. It's fine to go through the checklist once with their "demo" repository. But you need to know how easy it is to load in your data, or at least a significant portion of it. And you need to look at performance with a larger, not perfectly tuned, set of data. The history graph was fine, and very pretty, with only 2 branches and 4 revisions in each. How usable is it when you scale to 10 branches with dozens of revisions in each? As well, evaluating with your own data will help others on the team to more easily participate with real scenarios.

So which tools do you evaluate first? The highest ranked tools, right? Not necessarily. You do not want to get bogged down evaluating a tool. Pick your top 3 or 5 and then address them based on how easy they are to evaluate. Some tools take weeks to evaluate, and others require extensive preparation, such as special repository platforms or special hardware. Move those to the end of your list. There is little correlation between how much prepartion is required and how functional is the tool. Some good tools require significant setup. Some virtually none. Some bad tools require a lot of setup. Some virtually none. Some tools require significant training prior to evaluation. Some require a day or two. Some vendors will charge for this training, others might not. Some will charge you only if you don't purchase their tool.

As you're working through your evaluation, give your vendors a chance... let them know what you like and don't like about their solution. Maybe they'll