Evaluating CM Tools

This is not a new topic, but this is what I expect from a CM tool:

• Reliability and accessibility
• Low administration and small footprint
• Change packages
• Getting started fast
• CM capabilities
• Integrated suite
• Easy configuration and process capability
• Easy to use
• Low cost of ownership

The Key Evaluation Areas
1. Reliability and accessibility:  If my team is going to waste time, I would just as soon have SCCS. A CM tool must have ultra-high reliability and availability. If something does go wrong, I want to be able to recover - from disk crashes to disaster recovery. Even from data sabotage. Make backups as painless as possible so that they get done. Don't take down the whole team because a server is down. If my team is distributed, a multiple site solution must allow reliable access to all project data that I'm allowed to access.

2. Low administration and small footprint: These go together.  If it's not a small footprint, it's not going to be low administration. Ideally, it’s a "set-and-forget" technology and I don't have to buy special hardware or software to get the solution going. I don't want a CM tool that's going to force me to spend more on CM. I want one that's going to do the job by itself. I don't mind having to archive transaction journals, do backups, etc. But don't ask me to continually tune my system, redistribute my data, calculate and perform manual synchronization operations (e.g., between sites or data partitions). Also, don't ask me to spend weeks on course to learn the system.

3. Change packages:   Sure I want to know what files are in my tree and I want to organize them, but when it comes to making changes, I want to create changes, not just edit files. My changes should flow through the system. Downstream personnel, including builders, should not even need to know if my change involved one file or ten. I don't want to have to check in each file separately, perhaps forgetting one, do delta/difference reports on each file, do promotion of each file or even merge each file into a different branch. I want to do all of these things on a change basis. I want to track my structural changes (add, remove, delete) as part of my change. Ideally, the change would be the place to associate traceability data, integration information and some level of test data (unit testing).
4. Getting started fast:  I only have to get started once. Well, at least until I load in a new product, but I want to be able to get started quickly. Low training requirements, easy to install (and evaluate), easy to create a new CM repository, easy to load in source code (latest baseline or multiple baselines), easy to load in multiple products, easy to load in other data such as problem tracking data, planning data, etc.

5. CM capabilities:  Version control of files of course, but including binary files. Easy to retrieve any subset of files (point and click preferably) and simple CM strategies (don't give me a book I have to read to understand how our process works). I also want easy base lining without tedious work such as file by file labeling. Again, I want the CM tool to help me, not one that complicates my life. I don't care too much if I have to create my own process or if it's pre-packaged (and reasonable), as long as I don't have to spend weeks or months configuring this process. Don't make me branch to remember promotion levels. Don't make me branch to identify custom builds. Give me stream-based development, not branching chaos. And give me good build tracking so that I can identify every build of significance for all time. Minimize my need to merge and give me decent tools to do merging when I have to.

6. Integrated suite:  I don't want to spend months putting together a set of applications that work together only to find that when one of the tools is upgraded I have to re-do all of the glue. I don't want an internal tools group, since that's why I'm buying a tool. If I wanted an internal group, I'd just as soon create the tool myself. I do want easy navigation among the components: CM data, change data, problem Data, Project Management data, requirements, build and release data. And I want the extra apps thrown in almost for free as opposed to what I would pay if I had to buy them all separately - even before the cost of the glue!

7. Easy configuration and process capability:  I want to be able to do the basics, such as defining triggers and rules, defining basic work flow, adding data fields as necessary, changing the GUI menus, including context (pop-up) menus and what they do. I don't mind having to train a bit to do this, but don't give me a C++ compiler and tell me it’s configurable. I want to be CMMI 5. What will the tool do to help me get there? If it starts me out at 2 or 3, that's good, but if it helps to get me most of the way to 4 that's great. If I can add process, data and functionality to continually improve my process and export my results for use by the rest of the company, great! If I can easily extend my tool suite without having to acquire other tools, even better!

8. Easy to use:  It has got to cover my basic use cases. I don't want to be fighting with the tool. I don't mind having to do a bit more work for out-of-the-normal operations, but better still if I don't have to, especially because I'll forget how to do them if it's not dead easy. If I can tailor the tool to be familiar (change dialogs, use corporate grown terminology, etc.) that will make my life easier. If I can get a developer fully trained in under a day, more points.The Cost? Or the Savings?
9. Low cost of ownership: Most of this has really already been covered. Sure there are free tools and more expensive ones. As long as I can get my licenses for below 2% of the salary, it'll cost only a couple to a few hundred per year (amortized over 5 years). I shouldn't have to pay more than that for the entire suite of development management tools. The more costly items are things like: training (lost salary cost is 0.4% of salary per day of training); administration costs, unless it's set-and-forget technology; tool glue, unless it's an integrated suite; down time, unless it's reliable; lost user productivity, unless it’s easy to use; reliable and accessible; evaluation and deployment (and upgrade) time, unless you can get started fast.

But I'm not really interested in the cost. I really want to know how it's going to make me money. Where is it going to save me money?

• Will it eliminate tedious, time-consuming operations?
• Will it help improve the quality of the products?
• Will it reduce my process assessment costs and maturity levels?
• Will it support re-use?
• Will build production be more efficient and reliable?
• Will developer and management productivity improve?
• Will I be able to make more timely decisions?

Look at Your Use Cases
What about all the other features: ease of labeling, merge tool, interactive branch diagrams, compatibility with SQL tools, etc. Don’t look at the features look at your use cases. What operations are you going to do? How frequently? How easy are they to do?

We have a lot of prospective clients looking at CM+ (Neuma's CM tool) and asking questions like: how easy is it to label? How difficult is it to put together a configuration view description? What about merging, since we do a lot of that?  What's your API like, as we had a lot of difficulty integrating our last CM solution with a problem tracking capability. We tell them that it's easy to label, you don't have to put together configuration view description files, and you can plug in any number of differencing and merge tools, including the ones we ship with the product. We've got a variety of API capabilities. BUT these are really not that important [in CM+]. You won't need to do labeling any more. Pick a product/stream and optionally promotion level, or a build or baseline identifier and your view is set. Merging will move from a high-use to a minimal use frequency. You'll start out with a full suite of management tools and standard support for IDE integration, so you won't really be as interested in the APIs.

Be careful when you're evaluating to first put yourself in the context of the new tool. If you currently use a 7-way merge tool and your current process requires 7-way merges frequently, don't evaluate a new tool for its 7-way merge capability. First find out if you're still going to have to do 7-way merges. If you want SQL compatibility because you have to mine out all of the traceability links to feed into another database which will let you cross reference them and generate the data for building a release report (I'm out of breath, too!).  First find out if your new tool will let you generate a release report with a couple of clicks. Don't discount the tool because it doesn't have SQL compatibility. You're looking for a new tool to eliminate your complexities, your dependencies and your headaches. If a new tool will eliminate these problems, don't be looking for features that will help you deal with them.

As a CM manager, ask yourself, before you evaluate: What would I rather be spending my time on - tedious configuration management tasks that should be automated, or change management. If a tool makes you spend a lot of time creating baselines, doing promotions, dealing with exceptions, generating status reports and version/build documents, or if the information is not quickly available with a few clicks, you may want to cast your tool search wider, especially if climbing the process maturity ladder is one of your goals.

Scalability
What happens to my performance as my project grows from hundreds to thousands to hundreds of thousands of objects? Does response remain snappy? Look for "smart client" technology that scales easily from 2 to 2000 users.

How many users, roles and disciplines do you need to support? When you move from 10 to 50 users, is it a big deal or not? Do I eventually hit a wall and have to split my server and or repository up?

What's the Score?
When it comes down to ranking, do a hierarchical weighting. Don't give administration the heaviest weight because there were the most bullet items there to be evaluated. Decide on your evaluation weights at a macro level. Perhaps choose the areas I picked out above. Take 1000 points and distribute them to those areas. Then it doesn't matter how many bullet items are evaluated in each area. If necessary, go another level in some areas. Perhaps CM Capabilities should be broken down into workspace management, CM manager functions, and basic change and version control.

Then get some consensus on the weighting - different experts will have expertise on different areas of the ranking tree. Recognize that as part of your input gathering. Don't treat your testers' input on administration with the same weight as your SCM administrator's input. Recognize that there are 150 developers, 25 testers, 40 managers and 3 administrators.  A little feature for each developer is a lot more important that a big feature for an administrator.

Finally, look to the future. Don't pick a tool that's good for the next 5 years. Pick one that will last you 20 years or more. You don't know where your product is going to end up. The first CM tool I designed is soon to enter its 30th year of operation on its initial project. Did I foresee this happening 30 years ago? Did I build a nice GUI? (Hint: the term didn't even exist, although text-based CRT displays were getting popular). No. But I did make sure it was built on fundamental principles and that it was scalable beyond the day's technology. So look at the technology, look at the scalability, look at the performance, look at the standards - make sure it will stand the test of time.

So what are your favorite evaluation criteria? What did I miss? How do you approach your evaluations? Drop me a note or start a thread.