Levels of an SCM Product Evaluation and their Associated Risk

Performing an SCM product evaluation is important so that the product selected meets the needs of the application being developed.  Typically, there is not as much time spent evaluating SCM products as needed, even though an SCM product will be one of the more highly used tools in the application lifecycle.  It is with this in mind that it is important to understand the levels of an SCM product evaluation and the level of risk each assumes.

Communicating risk can help you raise the priority given to this task.  It can also lead to the appropriate level of SCM evaluation that is aligned with the level of risk an application team is willing to take.  There are several levels of a SCM Product Evaluation that may be performed, each with a different level of associated risk.  They include the Research Evaluation, the Demonstration Evaluation, and the In-House/Full Evaluation. 

Research Evaluation  This constitutes an evaluation based on reviewing existing documents (e.g., articles, research, etc) that discuss and evaluate SCM products published in technology trade magazines, journals, and evaluation documents such as (and not limited too) Software Development (SD) Times, Crosstalk, and Ovum.  This may also include reviewing input from folks with actual working experience from SCM discussion forums like CM Crossroads.  This will give you an insight into what you might look for in an SCM product.  In specific, at this level of evaluation, ensure the SCM product meets the basic entrance criteria for the environment.  Entrance Criteria will include and are not limited to development platform(s) you will be working in, basic versioning capabilities for the type of code you will be working with (source, documents, binaries, etc), and cost constraint.  After a review of these documents, a decision for selecting the product can be made.  If you perform this level of SCM product evaluation, you are placing yourself in the highest risk of incorrectly selecting the best SCM Product to fit your needs since little time is used and no demonstration and hands-on experience is occurring.   

Demonstration Evaluation   

This level of evaluation would include reviewing the output of the Research evaluation that includes focusing on the top two or three SCM products.   Then it includes defining clear SCM product requirements for the application’s needs.  This would involve forming an SCM product evaluation team, gathering SCM product requirements, weighting the requirements by importance, getting a demonstration of each requirement from SCM product vendors (and asking how their product meet the requirements), and scoring each requirement.  After the demos and scoring of the products, a selection may be made.  Performing this level of SCM product evaluation, reduces the risk of incorrectly selecting the best tool to fit your needs since you have a good idea of your SCM product requirements and you have seen demonstrations of SCM vendor products, and hopefully determined at a high-level which SCM product best meets your needs.

In-house/Full Evaluation

This level of evaluation would include the Research and Demonstration evaluation.  The top one or two products (or as many as desired) from the Demonstration Evaluation would be brought in-house for a 3-6 months to exercise and test the SCM vendor product(s) against the requirements specified and in a project scenario.  The input from Request for Proposal (RfP) from each SCM vendor can be reviewed as part of the evaluation. The SCM RfP should ask about the how the vendor delivers releases and patches, how they solicit customer requirements, and how they price their product (amongst other things).  Beyond this, you would want to establish an SCM pilot environment and select an application in which you