An Evaluation Framework for Continuous Integration Tools

[article]
Summary:
Tools that enable continuous integration are vital to any agile project. Learn how putting together a well-planned evaluation process for the selection of those tools enables your entire team to work more cohesively, while eliminating the waste and damage that ineffective tools can cause.

The Importance of Continuous Integration
As agile and other lean methodologies move to the forefront of software development, one thing is abundantly clear: continuous integration is the key process for making agile work. Without it, teams have no feedback as to the stability, quality or suitability-to-task of their software. Given the short cycle times in agile methods (2 week sprints, for example), this lack of feedback is the equivalent of driving on a dark, unfamiliar road with no headlights. By introducing continuous integration, teams can implement these agile micro-cycles more effectively, and ensure that the project is progressing appropriately.

Since continuous integration is so important, teams need to have a solid framework in place for evaluating tools. Choosing the wrong tool results in wasted time and effort, forcing teams to go back to square one in order to meet the goals of fast, effective build and test cycles. Conversely, when the right tool is introduced, developers become more productive, managers gain insight into the process, and business stakeholders (often resistant to agile) buy into the process.

The Evaluation Framework

With all of this in mind, I recently embarked on a continuous integration tool evaluation. You can read the first installment of the detailed results on my blog at Technistas.com. Rather than rehashing the specific results of the evaluation, my goal in this article is to present the evaluation framework that I used, discuss why it was useful – and to save readers from investing the time and effort of developing their own framework.

At a high level, the framework facilitates evaluation of continuous integration tools along several vectors:

    • Installation
    • Configuration and maintenance
    • Running a simple job
    • Availability of results and metrics
    • Interacting with development tools
    • Automating complex build processes

The remainder of this article expands on each of these points and provides tips on how to evaluate tools in these areas. For each, I provide a table listing several specific features that you should look for. The table also has a ranking column, and I suggest that you assign a rank from 1 to 5, with 1 being a poor score and 5 a high score. In this way, some simple math at the end of your evaluation will give you an instant snapshot of how the various tools in your evaluation stack up.

Installation

Often overlooked, the ability to easily and properly install a continuous integration tool is important. Complex and failed installations waste time, and rightly sour the evaluation. If the tool installs quickly and intelligently on the other hand, you as an evaluator have cleared the first hurdle and already have a sense of the quality-consciousness of the people who developed the tool.

Continuous integration tools are typically architected as client/server products, with a web server back end and typically a web interface on the front end. Consequently, you should look at several things when installing the tool:

Installation Criteria

Ranking (1-5)

Ability to change default ports

 

Clear description of installation options

 

Ability to perform basic configuration as part of the installation

 

Configuration and Maintenance

Once the tool is installed, you may need to perform some configuration. My bias with respect to configuration is simple: less is better. By this I mean that the tool should be immediately useful once installed. Any tool that requires you to spend too much time understanding its concepts and tweaking settings before you can use it should be treated with suspicion: if it’s hard to configure, there’s a good chance that it will be hard to use.

Any tool in your shop ultimately needs to be maintained,

About the author

TechWell Contributor's picture TechWell Contributor

The opinions and positions expressed within these guest posts are those of the author alone and do not represent those of the TechWell Community Sites. Guest authors represent that they have the right to distribute this content and that such content is not violating the legal rights of others. If you would like to contribute content to a TechWell Community Site, email editors@techwell.com.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Sep 22
Sep 24
Oct 12
Nov 09