An Evaluation Framework for Continuous Integration Tools


whether it is patching clients and servers, adding plug-ins, or tuning and tweaking. In better designed tools, this functionality is built into the user interface, so that configuration and maintenance are performed from a dashboard screen in the tool’s web client. Here are some of the important things to look at for configuration and maintenance:

Configuration and Maintenance Criteria

Ranking (1-5)

Ability to change default ports


Security (at a minimum, the ability to segregate administrators from users)


Ability to change install options


Administrative dashboard


Useful out-of-the-box


Running a Simple Job

If you’ve reached this point in the evaluation, you’re probably itching to do something with the tool. Rather than dive right into building code and running tests, I suggest that you first attempt to create a ‘hello world’ job that does something simple, like printing the current machine environment to the log (the equivalent of ‘set’ in DOS, or ‘set’ or ‘export’ in UNIX). By doing this, you get to take a single pass through the tool’s job setup mechanism, which will give you a good sense of what the day-to-day interaction with the tool will be like.

While setting up your simple job, you should look for one thing above all: flexibility. If the tool is forcing a rigid conceptual framework on you, this should raise a red flag. Similarly, if job parameters are always hard-coded (such as directory path names), this is less desirable than a tool that supports macros (variable substitution). Here are some of the important things to look at:

Running a Simple Job Criteria

Ranking (1-5)

Intuitive concepts


Ability to parameterize jobs (macro support)


Simple user interface


Useful defaults


Availability of Results and Metrics

After a job runs in a continuous integration tool, you need to view the results and gather metrics. Though development tools are often geek paradise, managers are an important set of tool users – and their needs are quite different from those of a software developer or QA engineer. Managers are tasked with controlling and monitoring the development process, which makes the continuous integration process their ‘factory floor’ – and the tool’s management console their command center. Developers might have a more focused interest, perhaps just wanting to see the results of recent jobs. Both groups of users must be satisfied with the tool, or there will be tension in the development organization, and likely lost productivity.

Here are the key points to evaluate when looking at how a tool provides feedback on jobs and on the development process metrics:

Availability of Results and Metrics Criteria

Ranking (1-5)

Ability to query for a subset of results


Management dashboard


Pass/fail and performance metrics


Integrated test results


Build farm view (ability to see all jobs on all remote machines)


Web access to individual logs and reports


Dependency reports


Interacting with Development Tools

From an application lifecycle perspective, continuous integration is a sub-process in the larger lifecycle, but likely touches more tools than any other phase of the lifecycle. Source code management, issue and bug tracking systems, test tools and profiling or analysis tools all tend to be integrated into the continuous integration process. The desired output of a continuous integration run is a stable piece of code from a known location in your source repository that meets certain quality criteria, so choosing a tool that lets your development tools work together intelligently is critical. Though the raw number of available tool integrations is important, you should also look at how easy or hard it is to add

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.