An Evaluation Framework for Continuous Integration Tools

[article]

whether it is patching clients and servers, adding plug-ins, or tuning and tweaking. In better designed tools, this functionality is built into the user interface, so that configuration and maintenance are performed from a dashboard screen in the tool’s web client. Here are some of the important things to look at for configuration and maintenance:

Configuration and Maintenance Criteria

Ranking (1-5)

Ability to change default ports

 

Security (at a minimum, the ability to segregate administrators from users)

 

Ability to change install options

 

Administrative dashboard

 

Useful out-of-the-box

 

Running a Simple Job

If you’ve reached this point in the evaluation, you’re probably itching to do something with the tool. Rather than dive right into building code and running tests, I suggest that you first attempt to create a ‘hello world’ job that does something simple, like printing the current machine environment to the log (the equivalent of ‘set’ in DOS, or ‘set’ or ‘export’ in UNIX). By doing this, you get to take a single pass through the tool’s job setup mechanism, which will give you a good sense of what the day-to-day interaction with the tool will be like.

While setting up your simple job, you should look for one thing above all: flexibility. If the tool is forcing a rigid conceptual framework on you, this should raise a red flag. Similarly, if job parameters are always hard-coded (such as directory path names), this is less desirable than a tool that supports macros (variable substitution). Here are some of the important things to look at:

Running a Simple Job Criteria

Ranking (1-5)

Intuitive concepts

 

Ability to parameterize jobs (macro support)

 

Simple user interface

 

Useful defaults

 

Availability of Results and Metrics

After a job runs in a continuous integration tool, you need to view the results and gather metrics. Though development tools are often geek paradise, managers are an important set of tool users – and their needs are quite different from those of a software developer or QA engineer. Managers are tasked with controlling and monitoring the development process, which makes the continuous integration process their ‘factory floor’ – and the tool’s management console their command center. Developers might have a more focused interest, perhaps just wanting to see the results of recent jobs. Both groups of users must be satisfied with the tool, or there will be tension in the development organization, and likely lost productivity.

Here are the key points to evaluate when looking at how a tool provides feedback on jobs and on the development process metrics:

Availability of Results and Metrics Criteria

Ranking (1-5)

Ability to query for a subset of results

 

Management dashboard

 

Pass/fail and performance metrics

 

Integrated test results

 

Build farm view (ability to see all jobs on all remote machines)

 

Web access to individual logs and reports

 

Dependency reports

 

Interacting with Development Tools

From an application lifecycle perspective, continuous integration is a sub-process in the larger lifecycle, but likely touches more tools than any other phase of the lifecycle. Source code management, issue and bug tracking systems, test tools and profiling or analysis tools all tend to be integrated into the continuous integration process. The desired output of a continuous integration run is a stable piece of code from a known location in your source repository that meets certain quality criteria, so choosing a tool that lets your development tools work together intelligently is critical. Though the raw number of available tool integrations is important, you should also look at how easy or hard it is to add

About the author

TechWell Contributor's picture TechWell Contributor

The opinions and positions expressed within these guest posts are those of the author alone and do not represent those of the TechWell Community Sites. Guest authors represent that they have the right to distribute this content and that such content is not violating the legal rights of others. If you would like to contribute content to a TechWell Community Site, email editors@techwell.com.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!