An Evaluation Framework for Continuous Integration Tools

[article]

a ‘test step’ or a ‘static analysis’ step to the build job. Also pay attention to the process for integrating new tools – since you almost certainly have commercial, open source and home grown development tools that your continuous integration tool doesn’t support out-of-the-box.

Here is the short list of features that you should look for when determining how the continuous integration tool will fit in with your existing development tools:

Interacting with Development Tools Criteria

Ranking (1-5)

Number of built-in integrations

 

Process for creating custom integrations (low coding effort is best)

 

Ability to display tool output in reports

 

Ability to store encrypted tool credentials

 

GUI support for common tool options

 

Support for SCM triggered or clock triggered jobs

 

Automating Complex Build Processes

Once you’ve figured out how to call your tools, it’s time to put together a real build job, one that orchestrates your code and your tools inside of the continuous integration process. This is really the end goal of the evaluation: to automate your process and tool chain. Continuous integration tools vary widely in this area. Some don’t allow any complex orchestration of tools – you can check out from source code, but then are constrained to run your existing build script. Others provide complete workflow engines that let you chain tools together into complex build workflows, and then trigger these workflows based on source code changes or other criteria.

Since you will likely have a small mountain of existing build scripts already in use, you should always be able to run an existing script as part of a build process. Beyond that, here are the points to be aware of when determining how well a continuous integration tool will let you automate your build process:

Automating Complex Build Processes Criteria

Ranking (1-5)

Support for tool call chains

 

Ability to run existing scripts

 

Ability to insert ad hoc scripts and command line calls into the tool chain

 

Nesting (ability to call one job from within another job)

 

Distribution (ability to partition individual steps in a job to run on different machines)

 

Virtual machine support (ability to spin up virtual machines to run job steps)

 

Summary

To recap, the goal of this article was to present the evaluation framework that helps continuous integration tool evaluators make sense of the wide array of features and functionality available today. Continuous integration has come a long way from the early days of ‘Cruise Control or nothing’, and has emerged as a key business process, affecting engineering, sales and marketing organizations. Engineering owns the process, and so has the responsibility to ensure that the results are unambiguously adding value to the organization. This framework will help you make a decision that is right for your organization, and pave the way for your team to benefit from the important practice of continuous integration.


Matt Laudato is an independent consultant and editor-in-chief at technistas.com < http://technistas.com> , and has over 20 years of experience in software engineering and IT. He has held engineering, marketing and sales leadership positions at Northeastern University, Dun and Bradstreet, OpenWave, AccuRev and OpenMake Software over his career. He is also a published expert on machine learning algorithms and statistical analysis. He holds a BS in Physics from Stony Brook University and an MS in Physics from the University of Wisconsin, Madison.

About the author

TechWell Contributor's picture TechWell Contributor

The opinions and positions expressed within these guest posts are those of the author alone and do not represent those of the TechWell Community Sites. Guest authors represent that they have the right to distribute this content and that such content is not violating the legal rights of others. If you would like to contribute content to a TechWell Community Site, email editors@techwell.com.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

May 04
May 04
May 04
Jun 01