An Evaluation Framework for Continuous Integration Tools


a ‘test step’ or a ‘static analysis’ step to the build job. Also pay attention to the process for integrating new tools – since you almost certainly have commercial, open source and home grown development tools that your continuous integration tool doesn’t support out-of-the-box.

Here is the short list of features that you should look for when determining how the continuous integration tool will fit in with your existing development tools:

Interacting with Development Tools Criteria

Ranking (1-5)

Number of built-in integrations


Process for creating custom integrations (low coding effort is best)


Ability to display tool output in reports


Ability to store encrypted tool credentials


GUI support for common tool options


Support for SCM triggered or clock triggered jobs


Automating Complex Build Processes

Once you’ve figured out how to call your tools, it’s time to put together a real build job, one that orchestrates your code and your tools inside of the continuous integration process. This is really the end goal of the evaluation: to automate your process and tool chain. Continuous integration tools vary widely in this area. Some don’t allow any complex orchestration of tools – you can check out from source code, but then are constrained to run your existing build script. Others provide complete workflow engines that let you chain tools together into complex build workflows, and then trigger these workflows based on source code changes or other criteria.

Since you will likely have a small mountain of existing build scripts already in use, you should always be able to run an existing script as part of a build process. Beyond that, here are the points to be aware of when determining how well a continuous integration tool will let you automate your build process:

Automating Complex Build Processes Criteria

Ranking (1-5)

Support for tool call chains


Ability to run existing scripts


Ability to insert ad hoc scripts and command line calls into the tool chain


Nesting (ability to call one job from within another job)


Distribution (ability to partition individual steps in a job to run on different machines)


Virtual machine support (ability to spin up virtual machines to run job steps)



To recap, the goal of this article was to present the evaluation framework that helps continuous integration tool evaluators make sense of the wide array of features and functionality available today. Continuous integration has come a long way from the early days of ‘Cruise Control or nothing’, and has emerged as a key business process, affecting engineering, sales and marketing organizations. Engineering owns the process, and so has the responsibility to ensure that the results are unambiguously adding value to the organization. This framework will help you make a decision that is right for your organization, and pave the way for your team to benefit from the important practice of continuous integration.

Matt Laudato is an independent consultant and editor-in-chief at <> , and has over 20 years of experience in software engineering and IT. He has held engineering, marketing and sales leadership positions at Northeastern University, Dun and Bradstreet, OpenWave, AccuRev and OpenMake Software over his career. He is also a published expert on machine learning algorithms and statistical analysis. He holds a BS in Physics from Stony Brook University and an MS in Physics from the University of Wisconsin, Madison.

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.