Because of the fast evolution of Continuous Integration (CI), the first generation of enablement tools proliferated at lightning speed. Open source CI tools became widely used due to the ease in which an engineer could install it and start tackling the initial CI challenges that he faced. Once proven effective, these apps (particularly Cruise Control) spread like wildfire among other build engineers, and in most cases, development shops began ‘sewing' several instances together.

While this works for a certain amount of time, eventually, the law of diminishing returns sets in - especially in cases where organizations grow along one or many dimensions (code base, target platforms, size of teams or locations).  My company receives weekly inquiries from build management professionals who say their CI environments have become too complex, and they're spending time and money on scripting or simply applying Band Aids to the system to keep it running.  I am often asked for advice on whether or not a development team or company should to upgrade their CI environment to a fully automated enterprise system.  To help them assess their needs, I always start by asking these seven questions:

• Are you creating builds more than once a day?
• Do you have multiple end products (targets, platforms) for your builds?
• Is your current CI environment made up of multiple, standalone systems throughout the company?
• Are your teams geographically dispersed?
• Is it difficult to scale your build-test-deploy environment to the next level?
• Are there barriers to sharing and reusing scripts?
• Is it challenging to track builds over time and conduct cross-project reporting?

If the answer is yes to two or more of the above questions, it's usually a sign that the CI environment has become too complex for open source.  If true, the next step is to define CI environment requirements.  Below are the classic needs that large-scale software production environments address:

• Scalability--to support multiple teams and projects with one system
• Turnkey automation - so that anyone, anywhere can use the system and that builds can be performed without specialized knowledge
• Rapid software build turnaround--with a highly complex code base, complete a large number of projects and builds on short timelines
• Virtualization--to leverage and control virtual resources managed by VMWare or Microsoft, and to enable productivity gains and reduced IT resource costs
• Auditing--addresses the need for end-to-end traceability of the entire software production process for Sarbanes Oxley compliance
• Reporting--identify build management trends, monitor utilization of build resources, track historical problem areas

I have supervised or consulted on dozens of projects for development teams across the globe that made the leap from open source-driven to consolidated, enterprise-wide CI systems.  While it may sound like a painful transition, benefits include performance improvements (2-5x improvement in build times, the ability to manage 1,000s of CI builds across locations, etc.), quality improvements, and even team morale.  And given the economy, teams are finding it beneficial for development team members to spend less time babysitting CI scripts and more time enhancing the product and adding features.  The release team is no longer the bottleneck, but instead, seen as a strategic player in the company's software production.