Continuous Staging: Scaling Continuous Integration to Multiple Component Teams

This month we will discuss some of the difficulties encountered when attempting continuous integration for multiple component teams working together to develop a large system. We describe the concept of a staging area to help coordinate the teams and stabilize the interdependencies between built versions of components.

Continuous Integration for Component-Based Development

In small team development, the practice of continuous integration [2] is an effective technique for keeping every one on the team coordinated with the latest results of all changes. Practicing continuous update [3] and private build [4] in one's private workspace [5] as part of a two-phased codeline commit strategy helps ensure that workspaces and work tasks stay in sync. Integration builds ensure that the team's codeline remains stable, consistent, coherent, and correct.

Any built version of a component that needs to be accessed by internal stakeholders (such as a QA/V&V group) needs to be identified by sing a label/tag). This ensures that anyone who needs to look at it, even after it is no longer the latest and greatest, can easily do so.  They can then also see which version of the component and its corresponding source code they are viewing.

In an ideal world, we could build the entire system directly from the sources in a one-step process, for everyone working on any component. Ideally, we would have the storage capacity, network bandwidth, processing power, and load distribution necessary to build the whole system.  At the very least, the ability to incrementally build the whole system every time before a developer commits their changes to the codeline.

Sometimes, for reasons of build-cycle-time, or network resource load, or schedule coordination (e.g., multiple time zones, or interdependent delivery schedules of components) this is not always feasible.  What happens when there are multiple teams and components, each with its own integration schedule or rhythm, that needs to coordinate with a larger-grained system integration strategy? There are many dependent factors to consider, including:

  • Relationships between sub teams, and their respective integration rhythms
  • Build-time dependencies between components (e.g., libraries and APIs)
  • Geographically dispersed teams and team-members and differences in time zones
  • Repository size and performance
  • System-build performance and available network resources

System Integration for Multiple Component Teams

Many large projects and systems require multiple teams of people to work together. In component-based development, it is common practice to see a system partitioned into multiple subsystems and/or components, with a team allocated to each component of the larger system (a component team [1]).  Each component-team develops a separately buildable part of the overall system, and typically develops and modifies only the source for their component.  The component team then takes ownership of the component's source code.  

  • Each component may be used or reused in one or more products of the overall delivered system (or within a product-family).
  • Some components may require delivery and distribution of their source code in order to build other components; while other components may require delivery and distribution of only binaries, or binaries and interface definitions (e.g., header files in C/C++).
  • The resulting delivered component versions can then be assembled together into the final system (subject to system and integration testing of course).
  • Each team should be responsible for ensuring that it delivers working code and executables to other internal stakeholders. When a two-phased codeline commit protocol is used, each task-level commit to the codeline is essentially stating that you have successfully compiled and tested the entire component with your changes incorporated.

If the team is large or dispersed enough to actually warrant sub teams, it may become necessary for each component-team to deliver tested, closed, sealed and signed, versioned binary libraries to the rest of the component teams. If the repository contains several components, and people build only their components to test their changes (rather than the entire system), then they are ensuring they have tested, closed, sealed and signed deliverables only for that one specific component! 


About the author

Brad Appleton's picture Brad Appleton

Brad Appleton is a software CM/ALM solution architect and lean/agile development champion at a large telecommunications company. Currently he helps projects and teams adopt and apply lean/agile development and CM/ALM practices and tools. He is coauthor of the book Software Configuration Management Patterns, a columnist for the CMCrossroads and AgileConnection communities at,  and a former section editor for The C++ Report. You can read Brad's blog at

About the author

Steve Berczuk's picture Steve Berczuk

Steve Berczuk is a Principal Engineer and Scrum Master at Fitbit. The author of Software Configuration Management Patterns: Effective Teamwork, Practical Integration, he is a recognized expert in software configuration management and agile software development. Steve is passionate about helping teams work effectively to produce quality software. He has an M.S. in operations research from Stanford University and an S.B. in Electrical Engineering from MIT, and is a certified, practicing ScrumMaster. Contact Steve at or visit and follow his blog at

About the author

Steve Konieczka's picture Steve Konieczka

Steve Konieczka is President and Chief Operating Officer of SCM Labs, a leading Software Configuration Management solutions provider. An IT consultant for fourteen years, Steve understands the challenges IT organizations face in change management. He has helped shape companies’ methodologies for creating and implementing effective SCM solutions for local and national clients. Steve is a member of Young Entrepreneurs Organization and serves on the board of the Association for Configuration and Data Management (ACDM). He holds a Bachelor of Science in Computer Information Systems from Colorado State University. You can reach Steve at

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, is the place to go for what is happening in software development and delivery.  Join the conversation now!