Lean Metrics for Agile Software Configuration Management


As mentioned earlier, measuring the overall cycle-times for each of these types/levels of build, as well as the overall cycle-time to progress through all the build types/levels from beginning to end, can be important measures for an Agile CM environment. Drilling down further to identify waste/bottlenecks could entail measuring the segments of each of those cycle-times that correspond to waiting, information-seeking, decision-making, and integration/build server processing time and utilization.

  • In the arena of change/request management, harmonic cadences would also address release cycle planning for a product-line of products that are releases of multiple (internal and external) component releases. One can think of an overall portfolio/project and governance/CCB strategy as encompassing progressively larger levels of scale: requirement, change, use-case, feature, iteration, project/release, product/program, etc. ITIL service management addresses the various levels of these across an enterprise. Nested synchronization and harmonic cadences address how to integrate the management of networks of queues and sub-queues diverging and converging across all impacted elements of an enterprise architecture and its applications, services, and the products the enterprise uses those to produce.
  • Nested synchronization would seem to apply to branching structures where development branches feed into integration/release branches and their relation to mainline branches, and the direction and frequency with which changes get merged or propagated across codelines. Here again the cycle-time for "transferring" changes from one branch to the next (as well as through all the branching levels) can be a useful measurement for ensuring "flow."

Of course, when you can manage without the "nesting", that is ideal for continuous integration. Continuous integration together with test-driven development seems to approximate what Lean calls one piece flow. An article from Strategos discusses when one-piece flow is and isn't applicable.

In the context of SCM, particularly continuous integration and TDD, one piece flow would correspond to developing the smallest possible testable behavior, then integrating it once it is working, and then doing the next elementary "piece", and so on. This is typically bounded by:

  1. the time it takes to [correctly] code the test and the behavior
  2. the time it takes to synchronize (merge) your workspace (sandbox) with the codeline prior to building & testing it, and ...
  3. the time it takes to verify (build & test) the result

So measuring each of the above (rather fine-grained) could also prove useful -- provided that it could be done unobtrusively. If one has instituted and automated a repeatable process for creating reproducible builds, then the measures can often be performed or triggered automatically during the automated process (or the data can be generated and logged for subsequent collection and analysis).

Note that working in such extremely fine-grained increments might not always work well if the one-piece-flow cycle-time was dominated by the time to synchronize (e.g., rebase), or to build & test, or if it usually resulted in a substantially disruptive or destabilizing effect on the codeline. In those cases, if the time/cost "hit" was more or less the same (independent of the size/duration of the change), since the penalty per "batch" is roughly the same for a batch-size of one piece as it is for a larger batch-size, then it makes sense to develop (and "transfer") in larger increments ("batch-sizes") before integrating and committing changes to the codeline. Hence making some basic measurements to monitor these values can give an indication of when to stop striving for one-piece flow, or else when to revisit the build process to eliminate bottlenecks!

Experiences with Version Control-related Metrics
We've seen one or more of the following being gathered (not necessarily all of them on the same project):

  • mean & median commit frequency
  • average commit "granularity" (delta-lines-of-code per commit)
  • average number of changed files per commit
  • average number of lines changed/added/removed per file changed per commit
  • average number of files with merge-conflicts per commit
  • average time to resolve merge-conflicts per commit
  • average build-time for a private build (usually an incremental build)
  • build-time for an integration build
  • average integration-test cycle-time (time to run all the tests)
  • change volatility report (a report of the most frequently modified files over a given time period)
  • change collision report (a report of files that had the most parallel development and/or checkout-lock-contention during a given time)

Granted, the last two items above aren't really metrics (they're reports), but hopefully you get the idea.

Metrics regarding change-tasks' size and commit-frequency are relative rather than absolute metrics. We want to observe the trend over time so we might notice when: integration frequency went from ~2hrs last iteration to ~3hrs this iteration while the change-task size stayed mostly the same; or maybe the task-size increased as well, in which case we might ask if we were perhaps integrating too frequently before, or are we integrating to infrequently now, and why?

Build-time and test-time are important by themselves as absolute numbers because they help give us a lower-bound on the minimum time required to commit changes. And might help us decide whether we should try to do full builds instead of incremental, just as an example.

Combine all of those above and they basically give us an idea of approximately how many changes might be committed (or be prevented from being serially committed) during the time it takes to build & test. This might help us decide when/if to switch from single-threaded commits to multi-threaded or vice-versa, or whether to go from running partial builds/tests to more complete builds/tests.

The information about number of conflicts and time to resolves them can also help feed into the above, plus also gives as some "trends" about how disjoint are change-tasks typically are. If there is an increasing trend of parallelism, then maybe we need to ask if we are doing a good job of defining the boundaries of a use-case (story), or of splitting up the work for story effectively.

About the author

Brad Appleton's picture Brad Appleton

Brad Appleton is a software CM/ALM solution architect and lean/agile development champion at a large telecommunications company. Currently he helps projects and teams adopt and apply lean/agile development and CM/ALM practices and tools. He is coauthor of the book Software Configuration Management Patterns, a columnist for the CMCrossroads and AgileConnection communities at Techwell.com,  and a former section editor for The C++ Report. You can read Brad's blog at blog.bradapp.net.

About the author

Steve Berczuk's picture Steve Berczuk

Steve Berczuk is a Principal Engineer and Scrum Master at Fitbit. The author of Software Configuration Management Patterns: Effective Teamwork, Practical Integration, he is a recognized expert in software configuration management and agile software development. Steve is passionate about helping teams work effectively to produce quality software. He has an M.S. in operations research from Stanford University and an S.B. in Electrical Engineering from MIT, and is a certified, practicing ScrumMaster. Contact Steve at steve@berczuk.com or visit berczuk.com and follow his blog at blog.berczuk.com.

About the author

Robert Cowham's picture Robert Cowham

Robert Cowham has long been interested in software configuration management while retaining the attitude of a generalist with experience and skills in many aspects of software development. A regular presenter at conferences, he authored the Agile SCM column within the CM Journal together with Brad Appleton and Steve Berczuk. His day job is as Services Director for Square Mile Systems whose main focus is on skills and techniques for infrastructure configuration management and DCIM (Data Center Infrastructure Management) - applying configuration management principles to hardware documentation and implementation as well as mapping ITIL services to the underlying layers.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!