Lean Metrics for Agile Software Configuration Management


As mentioned earlier, measuring the overall cycle-times for each of these types/levels of build, as well as the overall cycle-time to progress through all the build types/levels from beginning to end, can be important measures for an Agile CM environment. Drilling down further to identify waste/bottlenecks could entail measuring the segments of each of those cycle-times that correspond to waiting, information-seeking, decision-making, and integration/build server processing time and utilization.

  • In the arena of change/request management, harmonic cadences would also address release cycle planning for a product-line of products that are releases of multiple (internal and external) component releases. One can think of an overall portfolio/project and governance/CCB strategy as encompassing progressively larger levels of scale: requirement, change, use-case, feature, iteration, project/release, product/program, etc. ITIL service management addresses the various levels of these across an enterprise. Nested synchronization and harmonic cadences address how to integrate the management of networks of queues and sub-queues diverging and converging across all impacted elements of an enterprise architecture and its applications, services, and the products the enterprise uses those to produce.
  • Nested synchronization would seem to apply to branching structures where development branches feed into integration/release branches and their relation to mainline branches, and the direction and frequency with which changes get merged or propagated across codelines. Here again the cycle-time for "transferring" changes from one branch to the next (as well as through all the branching levels) can be a useful measurement for ensuring "flow."

Of course, when you can manage without the "nesting", that is ideal for continuous integration. Continuous integration together with test-driven development seems to approximate what Lean calls one piece flow. An article from Strategos discusses when one-piece flow is and isn't applicable.

In the context of SCM, particularly continuous integration and TDD, one piece flow would correspond to developing the smallest possible testable behavior, then integrating it once it is working, and then doing the next elementary "piece", and so on. This is typically bounded by:

  1. the time it takes to [correctly] code the test and the behavior
  2. the time it takes to synchronize (merge) your workspace (sandbox) with the codeline prior to building & testing it, and ...
  3. the time it takes to verify (build & test) the result

So measuring each of the above (rather fine-grained) could also prove useful -- provided that it could be done unobtrusively. If one has instituted and automated a repeatable process for creating reproducible builds, then the measures can often be performed or triggered automatically during the automated process (or the data can be generated and logged for subsequent collection and analysis).

Note that working in such extremely fine-grained increments might not always work well if the one-piece-flow cycle-time was dominated by the time to synchronize (e.g., rebase), or to build & test, or if it usually resulted in a substantially disruptive or destabilizing effect on the codeline. In those cases, if the time/cost "hit" was more or less the same (independent of the size/duration of the change), since the penalty per "batch" is roughly the same for a batch-size of one piece as it is for a larger batch-size, then it makes sense to develop (and "transfer") in larger increments ("batch-sizes") before integrating and committing changes to the codeline. Hence making some basic measurements to monitor these values can give an indication of when to stop striving for one-piece flow, or else when to revisit the build process to eliminate bottlenecks!

Experiences with Version Control-related Metrics
We've seen one or more of the following being gathered (not necessarily all of them on the same project):

  • mean & median commit frequency
  • average commit "granularity" (delta-lines-of-code per commit)
  • average number of changed files per commit
  • average number of lines changed/added/removed per file changed per commit
  • average number of files with merge-conflicts per commit
  • average time to resolve merge-conflicts per commit
  • average build-time for a private build (usually an incremental build)
  • build-time for an integration build
  • average integration-test cycle-time (time to run all the tests)
  • change volatility report (a report of the most frequently modified files over a given time period)
  • change collision report (a report of files that had the most parallel development and/or checkout-lock-contention during a given time)

Granted, the last two items above aren't really metrics (they're reports), but hopefully you get the idea.

Metrics regarding change-tasks' size and commit-frequency are relative rather than absolute metrics. We want to observe the trend over time so we might notice when: integration frequency went from ~2hrs last iteration to ~3hrs this iteration while the change-task size stayed mostly the same; or maybe the task-size increased as well, in which case we might ask if we were perhaps integrating too frequently before, or are we integrating to infrequently now, and why?

Build-time and test-time are important by themselves as absolute numbers because they help give us a lower-bound on the minimum time required to commit changes. And might help us decide whether we should try to do full builds instead of incremental, just as an example.

Combine all of those above and they basically give us an idea of approximately how many changes might be committed (or be prevented from being serially committed) during the time it takes to build & test. This might help us decide when/if to switch from single-threaded commits to multi-threaded or vice-versa, or whether to go from running partial builds/tests to more complete builds/tests.

The information about number of conflicts and time to resolves them can also help feed into the above, plus also gives as some "trends" about how disjoint are change-tasks typically are. If there is an increasing trend of parallelism, then maybe we need to ask if we are doing a good job of defining the boundaries of a use-case (story), or of splitting up the work for story effectively.

About the author

About the author

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.