From time to time we hear the question: "what does testing have to with software configuration management, anyway?" Testing is essential for agile SCM Environments, and that agile CM environments are built on testing. To explain this idea we need to talk about what SCM is and how testing helps.
A software configuration management system is in place to manage change. In a 1990 Software Engineering Institute Technical Report Susan Dart said
The goals of using CM are to ensure the integrity of a product and to make its evolution more manageable.
The traditional approaches to software change management focuses on the software lifecycle and identifying and controlling what changes happened in what context. Identification of the facts of changes is important. What really matters to many stakeholders is less the fact of the change, but the impact of the change to them. People care that their system works as expected, the functionality was added as desired and not removed accidentally. We want to ensure that the value of our configurations is increasing over time rather than decreasing! To verify these things you need to test the working application. In this sense you can't fulfill the goals of SCM without testing.
As a system evolves you want to know at what point risks increase. The more frequent (and automated) the testing the easier makes it is to identify the point in time when your system is in a configuration that does not work, which is where something like continuous integration comes in. The recent book Continuous Integration: Improving Software Quality and Reducing Risk by Paul Duvall discusses how a CI system is a central part of the software quality lifecycle as you can test not just for functionality, but also for various metrics.
We will now describe the tradeoffs between and identification-based approach to SCM and a verification based approach.
Stability and Progress
CM environments balance:
- Stability - how certain you can be that the code works at any given point
- Progress - how quickly a codeline evolves to encompass new features or fixes.
How to set the balance varies depending on the needs of the project. Stability is always important since without stability it's difficult, if not impossible, to make progress. Stable in this sense means that little changes, but by this definition, "stable" can become "stale" rather quickly. A more useful definition of stable is that the quality of the code is staying as good as it was. With this sense of "stable" positive change can happen.
One way to ensure stability is to add controls and processes to ensure quality, but it is easy to make the controls interfere with the business of an organization: delivering new functionality. The classic example of this is requiring changes go through extensive (manual) review before being worked on and then requiring extensive (and time-intensive) testing before making a change. The idea behind this is well meaning: software is complicated and we don't know the effect of a change, so let's be very cautious in making changes. These rules will improve stability, at least on the surface but at a great risk to your rate of progress. In our previous article, The Illusion of Control we discuss that it is often better to have more visibility into effects of a change rather than attempt to guarantee that a change will be "safe." (For more on how to emphasize transparency over tracability see our September 2007 article Lean-Agile Traceability: Strategies and Solutions ).
Another approach is to understand that it's extremely difficult to understand the effect of a change, and instead change