Occasionally we hear the question, "What does testing have to with software configuration management, anyway?" Testing is essential for agile SCM environments and agile CM environments are built on testing. To explain this idea, we need to talk about what SCM is and how testing helps.
A software configuration management system is in place to manage change. In a 1990 Software Engineering Institute Technical Report, Susan Dart said, “The goals of using CM are to ensure the integrity of a product and to make its evolution more manageable.”
The traditional approaches to software change management focus on the software lifecycle and identifying and controlling what changes happen in what context. Identification of the facts of changes is important. What matters most to many stakeholders is less the fact of the change, but rather the impact of the change to them. People care that their system works as expected, that the functionality was added as desired and not removed accidentally. We want to ensure that the value of our configurations is increasing over time rather than decreasing! To verify these things you need to test the working application. In this sense, you can't fulfill the goals of SCM without testing.
As a system evolves, you want to know at what point risks increase. The more frequent and automated the testing, the easier it is to identify the point in time when a system is in a configuration that does not work. This is where something like continuous integration comes in. In his recent book Continuous Integration: Improving Software Quality and Reducing Risk, Paul Duvall discusses how a CI system is a central part of the software quality lifecycle. It allows testing not only for functionality, but also for various metrics. We will now describe the trade-offs between and identification-based approach to SCM and a verification-based approach.
Stability and Progress
CM environments balance:
- Stability: How certain you can be that the code works at any given point
- Progress: How quickly a codeline evolves to encompass new features or fixes. How to set the balance varies depending on the needs of the project. Stability is always important since, without stability it's difficult, if not impossible, to make progress. Stable in this sense means that little changes, but by this definition, stable can become stale rather quickly. A more useful definition of stable is that the quality of the code is staying as good as it was. With this sense of stable positive change can happen. One way to ensure stability is to add controls and processes to ensure quality, but it is easy to make the controls interfere with the business of an organization: delivering new functionality. The classic example of this is requiring changes go through extensive (manual) review before being worked on and then requiring extensive (and time-intensive) testing before making a change. The idea behind this is well meaning: software is complicated and we don't know the effect of a change, so let's be very cautious in making changes. These rules will improve stability, at least on the surface but at a great risk to your rate of progress. In our previous article, The Illusion of Control we discuss that it is often better to have more visibility into effects of a change rather than attempt to guarantee that a change will be "safe." For more on how to emphasize transparency over traceability, see our 2007 September article Lean-Agile Traceability: Strategies and Solutions.
Another approach is to understand that it is extremely difficult to understand the effect of a change and, instead, change the criteria for approving a change to do what it was expected to do, and did not break any existing functionality. You can do this by having a codeline policy that requires that new code have:
- New unit tests
- Pass a workspace build, including unit tests
- Pass an integration build, including all unit and integration tests