Testing's Role in the Software Configuration Management Process

Some CM practitioners object to placing an emphasis on testing in a discussion of software configuration management, believing that testing is the domain of a quality assurance organization and developers. Such a perspective, while traditional, is flawed, because the goals of the QA, SCM, and development processes (or organizations) are closely connected. QA Engineers are concerned with identifying what they are testing and being able to reproduce results. As SCM Engineers, we want to be able to verify that configurations are valid. And everyone is (or should be) concerned with the quality of the end product) 

Testing is essential for agile configuration environments, and perhaps for any configuration management environment.

Best Practices, Attitudes and Principles
We will start with some thoughts on last month's theme of best practices of which we consider testing is one!

Many people think the term "best practice" means one specific practice is better than all the rest (for a particular problem). We think it was intended to mean it is one of many "best practice" solutions, even for a given problem. Also, the things that repeatedly and recurringly prove themselves to work better than many initial or naive solutions, form a set of "best practices". That doesn't mean each one is the absolute best at what it does, it means each one is proven to be "best" among the other things that had been tried at the time.

Agility and Feedback
Agile software development practices are based on feedback and adjustment. Testing is one way of providing feedback, and the agile practice of Continuous Integration, supported by testing is a key feedback mechanism. Paul Duvall's book, Continuous Integration , explains that your CI build can not only tell you if the software compiles, but can also give you test results, metrics (on, among other things, test coverage), and be a window into the overall quality of the system.  The right kinds of tests enable the people using the SCM process to assign some sort of quality metric to a configuration. The trick is figuring out what the right kinds of  tests are.

Attitude and Principles 
An important point to raise is the attitude and way in which we go about our jobs of developing software or managing configurations.

Continuous improvement (or kaizen from the Japanese) is an elegant expression of some powerful ideas (from http://en.wikipedia.org/wiki/Kaizen) :

To be most effective kaizen must operate with three principles in place:

• consider the process and the results (not results-only) so that actions to achieve effects are made apparent;

• systemic thinking of the whole process and not just what is immediately in view (i.e. big picture, not solely the narrow view) in order to avoid creating problems elsewhere in the process; and

• a learning, non-judgmental, non-blaming (because blaming is wasteful) approach and intent will allow the re-examination of the assumptions that resulted in the current process.

While kaizen (often associated with the Toyota Production System) usually delivers small improvements, the culture of continual aligned small improvements and standardization yields large results in the form of compound productivity improvement.

Thus there is a need to be aware of what we are doing and the effects of it, which brings a different quality of approach, and allows us to question current practices and seek to improve them.

This attitude and approach will lead us both to consider and adopt appropriate patterns and also to implement them in such a way that we can measure and improve (or even discard them if they turn out to be inappropriate for our particular requirements).

The Search for Simplicity
Richard Koch, in his book