Testing for Knowledge


Time to Change Paths
The expansion of the testing objective in the late 1970s signaled the rise of testing as a distinct, full-time occupation. We looked, quite naturally, at the advances taking place in the hardware quality movement led by Deming, Juran, and others, and we drew analogies. Unfortunately, most of these analogies were flawed, and still are. The barriers to productivity and profitability in software development are different than they are in hardware manufacturing. Following the path laid by our brethren in hardware will not lead us to the successes they found. We pursue the same goal, but we must take a different path.

That does not mean that our efforts at process improvement are wrong. They're not. We can point to process improvements in requirements management that decrease the probability that project costs will go out of control. We can point to process improvements using software test automation that improve software reliability by increasing test coverage. But we must face facts; in development, process improvements do not increase productivity or profitability. They provide insurance. Like any insurance, unless something goes wrong, the money spent on it simply adds cost. This is why focusing on process improvement has not increased the value and stature of our occupation.

For the software quality movement, success lies down a different path; one designed to address a different barrier than that addressed in hardware manufacturing. The barrier to improved productivity and profitability in software development is not process variability. It's knowledge. Knowledge not just of tools and processes, but knowledge of how our systems work, how they work together, and how the business uses them. And testing, not process improvement, will lead us to that knowledge. The solution lies in expanding the testing objective from "test to fail" to "test for knowledge." The expansion requires change both in what we say about testing and in what we do while testing. These changes will provide the basis for a new perception of software quality efforts and for achieving new levels of productivity; first within testing and then across the development process.

To change the business's perception of the value we provide, we need first to change how we talk about our objective. Today we say, "We test to find defects in the software." But "test to fail" gives the impression that we have an on/off, pass/fail view of things. That's not very business-like. Instead we should say, "We test to ensure the software and the documentation match. When they don't, we broker the discussion about which one to change. Change the software or change the documentation. When they match, we'll sign off." Our new job is to collect and disseminate knowledge; in this case, knowledge of whether or not the software and the documentation match. When they don't, we report it and make the business responsible for the cost decision. "How important are the specifics of this requirement to you? The software doesn't meet them all today. Do you want to spend more money to have the developers rework the software and have us check it again? Or would you rather modify the requirement?"

Next, we need to change what we do. To achieve the success we seek, we must demonstrate that our efforts improve productivity and, as a result, profitability. Automation plays an important part in improving productivity, in our case as in most others. Many of us use software test automation today. It allows us to do more testing in the same amount of time. We must expand the use of that automation-not to do more testing, but to gain knowledge that will let us do less. By driving trace utilities with our existing software test automation tools, we can discover the program modules executed by each user transaction with little or no additional effort or cost. Stored in a database, this knowledge can be queried from both directions. We can ask, "Which modules does transaction number 1 use?" and discover that it uses modules X, Y, and Z. We can also ask, "Which transactions use module X?" to discover that transaction numbers 1, 7, and 23, and only those transactions, use that code. When module X is modified, we can safely limit our testing to only those transactions. Contrast that approach to the fuller set of regression tests we typically run today based on our lack of knowledge of the system's inner workings. "Testing for knowledge" will allow us to safely do less testing in less time. That's quality and productivity improvement.

That same knowledge base can be used on subsequent projects involving that system to produce further cost improvements. The time the development group spends in the analysis and design phases to rediscover the system's inner workings can be reduced. This is typically a substantial component of the front-end effort on enhancement and maintenance projects. Part of those savings can be used to improve the requirements definition process. The knowledge can also be reused by the testing organization to decrease the cost of test planning. Test execution will enjoy even greater benefits than on the initial project, since the knowledge base already exists for the portion of the system not being modified. And if the requirements were improved, testing and rework will be reduced still further. That's continuous improvement.

"Testing for knowledge" addresses the real barrier to productivity improvement in software development: knowledge. It will allow us to deliver measurable productivity improvements "within the factory walls." It is difficult to overstate the importance of this. The quality movement in hardware manufacturing would not have succeeded in America had it not been able to demonstrate this. Downstream impacts like reductions in help desk calls or improvements in customer satisfaction are difficult to forecast and costly to measure. Business leaders are justifiably reluctant to make investments based on "iffy" forecasts. In software development, as it was in hardware manufacturing, they will make larger investments based on initial successes. Success for the quality movements in both hardware and software amounts to the same thing: delivering improved productivity and profitability for the business. The difference lies in who can deliver the initial successes. In hardware, it was the QA function that implemented process improvement. In software, it will be the QC function, "testing for knowledge."

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.