Testing for Knowledge

[article]
Summary:

In a number of manufacturing industries, the quality function has become an integral part of the business. The software quality movement has not had nearly that level of success, but the pressure is building to do so. The obstacle is cost—particularly the cost of delivering that improved software quality. Since the beginnings of our industry, we have emulated the quality movement in hardware manufacturing; unfortunately, that path will not lead us to the success they found. Software development and hardware manufacturing are different. Our goal is the same but the path to success is different. To achieve the success we seek, we must set our own path and expand the testing objective from "test to fail" to "test for knowledge."

Winners vs. "Also-Rans"
The success of the hardware quality movement, or rather, our inability in the software quality movement to achieve a similar level of success, is frustrating. In the automotive, computer, and consumer electronics industries, the quality function has become an integral part of the business with recognized value. We, on the other hand, continue to be treated in too many cases as a "nice-to-have," but not really necessary, function. The difference is easy to explain. Other industries have been able to show a cause-and-effect relationship between their efforts and decreased costs and increased productivity—we have not done so as definitively. The difference is particularly frustrating when we look at the computer hardware industry. The quality function enjoys a recognized role in that industry's success. But computers are little more than hard-wired software. How have they succeeded when we have not? The difference is simple; they had a manufacturing problem and we do not.

The factory gave the hardware quality movement a place to demonstrate initial, small-scale successes. Their successes were based on two fundamental truths about the manufacturing process: variability is inherent and variability drives cost. If you can drive down variability in the manufacturing process, you might drive down cost-unit cost-with an emphasis on "might." The cost of discovering the nature, source, and amount of inherent variability is high. The cost of reducing that variability is high. The decrease in unit cost is low. But when you manufacture enough units, the balance tips in favor of reducing variability. The key is the number of units produced. In businesses where manufacturing volumes were high enough, the math worked. Initial successes within the factory led to opportunities for continuous improvement initiatives involving the design, development, and purchasing functions. The math is why process-improvement initiatives succeeded in the high-volume automotive, consumer electronics, and computer hardware industries while they have not succeeded to nearly the same level in other, lower-volume manufacturing operations.

It's also why process improvement initiatives have not led us to success in the software quality movement. In software, we don't have a manufacturing problem; we have a development problem. In software, process improvement initiatives start in development and stay in development. They drive up costs—costs that can't be recouped by savings in manufacturing because our initiatives do not, cannot, impact manufacturing costs. Because we have a different problem, we must take a different path to find success. The pressure to do so is growing.

The Challenge of Web-Based Opportunities
Businesses today see opportunities to materially reduce operating costs and increase operating efficiencies in Web-based B2C e-commerce, B2C and B2E self-service, and B2B c-commerce applications. Seizing these opportunities means producing applications that have increased levels of reliability, performance, and security to accommodate a new, worldwide base of users. The challenge is to deliver these additional levels of quality through improved productivity rather than increased costs.

The amount and type of testing that's done before software is deployed has always been driven by the size and nature of the intended user community. The early users of computers did not demand much from software other than that it produce the right answer when used correctly. For this user base, "Test to Pass" was the appropriate testing objective. It met the needs of the business. The 1970s saw an explosion in the number and types of computer users as microprocessors, networks, and video display terminals materially decreased costs, increased accessibility, and brought computing to the general business community. But the new user community did not welcome computers as earlier users had. By the late 1970s, it was clear that the quality of software had to be improved; otherwise, the improvements in productivity and profitability that management sought to achieve would be undermined. More failures had to be found and fixed before software was deployed to the general business community. The testing objective, and the associated cost of testing, was allowed to expand to "Test to Fail" to accommodate the needs of the business.

Web-based opportunities to achieve new levels of productivity and profitability bring another expansion of the user community. The expanded user base is, in the aggregate, less experienced, less tolerant of system errors, and less patient than the user community we've served in the past. Reliability and performance have increased in importance, as customers are now only a click away from the competition.

At the same time, the new user base contains a malicious element. Security has become a major issue. E-commerce sites have been hacked, costing their owners millions of dollars in lost revenue. Consumers have had their credit card information stolen and misused. Medical records have been accessed and made public. The list goes on. What's worse, the hackers known to be responsible for the damage inflicted to date have had relatively benign motives. Government experts tell us we need to be ready for attacks from terrorists intent on bringing down our entire way of life. Software quality becomes a matter of survival.

It's not that we don't know how to deliver increased levels of reliability, performance, and security. It's the cost. As an example, Microsoft recently shut down Windows development, sent all the programmers to a week of training, and spent the rest of the month finding and fixing security-related bugs in existing code. Using an estimated average ratio of six programmers to one tester, this means that Microsoft increased its Windows testing budget by fifty percent this year. At the same time, it decreased programmer productivity by eight and a half percent by taking them away from programming for a month. Programmer productivity will also take an additional, long-term hit as a result of the ongoing additional effort to code in security in future product releases. Given the effort being applied, the number of new test cases being generated and added to their standard regression test suite must be huge. Testing costs will also take a long-term hit. Add it all up, and that's a nontrivial number. And that's just for security.

While it's likely that Microsoft will find a way to spin this to their benefit, there's no new revenue in this for the typical business. To make matters more difficult, this happens at a time when businesses are particularly sensitive to costs. Chances are that IT budgets will not be increased to fund additional testing. Businesses are compelled to maintain and improve their profit margins. IT budgets, as a percentage of the business's revenue, will likely remain at or near current levels for the foreseeable future. Software quality must be improved to achieve important business opportunities. The obstacle is cost. We need to anticipate management's directives: Do more with less. Work smarter, not harder. We need to respond with "We can do that!"

About the author

Bill Walton's picture Bill Walton

Bill Walton has fifteen years of experience in software development organizations, mostly at large companies like IBM, Compaq Computers, and American Airlines/Sabre. He now works as an independent IT consultant specializing in project management, requirements definition, and software testing. He believes that advances in these areas are necessary to turn software development into a true engineering discipline. He welcomes comments, criticisms, and suggestions via email at bill.walton@jstats.com.
 

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Nov 09
Nov 09
Apr 13
May 03