strategies is still so undeveloped. The technologies available for testing are usually categorized into two categories – tools that support the manual testing process and tools that execute some automated testing.
The current ‘sweet spot’ for QA technology – the area where technology can yield significant ROI lies in automating as many aspects of the manual testing process as possible. According to the IT industry consultancy, Gartner, 80% of all applications are either not tested or are only tested manually before being delivered into production. There are a number of solutions that manage the administrative process of testing (test planning, execution tracking, defect tracking) such as Test Director from Mercury Interactive and ClearQuest from IBM’s Rational division.
But, many companies are beginning to realize the powerful advantage that comes from automating the tasks of the QA Engineer, not just the QA Manager. For example, consider what happens when the QA Engineer does discover a problem – the process is particularly cumbersome. First, the problem must be replicated (to ensure that it is reproducible.) Then, detailed documentation is prepared to record the exact sequence of steps executed, data entered, etc. The process of replicating and documenting problems found in the QA process can consume 25-50% of the total QA effort.
This documentation is sent to the development team for analysis, which usually starts with the developer’s attempt to recreate or replicate the problem situation so that traditional debugging tools can be deployed. This generally requires the developer to try to recreate the application environment in which the problem occurred and recreate the problem using the information sent by the QA team.
Unfortunately, although a lot of time is spent on documenting problems found in QA, many problems are incorrectly communicated or insufficiently described, which results in development not being able to recreate the problem to analyze for root cause. Despite a significant amount of “ping pong” communication between developers and testers, a troubling 22% of problems found in QA are not reproducible by development. With more complex composite applications, that number is rising towards 30%. The net result is a significant number of problems that were actually found in the QA process are not fixed before the product is released. As we learned from Barry Boehm’s 1981 research, fixing a problem before the product reaches general availability can reduce the cost of fixing it by orders of magnitude.
One innovative organization, Cerner Corporation has implemented an application problem resolution system from Identify Software, called AppSight, to automate the problem handling aspects of software bugs discovered during QA. As a world leader in healthcare software solutions, Cerner is replacing paper charts with intelligent, interactive electronic forms designed to improve patient care and business management. Until recently, Cerner took the industry-standard approach to manual QA testing, with industry-typical costs and results. ”Given the labor it takes to assure the software quality demanded in healthcare, where lives are literally at stake, we were immediately intrigued by AppSight,” commented Owen Straub Cerner’s VP of Engineering.
Beyond the Testing Mandate – Dealing with the software quality “afterlife” in production
Any organization that is serious about software quality needs to internalize the concept no matter how effective your quality testing processes are, they can not completely ensure that your software is free of bugs. As is often observed, testing can only demonstrate the presence of defects, not their absence.
Today’s applications are complex systems, comprised of numerous loosely coupled processes owned and managed by multiple parties. There has never been an instance, in recorded history, of a complex system (organic or man-made) that doesn’t sometimes fail –