Managing the Testing Process

Opening the Black Box

necessary to run multiple tests from several areas to completely test software components. Following this process can reveal problems in areas of the software that have been passed over in earlier tests. This is also why it is important to see testing from the code perspective.

When considering functionality in testing, it is important to think also about how the functionality is being tested. It is common to develop a relatively "quick-and-dirty" set of tests that exercise the functionality of the code in a "nominal" way, to make sure that all of the pieces are in place and a given version of the software is worth the added effort of doing a complete set of tests on. The complete testing suite should be targeted at testing the functionality to a more exacting standard, in essence deliberately trying to push the software to its limits. This type of functional testing includes load, performance, and endurance testing where appropriate.

Understanding how tests and problems map to the code is necessary to understand how far along testing has progressed, what remains to be tested, and how complete the testing has been. A detailed knowledge of the code is not required for the entire project team. However, management does need to understand how much of the code has been tested, how much remains to be tested, and how many modules are affected by problems discovered during testing. With this information, informed decisions can be made about the consequences of fixing or leaving problems alone. Armed with such information about test coverage, it is possible to understand how thoroughly the software is being tested, and how much of the software is being exercised by the tests. Often this comes as an unpleasant but enlightening surprise the first time it is measured.

When considering code coverage as a measure of the completeness of testing, it is often important to decide what a reasonable level of coverage is for the software being tested. There are many different ways to measure completeness of code coverage, some of which are extremely difficult to achieve for a typical piece of software. These higher levels of coverage are commonly used for software with a correspondingly high level of criticality-either software that is widely used, such as operating systems or Web servers, or software that is used in very critical functions, such as medical devices or aircraft controls. Regardless of the project, however, teams need to decide on a level of coverage that is most appropriate to the particular software under test.

Measuring the impact and consequences of problems that arise during testing is a critical step in the process. This should include how much of the software is affected by a given problem, at what point during testing a problem was found, and what kinds of problems regression tests are attempting to uncover. This information is needed to monitor the overall progress of the software through testing and to make informed decisions about software release. So by combining all of the different perspectives of schedule, functionality, code, and problem resolution, it is possible to understand and manage software testing, rather than treating it as a black box.

The catch to this is that while these elements of information are usually present in a software project, it is often difficult to find them. Also, typically such information is presented at a level of detail and terminology understandable only by testers and developers. Recently, test management tools have been developed to collect and present this information in the different forms that are better targeted to different members of the software development team. For

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.