Managing the Testing Process

Opening the Black Box
  1. been tested for a given version of the software?
  2. Code: What parts of the code are exercised by the tests? What problems have been found? How much of the code in a given version has been executed during testing?
  3. Problems: What problems are tested for? What problems have been found? How significant are the problems? What parts of the software are affected by the problems? What versions are affected by the problems? What requirements are impacted by these problems? What is the impact of these problems on the testing?

The questions posed for each of these areas must be carefully examined in order to properly understand and track the status of project test activities. In addition, having a solid understanding of the planning and preparation requirements for each testing phase is critical to making correct decisions about project schedule, status, and release.

Table 1 below summarizes commonly used testing metrics and where in the
testing process they apply.

Metric Type

Test Development Metrics

Test Execution Metrics

Functional Metric

  • Number of requirements allocated by test
  • % of requirements by test
    development phase
  • Number of requirements verified
  • % of requirements tested by version
  • % of requirements tested by major software component
  • Stability of server/platform per user

Code Metric

  • % of code covered per test
  • % of code coverage per major
    software component
  • Code coverage of tests completed for each version under test

Problem Metrics

  • Problems tested for in regression
  • Extreme conditions tested for in
    functional tests
  • Problems found per version tested
  • Problems found per software
  • Number of critical/high problems
    found per version

Schedule Metrics

  • % completion of functional test requirements by testing phase
  • Weighted functional requirement
  • Tests completed per version
  • Estimated number of days to complete
  • Test cycle completion time
  • Time to complete testing per
    functional area

TABLE 1: Commonly Used Testing Metrics

Managing a successful testing effort requires knowing all about project schedules. The process begins during test planning and involves not only understanding how long it will take to prepare tests, but also estimating how long it will take to run the tests and what resources will be required. During planning, preemptive decisions about required resources and the scope of testing can be made before testing actually begins. Tradeoffs between resources and time are possible at this stage. Also, decisions about what tests to automate and how much to automate can be made and tracked. At the same time, estimates of costs and numbers of test runs for different versions can be made and refined as test time approaches. This allows estimates of how much testing needs to be done concurrently, and how to plan for it. As testing is performed, the number of times the software needs to be tested and the amount of time each version takes to test can be factored in to decision making. However, it is not possible to make scheduling decisions without knowing the functionality, code, and problem aspects of the testing to be done.

Functionality is a critical measure of testing completeness. Decisions need to be made up front about the degree to which functionality will be tested, and which parts of the functionality will be emphasized during testing. Functionality is usually described in terms of requirements, which are divided among different functional and systems tests according to the nature of the requirements and the needs of the system. Often, for reasons of efficiency and to get a better understanding of how software will perform after release, test requirements are allocated very differently from the allocation of requirements among different components of the software. For this reason, it may be

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.