The Importance of Software Testing
In March 1992, a man received a bill for his unused credit card stating that he owed $0.00. He ignored it and threw it away. In April, he received another and threw that one away too.

The following month, the credit card company sent him a very nasty note saying they were going to cancel his card if he didn't send them $0.00 by return mail. He called them and they said it was a computer error and told him they'd take care of it.

This is a rather old but amusing story found on the Internet, but current reality is not very different. According to a recent study on the economic impact of an inadequate infrastructure for software testing in the U.S., the total national annual cost resulting from the inadequate infrastructure is estimated to be $59.5 billion.

It is generally believed that software can never be made perfect.

Testing is necessary because the existence of faults in software is inevitable.

Consequently, we test software to find as many faults as we can to ensure that a high-quality product with a minimum of faults is delivered. Identifying defects is the testing team's primary responsibility. However, there is another important responsibility-verifying that the application's functionality meets the user requirements.

Testing and Test Processes
Testing is the process of exercising software to verify that it satisfies specified requirements and to detect errors. The main objectives of testing are to identify defects, nonconformance, and associated risks in a work product; to communicate all known issues to the project team and ensure that all issues are addressed in an appropriate manner before release; and to ensure that the system meets the requirements of the customer.

A generic functional testing process is illustrated in the diagram below:

Testing is not the last step; it is an ongoing part of the iterative development process.

Testing is not a one-time activity-applications need to be tested throughout their lifecycle. Every version upgrade, module addition, or enhancement, as well as every implementation at a new site or increase in user load, needs to be put through comprehensive testing.

Testing Efforts in the Development Lifecycle
A program written in one hour can have thousands of possible logical branches to test. That makes testing the most time-consuming, labor-intensive, and costly part of the development cycle.

Depending on the risk and complexity of the application under test, the average percentage of the software development lifecycle that is normally devoted to the testing effort is between 30 percent and 50 percent. The cost of testing in the majority of commercial system development can be between 40 percent and 60 percent of the total cost. The percentage may be more or less in different environments, but the important issue is that the cost of testing is significant.

The Need to Reengineer Test Management
If testing is such an important and significant part of the project lifecycle, how is it managed in most projects?

Most organizations don't have a standard process for defining, organizing, managing, and documenting their testing efforts. Often testing is conducted as an ad hoc activity, and it changes with every new project. Without a standard foundation for test planning, development, execution, and defect tracking, testing efforts are nonrepeatable, nonreusable, and difficult to measure.

Generating a test status report is very time consuming and many times not reliable. It is difficult to procure testing information such as

• How much testing needs to be done?
• How much testing has been completed to date?
• What are the results of these tests?
• Who tested what, and when was it last tested?
• What is the defect number for this failed test case?
• Do we have test results for this build?
• Do we have a history of test case results across different builds?
• How can we share the test cases with a remote team?
• Does our product meet the requirements that we originally set?
• What is the requirements test coverage?
• Is our product ready for release?

Getting this information fast is critical for software product and process quality. But many times, it is difficult to get this information, depending on the way test cases and execution results are defined, organized, and managed.

Most organizations still use word processing tools or spreadsheets to define and manage test cases. There are many problems associated with defining and storing test cases in decentralized documents:

• Tracking. Testing is a repetitive task. Once a test case has been defined, it should be reusable until the application is changed. Unstructured testing without following any standard process can result in creation of tests, designs, and plans that are not repeatable and cannot be reused for future iterations of the test. It is difficult to locate and track decentralized test documents.
• Reuse. Because it is difficult to locate test cases for execution, they are seldom used in day to day testing execution.
• Duplication of test cases and efforts. It is difficult to locate a test case, so there are chances of duplicating the same and wasting the testing effort.
• Version control. Since there is no central repository, version control becomes difficult, and individual team members may use different versions of test cases.
• Changes and maintenance. Changes to product features can happen many times during a product development lifecycle. In such scenarios, test cases can become obsolete, rendering the whole effort in test planning a fruitless exercise. It is important to keep the test case list updated to reflect changes to product features; otherwise, for the next phase of testing there will be a tendency to discard the current test case list and start over again.
• Execution results-logging and tracking. The test execution result history is difficult to maintain. It is difficult to know what testing has been done, which test cases have been executed, results of each test case that is executed, and if a problem report has been written against this failed test case.
• Incomplete and inconsistent information for decision-making. A defect database provides only one side of the information necessary for knowing the quality of a product. It tells what is broken in a product, and what has been fixed. It does not tell what has been tested and what works. This is almost as important as the defect information.
• Test metrics. If we cannot have a history of test results, it is difficult to generate test metrics like functional test coverage, defect detection effectiveness, test execution progress, etc.
• Difficult to associate related information. We also need to deal with huge quantities of information (documents, test data, images, test cases, test plans, results, staffing, timelines, priorities, etc.).
• Nonpreservation of testware. It is essential that testware (test plans, test cases, test data, test results, and execution details) be stored and preserved for reuse on subsequent versions of a single application or sharing between applications. Not only does this testware save time, but over a period of time it gives the organization a pattern, knowledge base, and maturity to pinpoint the error-prone areas in the code development cycle, fix them, and prevent the errors from recurring.
• Inconsistent processes. Organizations are not static. People move from project to project. If the testing job is performed differently for each project or assignment, the knowledge gained on one assignment is not transferable to the next. Time is then lost on each assignment as the process is redefined.
• Requirements traceability and coverage. In the ideal world, each requirement must be tested at least once, and some requirements will be tested several times. With decentralized documents, it is difficult to cross-link test cases with requirements. Test efforts are ineffective if we have thousands of test cases but don't know which one tests which requirement.

Reengineering Test Management
Reengineering, as defined by Michael Hammer in Reengineering the Corporation, is "the fundamental rethinking and radical redesign of business process to achieve dramatic improvements in critical contemporary measures of performance, such as cost, quality, service, and speed."

Information technology is essential to any reengineering effort. In the article "Reengineering Work," Hammer also suggested that instead of embedding outdated processes in software, we should obliterate them and start over. We should use the power of modern information technology to radically redesign our processes in order to achieve dramatic improvements in their performance.

The problems due to unstructured, decentralized test management can be solved by reengineering the test management process. A testing project starts by building a test plan and proceeds to creating test cases, implementing test scripts, executing tests, and evaluating and reporting on results.

The objectives of reengineering test management are to

• assist in defining, managing, maintaining, and archiving testware
• assist in test execution and maintaining the results log over different test runs and builds
• centralize all testing documentation, information, and access
• enable test case reuse
• provide detailed and summarized information about the testing status for decision support
• improve tester productivity
• track test cases and their relationship with requirements and product defects

Architecture of Test Management
Without a central point of control and clear, repeatable methodologies, it is difficult to keep the testing project on track and deliver quality applications on time with limited resources. Only a well-defined, structured, and intuitive test management process can help ensure that the testing process meets its goals and contributes to enhancing application quality.

Test management goals can be achieved with the help of IT-enabled tools built using

• a centralized relational database scheme for storing all testing information
• a Web front end facilitating testing processes and providing information for decision making

Taking this into consideration, a new architecture is proposed:

Architecture of Test Case Management System

The Relational database can be used as a container embracing all test data, test cases, test results, and related product information.

We can access and share this information by using a user friendly, interactive, collaborative, global Web front end.

We will then have consistent information for all team members to share, that is secure and in just one location, easily accessible, able to grow and be easily adapted to our changing needs.

We can make use of global interactive Web technologies and the power of databases to create a high value (low cost) test management tool, which can help in defining, viewing, editing, execution logging, and reporting on the status of the test project.

The use of a relational database as the core of a test case management system can provide the following advantages:

• Each piece of data need only be entered once. With other less unified approaches, it is often necessary to enter the same information many times. This can be time consuming, and can lead to data errors and inconsistencies.
• Several records can be updated at once by having their common data elements placed in a related record. When this one record is modified, all the records that are related to it automatically get the new information.
• It's easy to filter and sort the information to view only what's needed.
• We can generate dynamic reports from the database to select exactly the
  information we want, when we want it, to meet the specific needs of our software development and test process.
• If certain information must be kept private to certain individuals, the database can implement a security scheme that will restrict individual users' access to the data as appropriate.
• Most database products support a wide range of data-exchange formats. This means that data can be readily imported from or exported to other systems.
• The database supports multiple concurrent users.
• Storing all the information together means that we have only one area that we need to be concerned about backing up. When our database is backed up, we can be assured that all our information is safe.

The advantages of Web technology for client implementation include

• A well-designed Web form can ensure that data is collected consistently throughout the organization.
• By using "smart" data entry screens, data can be checked for errors before it is saved to the database.
• It helps team members to easily connect globally.
• It allows easy selection of data by use of pull-down menus, check boxes, and radio buttons.
• It exploits the incorporation of "value" fields to "hard-code" entries after they have been entered. This ensures repeatability and soft documentation of software test procedure inputs.
• A known template makes following standard operating procedures easy to follow.
• Placing testing information on our testing Web site will help remove all communication obstacles; everyone in the team will have the most recent information. This comes in handy when team members are from different locations. It even enables collaboration with different project partners.

A test management tool can help automate key processes in test definition, tracking, execution, and reporting.

Salient Features of Reengineered Test Management

Test design and procedure development

At the test design stage, testers will create a description of each test, document its scope and objective, and include any information that helps illustrate the purpose of a specific test such as requirements documents, functional specifications, etc.

During the test development phase, testers will document detailed test execution steps and define the expected results for each step. The test management system will help in defining and documenting test cases by providing standard Web-based, pre-formatted template forms with fields based on the product and component information for editing and creating test cases. These get posted to the centralized database. This enables standardization and consistency across the testing team. It will also help in linking to the requirements specification to ensure traceability and test coverage. The test case may have been created due to a known defect and gets an association created with that defect. We can define the sequence in which test cases should be executed. This may be based on functional dependencies or some other factors like risk and other priority.

Organization

To verify application functionality and usability, tests have to realistically emulate end-user behavior. To achieve this, test execution should follow predefined logic, such as running certain tests after other tests have passed, failed, or been completed. For example, a user logs into the system, enters a new order and then exits the system. To emulate this simple business process, it makes sense to run the tests following the exact same sequence: log in, insert order, log out. The execution logic rules should be set prior to executing the actual tests.

Review

Once the test cases have been created, we can get them reviewed by required team members and customers. It is easy to communicate the test cases to the team because of the Web interface. This will verify the test cases developed by the test team and improve them further if required before actual testing begins.

Execution

The test cases can be accessed from any computer over the intranet/Internet, depending on how the test management tool is deployed. The test management system will help in locating a test case and provide a Web interface to process the test case. As the test is processed, the tester can immediately log the actual results along with pass/fail results and additional comments. The Web-based process supports parallel execution of test cases by many team members, which is not possible with a single flat file that gets "routed" around. If the test ever fails, it has an associated defect number which the tester can look at to see if a previous defect report should be opened, or a new one created. This helps to ensure that nothing falls through the cracks.

Maintenance

The test management system will maintain an accurate history of each run, including execution configuration, date and time of run, who ran the test, and any defects that were uncovered during the run.

Defect management

When a test case fails, the tester can enter the ID of the defect that caused the case to fail. The defect is, of course, inserted into the defect tracking system. The defect can be linked with the test cases. This will provide information to reproduce and analyze the defects.

Quality testing dashboard

The tool would gather information as test cases were created, as problem reports were entered, and as test cases were executed. The data would automatically be gathered into a database and online up to the second, and reporting would be available at all times.

Because the test management system fosters a structured test process, it can provide several reports and processes that would otherwise require extensive manual data collection, organization, analysis, and reporting.

Throughout the lifecycle of a project, the test management system can provide relevant status reporting to facilitate planning, test execution, results tracking, and release decisions.

• During test development, reports are available to determine what work has been completed and what tasks remain open.
• During execution, the test management system tracks scripts that have been executed and those that have not, the result of the execution of each script, and the requirements coverage achieved and links to defects reported due to failed test cases, to provide a complete view of the release readiness.

Reports just based on defect tracking data show incomplete status; for example, a report that there are ten open defects does not tell much, unless we know how many test cases have been executed and how much requirements coverage is achieved by these test cases. We can use test management data to generate this missing information. Test case metrics complement defect reports metrics and give a better view of product quality.

A cumulative report as presented below gives better knowledge of product quality and testing status. Here it shows that there are thirty open defects and only fifty test cases are pending execution, with 900 test cases successful.

Test case cumulative report table

Apart from this, other reports can be generated based on different attributes like type of test, modules, etc. Test management can provide objective, accurate, real-time information, which is just what is needed for deciding on the quality of a product. This is the most important benefit of having a structured testing process and tool. Based on test reports available, the product manager can make informed decisions about the quality of the application under development.

Operational and Business Benefits
Reengineering test management provides a structured approach to testing, enables test process integration and improvement, and helps institutionalize process adherence.

Encouraging the use of a standardized, structured approach to testing will improve the quality of the product released to the business units.

Since testware is well preserved and maintained in a central repository and is easily accessible whenever required, it is reused throughout the lifecycle. The ability to reuse testware will improve testing productivity. The value gained by using test management tools increases as the number of software builds grows. It is the iterative use of tools and test cases throughout the project lifecycle that yields the greatest cost and time benefits.

Reuse of test cases will help save in testing time, so that test coverage can increase. This directly contributes to improving the quality of the products. The early completion of testing activities also reduces the total project time.

Test management can also help facilitate in building a knowledge repository and aiding knowledge transfer. Testers new to a project can easily jump right into a project and easily begin executing test cases. The entire test approach for the project will be laid out for them in test management, making it much easier to insert new test cases and execute existing ones.

Based on test reports available, now we can make informed decisions about the quality of the application under development.

In summary, the test management system helps improve quality, reduce testing cost and time, support decision making, and build a valuable knowledge
repository.

References
"Bringing Your Test Data to Life", by Len DiMaggio, STQE magazine, March/April 2001.