The test plan should document the techniques and methods that will be employed to validate the system under test. The test plan should detail the estimates of the test cycles in relation to the delivery plan. A test estimation algorithm and how one derived these estimates are best guesses at this point, but be sure these assumptions are reviewed with the delivery team.
The same holds true for the development team. If the project is to be delivered in iterations, then naturally the team will jointly develop the estimated costs and duration. It is important to highlight to the delivery team the estimated defect rate for the delivery. As this rate is approached or exceeded, the impact to remaining deployments and regression can cause not only the timeline to skew but also costs to escalate. Feedback from these defect trends should trigger a re-estimation of remaining iterations or sprints, thereby increasing accuracy and confidence. The methods and techniques documented in the plan will support the estimation of costs and duration. Examples of these techniques include requirements-based, combinatorial, model, and scenario-based testing. Each technique has unique attributes that will be associated with the various levels of structural and functional testing. Test estimates will be challenged, so teams need to stay focused. Should all business-critical features and functions have more than 90 percent test case permutations covering the various combinations of valid, error, and user profiles? Do all the users view the feature set similarly and agree about what is critical to the operation of their needs and business?
Deciding at what level to stop testing is difficult, but there is the ever-present law of diminishing returns. Review the approach and risks with the business and product owners, and gain their insight into where there could be excessive testing and what is an acceptable level of risk. As a value-adding testing team, you must quantify the costs by articulating the number of test cases and how long it will take to deliver the quality the end-user is expecting. Understand where the greatest risks to the business exist—features frequently used by most or all customers, financial impacts from failure or errors, feature complexity, defect density, and historical data on problem areas or feature sets.
When it all comes together, the team can circle back to those quality dimensions highlighted earlier. Reliability, usability, and accuracy will manifest in the number of test cases and techniques used to satisfy the level of quality that the end-user is expecting and on which the business owner must plan to spend. Complete transparency enables the team to make sound business decisions and decide on appropriate levels of risk and tradeoffs when plans are not being met. The cost-risk-benefit equation of quality will be used as the team makes adjustments to content time and cost. There should be no surprises when the team is faced with the tough decisions that always arise during the deployment of software.
In the delivery of software, testers can wear many hats. Teams that are able to think like the end-user will make a significant contribution in ensuring that the test team is adding value and focused on meeting the client’s expectations.
Pages
About the author
Paul Fratellone is program director of quality and test consulting in the testing business unit of MindTree. Paul’s career of more than twenty-five years in information technology has been concentrated in testing, compliance, and quality management. He strives to achieve consistent execution to attain a predictable level of quality that is commensurate with the investment and enables leadership to objectively measure the success and continuous benefits from these investments.
