A good CM suite will help you to organize your test cases, will relate them to the other development artifacts, and will provide a timely sharing of information among all the stakeholders.
Test Case Categories
Let's start with test case organization. I like to divide test cases into seven basic areas according to the purpose for creating the tests:
- Black box testing: to verify the product requirements
- White box testing: to verify the design (or software requirements)
- Change testing: to test change packages.
- Bug testing: to reproduce defects and/or to verify defect fixes
- Sanity testing: to test the basic sanity of a deliverable, as well as the sanity of the platform/infrastructure
- Stress testing: to identify the response of a system when it is operated (inside and) outside of the product requirements envelope
- Beta testing: to identify the defects resulting from incomplete requirements, unforeseen use cases or imperfect testing.
Black box testing deals with testing the product without knowing what's inside the box. Your requirements dictate how it is supposed to behave. Your tests verify that behavior. The establishment of black box test cases flows directly from your product requirements. So should almost all of your product documentation. If your organization has the luxury of working from a fixed set of product requirements, your development, verification and documentation teams can all work in parallel, and can even help each other out by cross-presentation of intermediate deliverables. In many shops, requirements are changing rapidly and dynamically due to new competitive features or the advance of technology components. Although the three teams can still work in parallel, a more iterative approach may be preferable. Another key feature of black box test cases is that they can often survive from one product family to another, largely intact, minimizing the need for full re-development of the test suite when a new product definition is in the works.
White box testing deals with testing the product based on what you know about the internals of the product, that is, how it was designed and implemented. In software, a focus might be on testing internal APIs and Message Sequences. White box testing is a key component of modular design. You design a module, by specifying its interfaces, and then build your test suite to those interfaces. Once verified, you know that errors in the product will not be traced back to that module. When object-oriented design is involved, a white box test suite can be used to verify multiple objects which behave according to the class definition. A standard set of test cases may be used to test devices from multiple vendors, for example.
Change testing is primarily a developer level test sequence. When a change is made to a product (or a product-to-be), it is the developer's responsibility to ensure that the function of the change can be tested and verified to