Do you need to improve an area of your team's performance? Measure it and post the results. Over time you will see the performance improve. If you want to improve the quality of your product, measure it. Over time you will see the quality improve. Testing is the key to measuring quality, and CM is an equal partner. A good CM partner will provide both the communication capabilities and the information base to enable the relationships needed for a quality marriage.
A good CM suite will help you to organize your test cases, will relate them to the other development artifacts, and will provide a timely sharing of information among all the stakeholders.
Test Case Categories
Let's start with test case organization. I like to divide test cases into seven basic areas according to the purpose for creating the tests:
- Black Box Testing: To verify the Product Requirements
- White Box Testing: To verify the Design (or Software Requirements)
- Change Testing: Test change packages.
- Bug Testing: To reproduce defects and/or to verify defect fixes
- Sanity Testing: To test the basic sanity of a deliverable, as well as the sanity of the platform/infrastructure
- Stress Testing: To identify the response of a system when it is operated (inside and) outside of the product requirements envelope
- Beta Testing: To identify the defects resulting from incomplete requirements, unforeseen use cases or imperfect testing.
Black box testing deals with testing the product without knowing what's inside the box. Your requirements dictate how it is supposed to behave. Your tests verify that behaviour. The establishment of black box test cases flows directly from your product requirements. So should almost all of your product documentation. If your organization has the luxury of working from a fixed set of product requirements, your development, verification and documentation teams can all work in parallel, and can even help each other out by cross-presentation of intermediate deliverables. In many shops, requirements are changing rapidly and dynamically due to new competitive features or the advance of technology components. Although the three teams can still work in parallel, a more iterative approach may be preferable. Another key feature of black box test cases is that they can often survive from one product family to another, largely intact, minimizing the need for full re-development of the test suite when a new product definition is in the works.
White box testing deals with testing the product based on what you know about the internals of the product, that is, how it was designed and implemented. In software, a focus might be on testing internal APIs and Message Sequences. White box testing is a key component of modular design. You design a module, by specifying its interfaces, and then build your test suite to those interfaces. Once verified, you know that errors in the product will not be traced back to that module. When object-oriented design is involved, a white box test suite can be used to verify multiple objects which behave according to the class definition. A standard set of test cases may be used to test devices from multiple vendors, for example.
Change testing is primarily a developer level test sequence. When a change is made to a product (or a product-to-be), it is the developer's responsibility to ensure that the function of the change can be tested and verified to operate according to the change description. In some cases, this is similar to black box testing. The difference is that the tests are geared to the changes in functionaliy, whether those are the result of a change in requirements, staging the implementation of a feature, identification of non-compliances that must be resolved, etc. The goal is to understand what the intent of the change is and to verify that such intent is met. In our shop, we will non-traditionally refer to this a "unit" testing, where the unit is a "change" rather than a configuration unit.
Bug testing serves two purposes: to enable easy replication of a problem in the "before" scenario; and to verify that the problem has been fixed in the "after" scenario.