Testing and CM - A High Quality Marriage

Strictly speaking, a test case to resolve a problem should be spelled out in the problem report itself. This is how to reproduce the problem... <test case>. If I can't reproduce it, it was a transient symptom of some other issue. There may be an underlying problem, but until we can get some better frequency of reproducing the problem, I don't want to do anything more than track the symptom as having occurred. When a new deliverable is being prepared, every problem that is reported as being fixed must pass its test case or it is re-opened.

Sanity testing is done to verify the stability of the resulting build or integration. It's purpose is not to identify if specific problems have been fixed or if specific features work as advertized. Instead, the purpose is to ensure that the basic set of functionality is working to approximately the same degree, or better, as the previous build/integration. Later on in the release cycle, sanity testing may be more stringent than earlier on. The sanity tests also verify changes to the build environment. Perhaps new compilers and design tools have been introduced. Perhaps the underlying hardware platform is also evolving. The key output of sanity testing is to indicate whether or not the new deliverable is sufficiently stable to allow the team to continue development in an effective manner. It will ensure that the integrated set of components and the tools used to prepare it are in reasonable shape.

Stress testing is sometimes considered a part of black box testing. This is especially true if the product requirements indicate, for example, that there will be no catastrophic behaviour when the product is operated outside of the requirements envelope. It is also part of black box testing when specific parts or operations need to be stressed to a certain level within the envelope. An engine must be re-usable for N hours without maintenance. The database must be able to handle 10,000 transactions per hour. These are stress tests which, strictly speaking, cover the product requirements. As such, they are a special subset of the black box tests. Additional stress tests will fall outside of the product requirements. For example, how long can that engine run without maintenance. How many transactions per hour can the database support. These tests are generally run to "rate" the deliverable or component. Establishing a rating not only makes it easy to assert that a requirement is or is not met, but it also permits the establishment of additional claims such as an MTBF. A light bulb may have an MTBF of 2000 hours, even though the occasional one may fail after 100 hours. From a "spares" perspective, I would likely only need one light bulb, but I might want two or three light bulbs to guarantee a 1000 hour mission time.

Beta testing helps to identify the defects resulting from incomplete requirements or imperfect testing. As hard as you might try, requirements are going to be incomplete. A new use case for an existing capability pops up in a customer's head and guess what? It breaks the system. Your testing is not going to be perfect... when I do x,y,z three times and repeat them in the reverse order it prints out a funny message.

All of these test cases will need to be managed. They will need hierarchical groupings, change control, revision control, release management, etc. They will also need to be related to other elements of the CM system.

Other Categories of Testing

There are a number of other test case categories not mentioned above. They do