Testing and CM - A High Quality Marriage

not generate new test cases, but are a grouping or application of the above sets of test cases:

• Regression Testing: This a re-application of (typically black box) test cases to ascertain already tested features still work.
• Feature Testing: This is a subset of black box testing focused on a particular feature.
• Performance Testing: This is also a subset of black box testing dealing with real-time issues.
• Environment Testing: This is a subset of black box testing identifying the effect of a product on the environment and vice versa.
• Validation Testing: This is a subset of black box testing targeted to meeting specific, required standards.
• Unit Testing: To test configuration units. This is typically a mix of Black Box and White Box testing at a card, module or subsystem level.
• Alpha Testing: Initial "beta" testing is typically done internally using "alpha" test sites.

This latter set of categories are quite valid and need to be performed and tracked. They will directly influence the formation and tracking of test runs and test sessions. They will generally not introduce any new test cases. However, they will introduce identifiable test case groupings which will have to be managed. Some of these will already be covered under the Black Box testing organization.

Revision and Change Control of Test Cases

So how do all of these test categories influence the design of your CM environment. Well there are several areas to consider. First let's look at where the test cases belong and how they are releated to other parts of your CM environment.

Test cases are part of your product. They are much like software, yet have significant differences. You may wish to avoid having to name each test case, a task diligently done for software files. Instead, you may just want your CM tool to generate test case identifiers as you add them to the system. You may still wish to group them into named directories or test groups.

Test cases come in two basic flavours: automated and manual. The good thing about automated test cases is that they are very easy to run once the test bed has been established. Manual test cases require significant manual effort. As a result, you'll run them less frequently. It is important to track this attribute against each test case, understanding that it will change as you endeavor to automate more and more of your test cases.

Test cases, like software, will change over time. They will need revision control. [An exception may exist for change tests and for problem tests.] If they are part of the product, as most black box and white box test cases are, they will need to track the product. When a new software baseline is created, it's likely that you want a new test case baseline following right on the heels. Your release 1 software will have to be maintained and supported. Well, so will your release 1 test cases. If you ever need to re-release, you'll need to re-test. So test cases will have branches which follow release streams in the same way that software does.

You will likely wish to manage changes to test cases in much the same way that you manage changes to your software files. Checkout and checkin operations should work with change packages so that you can make related test case changes in a single package. Whether or not you put your test cases through the scrutiny of a CCB is going to be a project decision that may change from time to time.

Your CM tool must let you group test cases together. Whether this