Testing and CM - A High Quality Marriage

your shop, developers would use the new build to verify that their changes are working correctly. Some shops do this as part of the implementation process (prior to integration) and some have the integration team do this verification after sanity testing. In any event, developers should verify their changes, and the integration team should independently verify them based solely on the data stored against the change record. Some organizations may even have the developer do a code review with the integration team in order to ensure that the change is exactly what it is supposed to be, and that nothing extra "snuck" in. Change testing, including the test description, the results, and the code review results, should be tracked against the change record.

White box testing should be defined as a test case hierarchy running in parallel with or as part of your design hierarchy. This should be no different than your source tree structure. Test cases should exist to exercise all of the internal programming interfaces, as well as any application prograrmming interfaces which are to be exported for use by other applications. Message protocols should also be exercised. White box testing may include running certain black box tests based on the knowledge of the design. Typically, these tests exercise the various success and failure paths of the code. Ideally these tests can be exercised on a module by module basis so that each module may be certified and then itself treated as a black box. When a module is modified, it has to be re-certified. This approach helps to reduce the complexity of white box testing.

Black box tests are generally grouped hierarchically into features, often reflecting the structure of the product requirements tree. In some systems it is possible to have the test cases attached directly to each requirement. However, this becomes more complex unless requirements to test cases form a 1:N mapping. Black box testing needs to be done against a specific build. It may be OK to run an entire database of test cases across a series of builds, but in the end, there should be a final run against the candidate build. Successive test case runs over a series of builds will help to establish metrics showing how quality is improving over time. Stream to stream comparisons may give some additional information, such as telling you the effectiveness of a change in development process or tools. A key approach to ensuring that black box testing is effective is to define your requirements for testability. Be clear. If the requirements are not sufficiently detailed, a product specification (such as a detailed user manual), should be developed to help enable better black box test case definition.

Don't expect your black box testing to give you a product quality guarantee. Once the product is in the customers' hands, additional problems will arise. A wide beta test effort can help to identify the most common outstanding problems - those most likely to be detected by your customers. Beta testing should not be an excuse for less black box testing. Nevertheless, you need to identify the percentage of problems which find their way through to the beta release. This is a guide that you can use from one release to another. If you change your testing methodology, this percentage may change and you will have some idea as to the effectiveness of the new methodology.

At last, you're ready for production. The ultimate test bed. Even a small number of problems will cause you big problems if your production volume is high. But if you've been measuring along the