CM The Next Generation: Test Case Management

Testing is a complex discipline.  There are various approaches, methodologies, strategies.   So where is the connection with CM?  As with development, requirements specifications and other aspects of product development, the connection is on the management side.  A software configuraion management audit is really about demonstrating that you have test case coverage for your requirements and that the test cases have been successfully run against the target build. 

Viva la difference!

Test case management is similar to software management and to requirements management, to some extent.  But it's the differences that make test case management interesting. Let's explore a few of these differences:

(1) Just as requirements may be applicable to more than one product, so too test cases may be common from one product to another, especially when the same manufacturer is involved, or when an industry standard is being addressed.

(2) Test case management also involves tracking and management of test case results so that the FCA can be automated. Unlike the test cases, the test case results are specific to a particular product build.

(3) Whereas software tends to evolve, changes to test cases tend more to be additive.  That is, new test cases are written to supplement the existing ones, much more frequently
than upgrades to existing test cases are made.  This depends on the granularity of a test case, however.

(4) Test cases are clearly divided along automation lines.  Automated test cases can be run frequently, repeatedly and without human intervention. Presumably, the results can be extracted automatically as well.  Manual test cases chew up resources. So the goal is always to move more and more of these test cases to the automated side of the fence.

(5) Test cases require specific conditions under which they must be run.  Management of these conditions, the test environment, are as much a part of the test case as the operational test itself is.

(6) Depending upon granularity, the number of test cases may be very large.  So it is both important that test case management has some degree of automated identification, as well as hierarchical, and other organizational capabilities.  This includes option and variant tagging.

So what is the impact of these attributes on current and future CM systems?

Product Management
The CM process/tools must incorporate product management. The CM repositories must share test cases among products in a product family, and beyond.  It's not good enough to be able to export test cases from one product repository to another.  That leaves you managing multiple copies of the same test suite, which is really no different than copying a complete set of source code for each development stream. 

Test Case Results
It is not sufficient to track the test cases in the CM repository.  The data runs must also be tracked.  It must be possible to associate a test case run with a specific build and with a specific tester verification session.  We need to be able to asked who tested what, when and what were the results.  This is the proof we need to conduct the Functional Configuration Audit, so we might as well automate that audit while we're at it, or at least leave the audit to auditing the CM/Verification tools and processes rather than having to wade through the mounds of data.  

Tracking test case results in the CM repository also allows a number of other capabilities and queries.  For example:

• When did this test case last fail.
• Which requirements have never been tested?
• Which requirements are not yet verified as complete?
• What's the test history for a particular test case?

A number of testing metrics are also easily generated from the combination of test case run data and its traceability back to test cases, builds, and testers.

Change Control of Test Cases
Change to the repository of test cases is most often in the form of adding in new test cases.  This