Software Test Case Engineering: Treating Test Cases as a Product (or An Approach for Finding Defects that have Low Albedo Value)

[article]
Member Submitted

complex art. The complexity comes from three sources:

  • Test cases help us discover information. Different types of tests are more effective for different classes of information.
  • Test cases can be “good” in a variety of ways. No test case will be good in all of them.
  • People tend to create test cases according to certain testing styles, such as domain testing or risk-based testing. Good domain tests are different from good risk-based tests.

 After the above discussion, Cem highlights some of definitions of a test case and ends the discussion with –

 In practice, many things are referred to as test cases even though they are far from being fully documented. Brian Marick uses a related term to describe the lightly documented test case, the test idea:

“A test idea is a brief statement of something that should be tested. For example, if you're

testing a square root function, one idea for a test would be ‘test a number less than zero’.

The idea is to check if the code handles an error case.”

Then, Cem present us his definition of a test case-

In my view, a test case is a question that you ask of the program. The point of running the test is to gain information, for example whether the program will pass or fail the test. An important implication of defining a test case as a question is that a test case must be reasonably capable of revealing information.

An information objective, which is an answer of a question- test case- is the purpose of running a test case. According to Cem,

Here are some examples of what are we trying to learn or achieve when we run tests:

  • Find defects.
  • Maximize defect count.
  • Block premature product releases.
  • Help managers make ship / no-ship decisions.
  • Minimize technical support costs.
  • Assess conformance to specification.
  • Conform to regulations.
  • Minimize safety-related lawsuit risk.
  • Find safe scenarios for use of the product (find ways to get it to work, in spite of the defects).
  • Assess quality.
  • Verify correctness of the product.
  • Assure quality.

To keep the discussion about characteristics of a good test case narrow and focused on one of the above information objectives, Cem says,

Let’s narrow our focus to the test group that has two primary objectives:

  • Find defects that the rest of the development group will consider relevant (worth reporting) and
  • Get these defects fixed.

Even within these objectives, tests can be good in many different ways. For example, we might say that one test is better than another if it is:

  • More powerful
  • More likely to yield significant (more motivating, more persuasive) results
  • More credible
  • Representative of events more likely to be encountered by the customer
  • Easier to evaluate
  • More useful for troubleshooting
  • More informative
  • Appropriately complex
  • More likely to help the tester or the programmer develop insight into some aspect of the product, the customer, or the environment

Cem further discuss about how apart from information objectives, other aspects of test designing come to play a role in characterization of a good test case.

He explains that “Test Styles/Types” such as following dominate the thinking of a testers and that influence the “Test Qualities”.

  • Function testing
  • Domain testing
  • Specification-based testing
  • Risk-based testing
  • Stress testing
  • Regression testing
  • User testing
  • Scenario testing
  • State-model based testing
  • High volume automated testing
  • Exploratory testing

A test case would be “good” within these test styles because, a ‘scenario tester’ might think that,

If I was a "scenario tester" (a person who defines testing primarily in terms of application of scenario tests), how would I actually test the program? What makes one

About the author

Ajay Bhagwat's picture Ajay Bhagwat

Ajay Bhagwat is presently engaged in test management consulting and designing and delivering world-class training programs (using a unique workshop model) in Software Testing at Cognisha, a company that he has co-founded.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!