Improving the Accuracy of Tests by Weighing the Results

[article]

our tests, try to find out if we can "peek behind the curtain" to access the underlying data. If so, the data may help strengthen our tests by providing a stronger correlation between the results received and the results expected.

As always, we may choose to ignore some indicators, such as the time and date on the page. If so, record what you've ignored for future reference. This will help you and others to differentiate between these factors and whatever else is provided in the results.

Try to find ways to insert guards (or assertions, as they're known in programming terminology) that validate your tests and help ensure that your tests detect major changes to the system. For example, if the airline ticket Web site's prices are currently limited to US dollars, put in a guard that will fail the tests if other currencies are detected. Deliberately aborting the tests is often preferable to blind execution, which might report erroneous results.

I recommend working with others to review the results of your initial tests to decide if your tests are trustworthy and to calibrate the weightings if need be. We need to focus on improving our tests. Look for both false positives (tests that claim to pass when they should fail) and false negatives (tests that claim to fail when they should pass). Whenever you find false positives or false negatives, review your test scripts and the weightings to find ways to make the tests more reliable and robust. Even after making improvements I still expect my tests to miss some problems. After all, they rely on imperfect heuristics; however that's ok as long as they find the important problems!

Implementation Notes
Pattern matching is a practical way to implement matching the factors. Simple patterns might involve using string matching and field comparisons (for example to verify the HTTP response code field). More complex patterns might rely on regular expressions or structured queries (such as using XPATH expressions).

Field comparisons are useful when the data is structured and you are able to reliably predict where the relevant field will be in the response. For HTTP, the response code is easy to locate and easy to match. I recommend always adding a test for the value returned.

String matching is generally well supported and easier for non- technical users to work with. However, the string might be found elsewhere within the response, so take care to match the string in the correct part of the response.

Regular expressions are very powerful, particularly when matching predictable patterns of data, such as the flight details, within the response. Examples of regular expressions include: flight number, price, class (of ticket), source airport code, and destination airport code.

XPATH is useful for matching structured data and to hone in on the relevant section of data to compare within the response.

We can-and generally need to-combine the various pattern-matching techniques to determine whether the factor is present and "correct."

The scoring should be simple to code: If a pattern is matched correctly, return the respective score; otherwise return to zero. For cases such as the presence or absence of Web forms (such as bookNow), return the relevant score to +2 if found, otherwise -1. More complex scores can be implemented using case statements, etc. The value of some results will affect whether or not you evaluate other results. For instance if you detect an "HTTP 500" error you don't need to check the flight data on that page as the server has already reported an error.

By combining heuristic test oracles with the weighting of responses

About the author

Julian Harty's picture Julian Harty

A senior test engineer at Google, Julian Harty finds ways to test lots of fun products, including the mobile wireless software used by millions of users worldwide. He's been involved in software and online systems for more than twenty years and enjoys working with others to find ways to solve testing challenges productively. A presenter at both STAREAST and STARWEST, Julian has been involved in international conferences and workshops on software testing.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

May 04
May 04
May 04
Jun 01