our tests, try to find out if we can "peek behind the curtain" to access the underlying data. If so, the data may help strengthen our tests by providing a stronger correlation between the results received and the results expected.
As always, we may choose to ignore some indicators, such as the time and date on the page. If so, record what you've ignored for future reference. This will help you and others to differentiate between these factors and whatever else is provided in the results.
Try to find ways to insert guards (or assertions, as they're known in programming terminology) that validate your tests and help ensure that your tests detect major changes to the system. For example, if the airline ticket Web site's prices are currently limited to US dollars, put in a guard that will fail the tests if other currencies are detected. Deliberately aborting the tests is often preferable to blind execution, which might report erroneous results.
I recommend working with others to review the results of your initial tests to decide if your tests are trustworthy and to calibrate the weightings if need be. We need to focus on improving our tests. Look for both false positives (tests that claim to pass when they should fail) and false negatives (tests that claim to fail when they should pass). Whenever you find false positives or false negatives, review your test scripts and the weightings to find ways to make the tests more reliable and robust. Even after making improvements I still expect my tests to miss some problems. After all, they rely on imperfect heuristics; however that's ok as long as they find the important problems!
Pattern matching is a practical way to implement matching the factors. Simple patterns might involve using string matching and field comparisons (for example to verify the HTTP response code field). More complex patterns might rely on regular expressions or structured queries (such as using XPATH expressions).
Field comparisons are useful when the data is structured and you are able to reliably predict where the relevant field will be in the response. For HTTP, the response code is easy to locate and easy to match. I recommend always adding a test for the value returned.
String matching is generally well supported and easier for non- technical users to work with. However, the string might be found elsewhere within the response, so take care to match the string in the correct part of the response.
Regular expressions are very powerful, particularly when matching predictable patterns of data, such as the flight details, within the response. Examples of regular expressions include: flight number, price, class (of ticket), source airport code, and destination airport code.
XPATH is useful for matching structured data and to hone in on the relevant section of data to compare within the response.
We can-and generally need to-combine the various pattern-matching techniques to determine whether the factor is present and "correct."
The scoring should be simple to code: If a pattern is matched correctly, return the respective score; otherwise return to zero. For cases such as the presence or absence of Web forms (such as bookNow), return the relevant score to +2 if found, otherwise -1. More complex scores can be implemented using case statements, etc. The value of some results will affect whether or not you evaluate other results. For instance if you detect an "HTTP 500" error you don't need to check the flight data on that page as the server has already reported an error.
By combining heuristic test oracles with the weighting of responses