Self Verifying Data: Testing without an Oracle

Summary:

In this paper, we describe "Self Verifying Data," a method that stores both the test data and the oracle in a single data set. Combining the data and oracle solves most of the expensive maintenance problems and makes verification simpler. This method easily scales to data sets of any size and it can be used with manual and automated testing. You can adapt Self Verifying Data principles to the data you use to test almost any kind of software application.

Most of today's software applications can work with large amounts of data. Whether you're testing a database, a word processor, an Internet browser or a 3D-action game, you need to do some of your testing with very large, rich data sets. Maintaining oracles for those data sets is time consuming and expensive.

But without a good oracle, a traditional data set is worthless for testing, because your tests' accuracy can't be verified.

About the author

Noel Nyman

Noel Nyman has worked in software product development and testing for over 20 years on an eclectic project mix including embedded controllers, shrink wrap applications and operating systems. As the test lead for the Microsoft Windows NT 4.0 32-bit Applications Test team, he pioneered the use of “dumb” monkey automated test tools to increase operating system reliability. Prior to the release of Windows 2000, Noel wrote the desktop applications test plan for the Certified for Windows 2000 program. The 400-page test plan has been called “the most rigorous testing outside the military.” Noel and James Bach co-created the Windows Exploratory Testing method used by the Microsoft Windows group manual testers to create test plans for their applications. Noel developed the Self Verifying Data method for use in automated search engine verification by the Semantic Platforms group at Microsoft. He’s currently responsible for automation, bug triage, and component/tools testing in the Windows Application Compatibility group. Noel wrote the “Black Box Monkey Testing” chapter in the Visual Test 6 Bible that demonstrates how to use sophisticated Windows API testing with Rational Visual Test. He’s a regular participant in the Los Altos Workshop on Software Testing and the Austin Workshop on Test Automation, and an occasional contributor to STQE magazine.