Activities: How do you test?: A common distinction is made between behavioral test design, which defines tests based on functional requirements, and structural test design, which defines tests based on the code itself. These are two design approaches. Since behavioral testing is based on external functional definition, it is often called "black box," while structural testing—based on the code internals—is called "white box." Indeed, this is probably the most commonly cited definition for black box and white box testing. Another activity-based distinction contrasts dynamic test execution with formal code inspection. In this case, the metaphor maps test execution (dynamic testing) with black box testing, and maps code inspection (static testing) with white box testing. We could also focus on the tools used. Some tool vendors refer to code-coverage tools as white box tools, and tools that facilitate applying inputs and capturing inputs—most notably GUI capture replay tools—as black box tools. Testing is then categorized based on the types of tools used.
Evaluation: How do you know if you've found a bug?: There are certain kinds of software faults that don't always lead to obvious failures. They may be masked by fault tolerance or simply luck. Memory leaks and wild pointers are examples. Certain test techniques seek to make these kinds of problems more visible. Related techniques capture code history and stack information when faults occur, helping with diagnosis. Assertions are another technique for helping to make problems more visible. All of these techniques could be considered white box test techniques, since they use code instrumentation to make the internal workings of the software more visible. These contrast with black box techniques that simply look at the official outputs of a program.
To summarize, black box testing can sometimes describe user-based testing (people); system or requirements-based testing (coverage); usability testing (risk); or behavioral testing or capture replay automation (activities). White box testing, on the other hand, can sometimes describe developer-based testing (people); unit or code-coverage testing (coverage); boundary or security testing (risks); structural testing, inspection or code-coverage automation (activities); or testing based on probes, assertions, and logs (evaluation).
So now that we've examined some ways to think about the differences between black box and white box testing, let me leave you with a few puzzles. Let's hear what you think.
A. A programmer tests a class to ensure that it meets its functional requirements. Is this black box or white box testing?
B. Your company develops software under a contract that stipulates that both white box and black box test techniques will be used. What tests are you obliged to execute?
C. A nonprogrammer uses a test tool that automatically instruments the code and then generates tests to ensure that a maximal number of lines of code are executed. The tests are considered to pass as long as the software doesn't crash or hang. Is this black box or white box testing?
D. What could it mean to perform "gray box" testing?