Whether its focus is on detecting functional defects, performance bottlenecks, scalability issues, or security vulnerabilities, virtually every automated Web-application-testing tool on the market today follows one of two separate testing methodologies: the tool either analyzes the source code of the target application (white-box analysis) or it runs the application and interacts with it through its UI (black-box analysis). The relative merits of each of these approaches are hotly debated within the testing community, but the truth is that neither approach is adequate separate from the other. Black-box analysis is like chocolate and white-box analysis is like peanut butter: each is good on its own, but by combining the two we can create a much more delicious outcome.
The main problem with using each type of tool individually is that one alone doesn't provide adequate code coverage. In this context, code coverage refers to the ability of the testing tool to identify the entire testing surface while thoroughly examining every path of execution of the application. Unless you can be sure that your entire Web application has been tested, you cannot rely on the results of the analysis. This assurance that the entire application has been examined is especially important for security analysis tools. A single functional defect in a page visited by only one user in a thousand may have little impact on the overall perceived quality of the application, but that same security vulnerability on that page may open the entire site to a devastating attack.
Between white-box and black-box analysis, which offers better code coverage? Source analyzers definitely have the edge at finding every page in the application-they already have a list of pages from the source code. They don't need to resort to fragile techniques such as spidering the site or prompting the user to provide a list of pages, as required by black-box analysis tools. Pages like administrator portals that are not linked to any other page in the site may be completely missed by black-box tools but would be found by white-box tools. White-box tools also have the advantage at finding unusual edge cases in the code. An online shopping site might process orders made on leap days (February 29) differently from orders made every other day of the year and a black-box tool might never test this condition.
In this way, the strength of each type of tool is used to overcome the other's weakness. The cooperation between the separate components is the key that makes code coverage more complete. Hopefully the white-box and black-box analysis camps will take inspiration from the peanut butter cup and start working together to create more reliable and more trustworthy Web-application analysis tools.