Whether its focus is on detecting functional defects, performance bottlenecks, scalability issues, or security vulnerabilities, virtually every automated Web-application-testing tool on the market today follows one of two separate testing methodologies. The tool either analyzes the source code of the target application (white-box analysis), or it runs the application and interacts with it through its UI (black-box analysis). The relative merits of each of these approaches are hotly debated within the testing community, but the truth is that neither approach is adequate on its own. Black-box analysis is like chocolate, and white-box analysis is like peanut butter: Each is good on its own, but by combining the two we can create a much more delicious outcome and more reliable test results.
The main problem with using each type of tool individually is that one alone doesn't provide adequate code coverage. In this context, code coverage refers to the ability of the testing tool to identify the entire testing surface and thoroughly examine every path of execution of the application. Unless you can be sure that your entire Web application has been tested, you cannot rely on the results of the analysis. This assurance that all of the application has been examined is especially important for security analysis tools. A single functional defect in a page visited by only one user in a thousand may have little impact on the overall perceived quality of the application, but that same security vulnerability on that page may open the entire site to a devastating attack.
So, between white-box and black-box analysis, which offers better code coverage? Source analyzers definitely have the edge at finding every page in the application-they already have a list of pages from the source code. They don't need to resort to fragile techniques like spidering the site or prompting the user to provide a list of pages, as required by black-box analysis tools. Pages like administrator portals that are not linked to any other page in the site may be completely missed by black-box tools but would be found by white-box tools. White-box tools also have an advantage at finding unusual edge cases in the code. An online shopping site might process orders made on leap days (February 29) differently from orders made every other day of the year, and a black-box tool might never test this condition.
On the other hand, source analyzers cannot analyze modules for which they don't have the source code. For Web applications, these modules could include controls or libraries purchased from third-party vendors, database-stored procedures, external Web services, or modules native to the underlying Web server or application framework. Also consider that newer Web technologies such as Ajax rely greatly on client-side JavaScript. JavaScript actually has the ability to programmatically modify itself at runtime. This makes reliable analysis of a complex JavaScript application virtually impossible when using only white-box tools, but a black-box analysis tool would handle this situation easily.
Given that neither type of tool provides assurance of complete code coverage on its own, your first instinct might be to use both a white-box analysis tool and a black-box analysis tool and combine the results. This is a great idea-when it comes to testing, two heads are almost always better than one-but unfortunately it still doesn't solve our problem. Remember the relative weaknesses of each tool: Black-box tools can have trouble finding unlinked pages and unusual execution paths in the site, while white-box tools can have trouble with pages that use third-party controls or rely heavily on JavaScript. What if an application contained a page with both of these troublesome attributes, like
About the author
Bryan Sullivan is a security program manager on the Security Development Lifecycle (SDL) team at Microsoft. He is a frequent speaker at industry events, including Black Hat, BlueHat, and RSA Conference. Bryan is also a published author on Web application security topics. His first book, Ajax Security was published by Addison-Wesley in 2007.
