Manual vs. Automated Code Review

The Fight for Superiority

parser errors. What if we wanted to perform fuzz testing manually? Could a human theoretically create millions of different malformed test files and test the application against them by hand? Sure. Would he die of exhaustion and/or boredom long before this? Definitely.

Another situation in which machines outperform people is in finding inadvertently exposed resources. Many sites have "/admin" directories, backup files, password files, or any of thousands of potentially sensitive resources that should never be viewable by the public. Through some misconfiguration or error on the part of the site's administrators, however, they are accessible. Again, could a security expert manually sit down at a browser and try thousands of different resource variations? Yes. Again, though, he would surely die of boredom first. More seriously, code reviewers rarely come cheap and paying experts to perform tasks that can easily be automated is just not a good use of time or money.

The use of both human reviewers and automated tools shouldn't be an either-or proposition. Only humans can find design-level issues such as poor identity-verification questions, while automated tools should be used for brute-force situations like fuzzing or directory enumeration where manual testing would be too tedious and expensive.

Author's Note: Thanks again to Vinnie Liu for sharing his personal experience in this area.

About the author

Bryan Sullivan's picture Bryan Sullivan

Bryan Sullivan is a security program manager on the Security Development Lifecycle (SDL) team at Microsoft. He is a frequent speaker at industry events, including Black Hat, BlueHat, and RSA Conference. Bryan is also a published author on Web application security topics. His first book, Ajax Security was published by Addison-Wesley in 2007.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, is the place to go for what is happening in software development and delivery.  Join the conversation now!