Fuzzing Through the Side Door

Since the fuzzing automation was essentially pummeling each public order entry field with different kinds of data, we found different kinds of errors. Fuzzer payloads might exploit input overflow tests, and, on some fields, large values caused errors anywhere from the database on up to the code processing the message. Other fuzzer payloads exploit entering data of the wrong type, and that could cause the database to fail while trying to insert the incorrect data into a field. In other cases, the code that processes messages might fail when trying to process a character from a character set it wasn’t expecting. Any number of libraries in between might fail similarly.

We discovered inconsistencies in the way code handled error conditions for different inputs in an order. The order ID field might restrict data of the wrong type or data that is too long at the messaging layer. The price field might not restrict those data types or lengths at all. In other cases, error handling might break down after processing a large number of orders. We also discovered memory leaks and other problems that were revealed by processing a large number of orders over a period of days. We just didn’t see that with manual testing or automated functional testing in the past. Those efforts didn’t generate enough data or conditions on the server that were closer to what you would see in production.

This approach to test automation is a bit more technical than some testers might be used to, but it can be highly effective. Using the well-known approaches of mutation testing, high-volume test automation, and fuzzing against an order-entry system proved to be a powerful combination. If you have the technical resources to be able to try this sort of thing out on your system, I encourage you to do so. A word of warning, though: Do not try this on public systems or systems you don’t have permission to run a fuzzer against. Fuzzing in this manner can quickly bring a test system down. If you’ve been asked to test a messaging system, propose looking at this kind of approach, and happy fuzzing!

Jonathan would like to thank Yiannis Pavlosoglou of the JBroFuzz team for his help and support.

References

• Fuzz Testing of Application Reliability
• JBroFuzz Tutorial