“Fuzzing” is a software-testing technique that involves using tool-generated test data to try to cause failures in a system. It’s a brute-force approach—a fuzzing tool generates random data of a certain type over and over, and that data is inserted somewhere within a software program. It is simple, yet incredibly powerful. Weird data types ending up where the program doesn’t expect them to be is a common source of errors. Fuzzing can help uncover security weaknesses in a system, and that’s where it gets most of its exposure.
I was working on a web-based system that processed customer orders. One of the problems in the test approach was that in production, thousands of orders could be processed over an hour, but in test, there might be a dozen orders processed per day. It could take a while for each order to be entered manually through the system, and it didn’t take much less time to have an automated tool process that order. You could also generate orders by running transactions against the back-end database to quickly create a lot of test data, but that wasn’t really testing the order system. I wanted to get more production-like volume in a short period of time. One program interface caught my attention: Most of the orders in the production system were processed through a web services interface. That means that third-party systems created orders that were sent to a non-GUI interface in this system using an XML format. I can run automated tests much more quickly if I don’t need to use the GUI. Furthermore, if there is a published, stable interface already in use, I can exploit that for testing—enter test automation through the “side-door.” I got this term from consulting developer and tester Jennitta Andrea. The “front door” is the GUI, the “back door” is the database, and the “side door” is other testable interfaces in between, like messaging APIs.
There are a lot of fuzzing tools out there, but many of them seem to work by trying to cause a public interface such as a login page or URL to fail. Others analyze your code and use fuzz payloads to cause problems at a very low level. Still others are powerful, sophisticated tools to run throughout your software system. I wanted to do something different. Both mutation testing and high-volume test automation are well-known, powerful techniques for finding problems in systems. I wanted to run a high volume of tests in this online ordering system to see how it could handle a large number of transactions over hours or days. I also wanted to see how the system could handle problems in various parts of an individual order. I decided to use a fuzzing library to generate the brute-force data, insert that data into parts of an order, and run this test thousands of times. The fuzz data would mutate the order messages to simulate problem orders that were seen in production. The test program would generate messages quickly and run for a long period of time to simulate production traffic.