to Excel to generate graphs. Be wary of reporting early results: If the results from early testing appear to be too good to be true, investigate before reporting to ensure they are accurate.
All the relevant data sources need to be brought together for reporting:
- Data from the load tool
- Data from all relevant monitors
- Data from any probing tool
- This is why clocks must by synchronized
Ideally, run the final set of test loads for each test type several times before reporting "final" results. If the test was set up correctly, there will be some variance in each execution. This is exactly what we would expect if the load generator is properly executing the operational profile we defined.
No two runs of a test type should ever be identical. One of the primary goals of a performance test is to model or mimic the real world. The real world is "chaos"-every day, every hour, every session is different each time it occurs. Our load test needs to create this same type of chaos. Therefore, we run the final tests three to five times and average the results. This helps ensures consistency and accuracy in the test results.
I have found that the first time you attempt to build a software performance test it can take anywhere from six to sixteen weeks. This may seem like a long time, but we are dealing with calendar time, not 24/7, full-time involvement. There are natural delays between activities that make this happen. For example, when creating an operational profile it may take several days for the groups providing the data for analysis to get the data together. These delays occur at several points in the process and are normal.
There are many details and issues this series of articles did not address. The goal was to provide an understanding of the basic problems and complexities involved in building a successful software performance test.
Read "Understanding Software Performance Testing" Part 1 of 4 .
Read "Understanding Software Performance Testing" Part 2 of 4 .
Read "Understanding Software Performance Testing" Part 3 of 4 .
Read "Understanding Software Performance Testing" Part 4 of 4 .