Purchasing or leasing a tool or a combination of the two is the most common approach to acquiring load-generation tools. For smaller applications or systems, the purchase of a tool can be effective. However, as the number of users to be simulated increases, the additional cost of tool licenses increases-as well as the hardware and infrastructure required to support the load generator.
The final possibility is to outsource the performance testing to an ASP specializing in performance testing. The advantages are that these vendors typically have access to the necessary infrastructure and tools. The disadvantage is that they still have to obtain the same information as an in-house performance team. To properly leverage an ASP, all the initial work on defining goals, load profiles, etc. noted in previous sections should be completed before a contract is signed.
Regardless of the method used, utilizing the maximum number of virtual clients or events that a performance testing tool license allows may not actually generate the greatest load. Depending on the capacity of the load-generating machine, the load generator may be wasting too much time juggling the various virtual clients or events in and out of memory instead of actually processing their requests. At some point, as more virtual events or users are added, they may, in fact, slow down the total number of requests that are being submitted, as shown in figure 1.
Before conducting any server measurements, it is important to calibrate the load generators to determine what percentage of degradation will be attributable to the load generators' being stressed and at what point the load generators max out.
Since each virtual client requires resources, when selecting a performance testing tool, find out what the resource requirements are for each virtual user. For example, if each virtual client needs 2 MB of RAM and you have four test PCs, each with 64 MB of available RAM (4 x 64 / 2 = 128), a maximum of 128 virtual clients may not be enough if you are expecting several thousand concurrent users.
Factors affecting the amount of resources each virtual client needs include:
- Size of the test script being executed
- Virtual storage used by the script
- Resources used such as threads, etc.
Creating the correct distribution of user activities in the tool is also critical to proper load generation and is critical to avoiding problems that can over-enhance or degrade performance. Factors that can affect load distribution include:
- How the load mix is defined to the tool
- How the tool handles think times
- How the tool starts the load
- How the tool increments the load
- How the tool records and manages session ids and cookies
- How the tool handles re-execution of the scripts and virtual clients
- How the repetitions are treated (new users or the same user?)
- This can affect concurrent user counts, etc.
Some performance testing tools allow multiple threads per virtual client, thereby making the load more realistic and increasing the load on the server. Another feature that some performance testing tools offer is the ability to run each virtual client in its own process. One virtual client per process can be extremely demanding upon the load generators, but may make the load more realistic. For larger loads, it becomes particularly convenient if the load generators can all be coordinated via a single console-a feature provided by most commercial tools.
Where and how the load is applied to the network also is a critical decision. If you have a network with security features and you place the load internal to the network, you are leaving out