As more IT organizations become comfortable with the public cloud and look to leverage it for critical applications, performance testing engineers must be prepared to test effectively in the public cloud environment. As a performance testing engineer, you must understand that performance testing in the public cloud requires a detailed understanding of the cloud infrastructure and the effects that infrastructure can have on performance testing results. You should be aware of important infrastructure considerations that can significantly affect your load testing efforts while setting up your cloud-based load testing environment and interpreting the results of your efforts.
The more that you know about cloud-based infrastructure environment variables, the more confident you will become with the performance metrics that your load testing tool produces and the more information you will have to apply to the interpretation of the load testing tool results. At the core of a typical instance of a public, cloud-based environment sits a physical server or servers, of which the physical characteristics (server class, CPU, memory speed, disk speed, etc.) may or may not be known to you. Some public cloud vendors are not straightforward with the hardware configuration that your instance runs on. Do not be afraid to contact the cloud vendor for this information.
Typically, you or the administrator will know the instance size or type. For example, on the cloud-based Amazon Web Services (AWS) platform, you have the option of several sizes. Some of the hardware-specific details for each size are published in the documentation, providing transparency to the physical characteristics of the machine that your instance sits upon. Just as with a non-cloud-based load testing environment, you need to be fully aware of the instance type and related characteristics that are targeted for your production environment. You must run the load tests with the appropriately matched instance type that is targeted for production. Not doing so is akin to testing one class of server in a typical non-cloud load environment while knowing that your production environment will have a completely different class of server.
The next critical component in the public cloud architecture stack is the virtualization software, which sits on top of the physical machine. You likely will not be aware of which virtualization product is in use, but the specific product is not important. What is important is that the cloud vendor should have only one product across its entire offering, Check with your vendor prior to making that assumption.
Virtualization software introduces overhead that you would not have in a typical on-premises load testing environment. Effectively, this will reduce the user-facing processing power and throughput of that server (and therefore the number of users that it can handle simultaneously) by a marginal amount. Some studies suggest the average reduction is about 10-15 percent, and my personal testing experience has supported that number. A key variable working in your favor is that the virtualization software is supported by the underlying hardware, and the two work together to make the virtualization experience as efficient as possible.
The next important aspect you need to consider is that you may not know how many other users you share your server with. The whole purpose of the previously described virtualization layer is to allow the cloud vendor to “rent” space to several customers on a physical server. Obviously, this is a critical variable that you need to be aware of. Your challenge is to decode the physical infrastructure resources that others may be sharing with you. This data will be highly dependent on your particular vendor’s internal infrastructure, so you are going to need to