This paper talks about testing for reliability of server software, and highlights the challenges in planning, automation, execution, collection, and interpretation of the data for determining reliability as well as some practical challenges/difficulties faced. It also talks about the relationship of reliability testing to other areas of testing such as performance and functionality.
In the current competitive Internet scenario, as the software products are expected to offer more and more complex functionality, expectations on higher reliability too goes hand-in-hand. Further, this is more critical in case of server software products. Needless to say, reliability of a server is a key factor both to the software developer and the customer, to make decisions, and hence a key factor for quality. The server software is expected to run without any failures for a continued period of time to support present business needs. In the e-commerce scenario where the server is expected to run round the clock catering to the customers throughout the world, spanning different time zones, there is no chance to reload or restart the server application. These requirements make the reliability testing more complex and critical.
This paper talks about "Software Reliability", the case of testing for reliability of server software, and highlight the challenges in planning, automation, execution, collection and interpretation of the data for determining reliability as well as some practical challenges/difficulties faced. It also talks about the relationship of reliability testing to other areas of testing such as performance and functionality.
Software reliability can be defined as the degree to which there is failure-free functioning of the software for a specified period of time or specified number of iterations/transactions, in a specific environment. Software reliability is an attribute and key factor in determining software quality.
Reliability of software describes the customer's expectation of satisfactory functioning of the software in terms that are meaningful to the customer. This description may be significantly different from the theoretical definition. The customer must tell the circumstances under which they will 'trust' the system that will be built.
The following are the key factors for reliability of the software:
- No or very few errors from repeated transactions
- Zero downtime
- Optimum utilization of hardware resources
- Less utilization of network resources
- Consistent performance and response time of the product for the repeated transactions for a specified time duration
In a typical Client-Server scenario where the server products are assumed to be up and running all the time, it may be ensured performance of serving numerous clients without any considerable response delays by consuming optimum amount of resources of the system such as memory, CPU...etc.
The power and efficiency of the hardware resources have been increasing tremendously over a period of time and the software should be designed to utilize all the resources optimally to provide maximum returns for the investment of the customer. At the same time the server application under test, should yield these resources to other critical applications that may run on the same hardware as necessary.
In today's Internet scenario the server software performing reliably on a system is not enough, the implications of the software on the network are also equally important. The software should be designed to work utilizing less of network resources, as there are many customers who still have low network bandwidth and may be running other applications on the same network. So, a software reliability testing should also include the impact of the software on the network.
What is important is, the customer needs or expectations are described in a quantifiable manner using the customer's terminology.
Some examples of reliability are:
The system will be considered sufficiently reliable if:
- 10 or fewer errors result from 1,00,000 repeated transactions
- It supports X transactions per second throughout the specified period of time (i.e., there is no degradation of performance because of reliability)
- The software optimizes hardware, software and net resources for repeated transactions
- There are no side