Read about an approach used by nFocus to improve and maintain reliability through and beyond the project development lifecycle.
There is incessant pressure to supply reliable systems at reduced cost and increased profitability. Object-oriented development has also made it easier to develop components iteratively in isolation, for example database access procedures can be developed independently from the application or the user interface.
Traditional testing methods attempt to improve rel iabi l i ty by debugging the system using a combination of reviews and unit test ing fol lowed by independent system and integration testing.
The later stages of verification and validat ion test ing (acceptance and pi lot programmes) should continue to improve reliability until the system is deemed reliable enough to be implemented. This applies whatever type of development lifecycle–waterfall, RAD etc.–is used.
How do we measure reliability?
There are many indicators of reliability. Some are listed below but the supplier, client or end-user should determine what is meaningful for the system under development. A good start can often be made by considering what criteria are used to measure the reliability of an existing system. One common thread throughout is the need for some ratio or reliability index based on problems found, so problem or incident collection and management is a key requirement for reliability measurement . My advice is to f ind as many measures as possible, plot them over time and look for a trend. These measurements could well be used as part of the acceptance criteria.
Some potential measurements to collect over fixed time periods:
- Tests executed per problem;
- Regression errors found;
- Time testing per problem;
- Database accesses per problem;
- Processor cycles used per problem.
Use these indicators with caution; we are looking for trends and small samples can give misleading results.
What often happens
During system development the schedule is affected by changes; examples include user requirements, technology, design, user interface, interaction with other systems and changes to deadlines.
This results in pressure from project personnel to compromise the agreed test process. Often test stages are overlapped, testing is deferred into later test stages (which were never designed for that purpose) and worst of all some testing is just abandoned. Everyone becomes a 'test expert' as time and budget pressures take over, resulting in poor quality of testing and poor reliability. The end result is often an unreliable system that is incomplete, doesn't meet specification, doesn’t perform to expectations, has poor security and is difficult to manage and maintain.
As testers we have to accept that changes will occur whether we like it or not. Trying to impose strict discipline in the early stages of development only alienates the rest of the project disciplines, so why not accept the inevitable and, as a team, plan for this change? However there will be some rigour required in the process if we are to maintain and improve reliability, particularly toward the later stages of testing where any change can have serious consequences.
How could we solve these problems?
If we accept that this situation is going to arise then we should have a test strategy that assumes that change is inevitable. It should define a change process that allows the plan to be flexible enough to cope with unforeseen changes that could adversely affect reliability.
The key to reliability improvement lies in the code and unit test stage of development.
What if we could improve by continually testing and retesting the system components as they are developed?
The effect of change could be seen and dealt with immediately, and errors could be fixed early, reducing the risk of poor reliability resulting from fixing the backlog of error late in the project cycle. Regression failures would immediately