Software Longevity Testing

How long can your software operate continuously? Is your answer measured in hours, days, months, or years? More to the point, how long do you let the software run during your test cycle?

Longevity testing is an often-overlooked part of software robustness testing for many projects—from networking routers to Web servers and even planetary rovers. Many software applications are intended to run indefinitely, in an always-on operating environment. And yet, few test plans include more than a memory leak test case or let the software run for longer than a few days.

Longevity testing (also known as soak testing or endurance testing) looks for software problems that appear only after an extended operational time. These problems fall into two broad categories: problems due to the passing of time and problems due to cumulative usage.

The Passing of Time

Accuracy Drift
The passing of time can cause software accuracy to drift. A tragic example is the Patriot missile failure in February 1991. The software uptime was stored as an integer and converted to a 24-bit floating-point number, causing the software to have noticeable inaccuracy in the target-velocity tracking after only eight hours of operation and to be inaccurate to the point of failing to track the target after only twenty hours of continuous operation.

The field-mobile Patriot missile launcher was intended to operate for only a few hours at a time. However, nothing prevented the system from being operated for much longer periods. At the time of the failure, the system had been running continuously for one hundred hours. [1]

Don’t rely on the software users to reboot their systems periodically. In fact, assume the opposite—that most users will operate the software continuously with never a reboot. This continuous usage goes against the practice in most software test labs, where the software is restarted every few days with a new build if not rebooted hourly between test suites.

Timeouts, Expirations, and Schedulers
Most software is populated with timers and schedulers, which start or stop activities minutes, hours, or days into the future. Sufficient time must be allowed during testing to exercise these timers. Timeouts and expirations to be tested commonly include user session timeouts, firewall session timeouts, ARP tables and MAC address tables aging out, routing tables expiring, SIP registrations renewing, DHCP leases renewing, DNS mappings expiring [2], and age-based passwords expiring. Software scheduler testing should include automatic maintenance activities such as daily virus scans, nightly database resynchronizations, weekly database backups, and monthly software patches or upgrades.

A software license is a type of software scheduler that commonly expires after six or twelve months. Forgotten software licenses tend to expire at the worst possible time. Even worse, a bug in the licensing software can have dramatic consequences, as demonstrated by VMware virtualized servers, which were prevented from starting up after August 12, 2008. [3] Gentle warnings should be given weeks in advance of a software license’s expiring. When the system boots up with an expired license, the user should be presented with a clear path to update the license key. Software applications that depend on the software license need to handle the expiration gracefully. Avoid a total crippling of all applications when just one application has a dependency on an expired license.

Some activities are postponed when the software is very busy, and other activities are only activated when the software is sufficiently idle. Longevity testing should include tests of long busy periods (soak testing) as well as long idle periods (quiescent testing).

Soak Testing: Testing the system under heavy load for an extended period is known as soak testing or