Testing COTS: When, How, and How Much?


Testing processes and practices are well defined and generally understood for internally developed applications, but what about those that are licensed from third parties? Granted, the vendor has responsibility for testing its own products, but the possibility of the software failing still exists and can be costly, even devastating; blaming others offers little consolation. If you rely on a commercial off-the-shelf (COTS) application, where does your trust in the vendor end?

While there are no hard and fast answers to how much trust you should instill in commercial, off-the-shelf applications, there are some basic guidelines that can help. Start by asking yourself these questions: 

  1. What is the risk that the application has defects?
  2. What is the risk that your use of the application could cause errors?
  3. What is the potential cost of a defect or an error?
  4. If you do test, what type of testing should you do?

Depending on the answers, you may find that your COTS application-testing needs and approach will vary widely.

Application Risk
The risk inherent in the application itself is a function of its scope of functionality, its breadth of users, and its maturity. For example, most would agree that mass-market utilities such as word processors, spreadsheet programs, slide presentation programs, and graphics tools pose little inherent risk. Their scope is narrow, they have millions of users, and most are highly mature.

Then there are more specialized applications, such as accounting systems, that are still somewhat narrow in scope but whose maturity and customer base may vary widely. Consider that there are mass-market accounting systems, niche products that address smaller segments such as non-profit accounting, and even custom solutions for highly differentiated customers. These products may also have wide variations in maturity. A new product for a small market with few customers poses inherent risk.

And then there are massive applications, such as enterprise resource planning systems (ERP), that encompass multiple departments across many industries and support complex, critical business processes. These have a wide scope of functionality, may have many users, and may or may not be mature. Due to their scope and complexity, there is inherent risk in these systems. These risks may be greater for newer products with fewer customers or for customers in a new industry or using a new component.

Usage Risk
The way you use an application can also change the picture. Consider a spreadsheet used by a construction company to manage multimillion-dollar bids. In this case, the software itself may have low inherent risk, but if the template contains complex formulas and is tweaked often, the risk of errors rises.

Many desktop utilities also can be programmatically integrated with other software. For example, an internal stock trading application might integrate a spreadsheet for its graphing capabilities. In this case, the risk of the spreadsheet malfunctioning might be low, but the potential that the integration has errors is higher. The wrong data might be mapped to the formulas, for example.

The usage of enterprise applications is usually controlled by configuration settings and master data. However, the rules that govern a business process can vary significantly from one company to another, and the sheer complexity of these applications with their multiple points of integration raises the risk of error dramatically.

In other cases the user might actually modify an application to accommodate unique requirements. Clearly this type of usage poses the greatest risk, regardless of the stability of the original state of the software.


User Comments

Annica Odina's picture
Annica Odina

Important to find out off what shelf the product came... There doesn't appear to be a 'legal' definition of what is actually 'off-the-shelf' and recently I tested a product that the vendor considered 'off-the-shelf' because they were done developing it but it turned out that it was actually not in commercial use! After 2 years of unsuccessful testing the customer (Government Agency) finally gave up and started looking for a truly off-the-shelf product that came with references from existing, commercial users.

August 8, 2012 - 3:16am
Lee Calladine's picture

I think this article provides a good insight into the test approach for COTS deliverables. In my experience I have regularly worked with a consultant from supplier to ensure that areas of risk are mitigated prior to the system even being purchased, network load capability, performance, functionality and then generally the test approach has only been if there are any integration between the COTS and legacy systems. In-house additional development is obviously tested but there is a general understanding that much of the product is already functionally tested and proved to a level of functional and non-functional acceptance and this is risk that is mitigated as part of the COTS purchase. Retesting an entire COTS system is a pure waste of effort and money.

September 7, 2016 - 6:39am

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.