Changing the Ground

I suggested reframing how they thought about the project. If you think of an infrastructure upgrade as primarily a testing project, then it would make sense to adopt a whole-team approach. Why not explore as a whole team the most effective and efficient ways to test throughout the project?

Even if they decided that the test team should document the test strategy and coordinate test development, testing shouldn't necessarily assume ownership of the tests. With infrastructure upgrades, there will often be other specialists better placed to design tests and find problems. So I suggested the project begin by assembling a cross-functional team of techies and testers to have overall responsibility for deciding the test strategy and testing the upgrades. It would need at least one representative from each technical team involved in any one of the upgrades: product and technical architecture, the DBA team, etc.

The cross-functional team could take a risk-based approach to the central test design based on answers to these questions:

• What is most likely to go wrong from a technical point of view with each upgrade?
• What kinds of problems would have unacceptable impacts for their business?
• How and where could those problems manifest? Which techniques would they need to catch each problem? Which data, and how much? Where could analysis or knowledge of the code preclude a need to test?
• Where is the earliest point in the upgrade process that someone on the cross-functional team could detect and fix each kind of problem they had identified?
• For which problems would they want extra insurance from additional testing later in the process?

The team could then design tests targeting each of the risks and decide who would be best placed to do each one. They could identify where it would be useful to pair testers with DBAs or programmers.

In addition, I suggested they supplement their central test design with:

• Sanity checks that would expose catastrophic problems quickly in each major functional area
• Some key scenarios covering things that they didn't think could happen, but which would critically harm their business stakeholders if they did

To find out the technical risks, I recommended two principal avenues. First, research each upgrade, and if possible, combinations of upgrades:

• Ask the vendors.
• Look on the web at technical forums and other resources. Someone out in the world has done what you're going to do and found problems with it.

Then, conduct a workshop with the merchandising system maintenance team and the technical experts responsible for each upgrade. Ask what kinds of errors this type of upgrade might introduce:

• For any processing modes that could change, which application technical components depend on the existing modes and could break or behave unpredictably with the upgrade?
• How and where could each change or error type manifest?

For the business impacts (problems that would hurt their users), we looked at another set of questions to ask and did a pilot workshop with a group of stakeholders.

Although this was not an agile shop, the project team embraced the whole-team approach because it targeted the upgrade risks directly and engaged each specialist’s expertise where it would be most effective in the analysis and testing. There was no need for a lengthy “test phase.”

It turned out to be a very successful strategy. The team found and fixed several problems in their applications that were illuminated by the upgrades before they went live, and they spent far less time and money than they would have with the comprehensive black-box regression test they'd originally intended.