Release Criteria: Is This Software Done?

For any project, the big question is "is the software ready to release yet?" Or more specifically, "When is the development and testing part of the project done?" You can't know if you’re ready to release unless you know what “done” means.

Why Release Criteria?
It’s certainly possible to release before you are done. In the mid-1980s, I worked as a test lead on a major operating system release that included a brand-new, window-based GUI. At the time, most operating systems had command line interfaces, so our customers couldn’t tell us what they wanted in a graphical interface. For many months we worked on it until management decided it was time to release. When senior management mandated the release date, we didn't have an easy way of knowing if the software would be good enough for our customers by the desired date. Despite many internal objections (from developers and testers), the system was released...and it was a disaster.

Our customers hated the way the GUI worked and wanted to get rid of it altogether. We hastily planned an incremental release, fixing the obvious defects. Our customers liked the initial point release and were thrilled about the next point release, when everything in the system finally worked in a way they could use. Now, we were essentially done.

When you don't know what "done" means, you may also go on testing after the software is ready enough. A few years ago, one software organization had already released a successful client/server product and needed to transition to the Web quickly. During the Web transition, a few more features were added to the product and a number of defects were fixed. The project was on an aggressive schedule, and the VP was under serious pressure from the rest of the management team to release the product. For the last four weeks of the project, the VP interrogated everyone daily on the project team, asking if they were done yet.

At the project retrospective, the VP described what he thought was necessary for the release to be done. A senior developer said, "If I'd known that was all I had to do, I could have been done a month ago."

The VP looked furious.

This organization didn't have release criteria; they only had a release date. Because they had no way to assess how good each product component had to be, the developers and testers determined each component's doneness themselves.

Another release problem occurs when someone who doesn't actually know what's going on decides to release the software. Deb, a QA manager, found herself in a tough spot. Development had turned over part of the software, and her team had finished preliminary planning and exploratory testing. They had a short testing cycle and felt like they had too much to do in too little time. Deb called her technical leads into a meeting to assess risks and figure out how to best allocate their scarce resources to testing the product. During the meeting, one of the testers ran in and said, "Don't even bother. They shipped the release at noon."

Deb asked her VP why they'd already shipped the software. The VP said, "You weren't going to be able to tell me anything about the software in less than a month, and the VP of Sales convinced our CEO that we had to ship. I was vetoed."

Fortunately, Deb saw this as a potential opportunity, and said, “Okay, let me hire more testers." The VP said, "If I give you more people, can we release software faster?” Deb said, “I don't know if we'll