For any project, the big question is "is the software ready to release yet?" Or more specifically, "When is the development and testing part of the project done?" You can't know if you’re ready to release unless you know what “done” means.
Why Release Criteria?
It’s certainly possible to release before you are done. In the mid-1980s, I worked as a test lead on a major operating system release that included a brand-new, window-based GUI. At the time, most operating systems had command line interfaces, so our customers couldn’t tell us what they wanted in a graphical interface. For many months we worked on it until management decided it was time to release. When senior management mandated the release date, we didn't have an easy way of knowing if the software would be good enough for our customers by the desired date. Despite many internal objections (from developers and testers), the system was released...and it was a disaster.
Our customers hated the way the GUI worked and wanted to get rid of it altogether. We hastily planned an incremental release, fixing the obvious defects. Our customers liked the initial point release and were thrilled about the next point release, when everything in the system finally worked in a way they could use. Now, we were essentially done.
When you don't know what "done" means, you may also go on testing after the software is ready enough. A few years ago, one software organization had already released a successful client/server product and needed to transition to the Web quickly. During the Web transition, a few more features were added to the product and a number of defects were fixed. The project was on an aggressive schedule, and the VP was under serious pressure from the rest of the management team to release the product. For the last four weeks of the project, the VP interrogated everyone daily on the project team, asking if they were done yet.
At the project retrospective, the VP described what he thought was necessary for the release to be done. A senior developer said, "If I'd known that was all I had to do, I could have been done a month ago."
The VP looked furious.
This organization didn't have release criteria; they only had a release date. Because they had no way to assess how good each product component had to be, the developers and testers determined each component's doneness themselves.
Another release problem occurs when someone who doesn't actually know what's going on decides to release the software. Deb, a QA manager, found herself in a tough spot. Development had turned over part of the software, and her team had finished preliminary planning and exploratory testing. They had a short testing cycle and felt like they had too much to do in too little time. Deb called her technical leads into a meeting to assess risks and figure out how to best allocate their scarce resources to testing the product. During the meeting, one of the testers ran in and said, "Don't even bother. They shipped the release at noon."
Deb asked her VP why they'd already shipped the software. The VP said, "You weren't going to be able to tell me anything about the software in less than a month, and the VP of Sales convinced our CEO that we had to ship. I was vetoed."
Fortunately, Deb saw this as a potential opportunity, and said, “Okay, let me hire more testers." The VP said, "If I give you more people, can we release software faster?” Deb said, “I don't know if we'll be able to release it faster, but we’ll know earlier if it's good enough to release. We can make release decisions without you being vetoed."
Maybe you've run into problems like the ones I've described. Maybe, as the test lead or test manager, you have found yourself in the role of gatekeeper, personally responsible for the content and quality of the release. Or perhaps your organization makes the release decision based on a gut feeling, without the data to back it up.
How do you know when testing is done? How do you know when the product is ready to release? Does the decision to release seem similar to deals done in dark, smoke-filled rooms, or as if there are rules of the game that you don't know? Does the release decision sometimes seem completely arbitrary? If you're finding it difficult to make rational decisions about when to release the software, developing and using release criteria can help.
Release criteria are objective measurements of the critical attributes of the product or project. Listing and referring to the criteria allow you to know whether the product is ready to release. They can help you make rational decisions about when to ship the software.
Developing Release Criteria
When you develop release criteria, first decide what’s critically important to this specific project—Why is the company doing this project? Why would the customers buy this product? If you don't know what success means for this project, start by defining it. Then quantify how you would recognize success during the project, and gain the agreement from the project and senior management teams on the criteria you generated that will measure the doneness of the release.
1. Define Success
Not every project has a clear definition of success. If you're in the fuzzy front end of a project, you may find it easier to interview the project manager and other stakeholders first, before trying to define what's important to the project or drafting release criteria.
I define what success means by understanding:
You'll notice that success has nothing to do with the number of defects found, or the number of tests defined or run. Success is about what the customers will be able to do with the product when you're done with it.
If your project manager doesn't initiate projects this way, you can learn what success means by interviewing the project manager and other managers in the organization. If you find out at the beginning that not everyone agrees on a definition of success, then you won't be able to agree on what makes the software ready to release. It's better to discover any disconnect at the beginning of a project, before you plan your approach to testing.
If not everyone agrees on what success means, make sure the project manager knows. If you're the project manager, make sure your project sponsor knows. Then, get help reconnecting everyone's definition of success. If necessary, use a professional facilitator to help with this reconnection.
After her management’s preemptive release, Deb realized that defining success would help her plan which testing to do when, for future releases. Defining success would provide enough information needed to plan a product's release.
When Deb started trying to define success, she found her questions weren't always welcome. “Not everyone was happy that I wanted to pin down what success meant," she said. "The Sales VP was unhappy because it was clear that if we all knew what success was, he couldn't make the decision to release or not by himself…he wouldn't be able to blame Engineering for his salespeople's inability to sell the product."
Release criteria are not about shifting blame from one group to another; they are to provide some objective measurement about when a product is ready to release.
Release criteria can also help you build whole-product responsibility into the product release. Can the salespeople sell the product that meets these criteria? Can the support staff support the product? Can the trainers develop and deliver training? When I work with people to define what success looks like, they realize not only are they accountable for their part, but they're pointing me toward project success as well.
Once you know what success means, you can define what's most important for this project.
2. Learn What's Important for This Project
What’s critically important to this project is the combination of what the company needs and what the customers need. Customers don't buy products based on the number or lack of defects. They buy products because they solve some problem the customer needs solved. When you define what’s most important, don't stop at the number or type of defects.
The customer model isn’t just important to commercial software organizations. If you're a non-profit or an inhouse organization, a customer-oriented business imperative still applies to you. Non-profits still need revenue; they just don't distribute the excess profit back to shareholders. In-house organizations create products because their users think they can create a product more cheaply than an outside organization. An in-house service exists to save time and money for the rest of the company, not to specifically create revenue. Whether you sell products or work for an in-house organization, you're solving problems for your customers.
Sometimes the date is most important. Sometimes, it’s a particular feature or set of functionality. More often, it's a combination of schedule, features, and low-enough defects. It all depends on your customers and their expectations. (For more information on customer expectations, see the StickyNotes section for a link to my article "What Does Success Look Like?")
Rita worked at a startup company driven by cash flow. The company was not fully funded, so they had a tremendous incentive to ship product early in order to get enough revenue to continue the company’s existence. For the first three releases, the only release criterion was the date the release had to go to customers, so the company could legally recognize revenue. Once they made it past the first couple of years and were adequately funded, the company developed other criteria including defect counts, test progress, and the states of "code freeze" (the times when developers would stop working on the release).
Rita described it this way:
"When we were a startup, we just needed to keep our heads above water. Our initial customers wanted pretty much what we gave them, but were willing to work with us. However, we were all blown away with last year's release. All of a sudden, the customers cared about defects, more than they ever had before. My management demanded to know what kind of a QA group I was running, and I felt completely besieged. The only thing that saved my sanity was knowing that I'd checked with the entire project team and senior management in advance, to know that the release criteria we chose were what we needed. Unfortunately, we didn't realize how much demand we would have for this product, and we now realize we can't use only the date or some defect or test numbers to assess the state of the release. We need a much bigger picture to know when we're ready to release the software."
3. Draft Release Criteria
I find it useful to draft a set of strawman release criteria, so we have a vehicle to start the discussion. I try to create a balanced representation of time to market and what our customers want, along with defect, performance, and reliability levels, if that’s appropriate. I don’t have to guess all the criteria correctly the first time; I just need to have something we can discuss as a group.
Rita first drafted a set of release criteria as a starting place to discuss the release with the project manager and the rest of the project team. These were Rita’s draft criteria:
However, once she started talking to the project manager, Rita realized that her initial criteria were not exactly what the customers or the company needed. Yes, the customers were concerned with defects, but not to the same extent that Rita was. In fact, if a couple of the major customers were satisfied with the release, then chances were good that it was good enough for the rest of the customers.
Initially, Rita's project manager was surprised that she had tried to look at the whole release from the customers' perspective and come up with a balanced idea of what would make a complete release. He'd expected that Rita would be much more concerned with a traditional test manager's perspective on quality—low defects. But Rita knew from her previous projects at the company that low defects were just part of the story when making a release decision.
Rita, along with the project manager and the rest of the project team, revised the release criteria:
These criteria don't address everything for Rita's release, but they cover what's critically important for this release: ship date, good-enough software, and a specific feature tested and found working by two specific customers. Rita was disappointed that only 90% of the planned QA tests had to pass—she thought there was too much risk with such a low passing number. But, after hearing what everyone else said, she was willing to go along with the rest of the criteria because the release date was so critical. Rita was also concerned about removing the criterion about the cross-functional team evaluating the open defects at the end of the release. However, the product manager reassured Rita that he'd discussed the issues with the project manager, and that the project manager could speak for the marketing and support staff.
4. Make Release Criteria SMART
When you draft the release criteria, make sure they can be answered by anyone on the project team in the same way. I use the acronym SMART—Specific, Measurable, Attainable, Relevant, Trackable—to test that I have reasonable and objective criteria.
Each criterion should be specific for this product at this point in its lifecycle. When you make a criterion measurable, you’re ensuring that you can evaluate the software against the criteria. Release criteria are not the place for stretch goals, so make each criterion attainable. Make sure your criteria are relevant by evaluating this product against what the customer wants and what management wants. When you make criteria trackable, you can evaluate the state of the criteria during the project, not just during the last week.
"No high-priority open defects" is an objective and measurable criterion. You can look at the open defects and verify that none of them deserves to be high-priority. "All open defects reviewed by cross-functional team" is another example of an objective and measurable criterion. Either the crossfunctional team reviewed the open defects or they didn't.
"Fast performance" is an ambiguous criterion. To change this into an objective and measurable criterion, make it something like this: "Performance scenario A (corresponding to use case A) to complete in under ten seconds." Here, you name the specific scenario, so people can refer to it, and give the performance criterion to verify that you met that performance.
5. Gain Consensus on Release Criteria
Now that you have reasonable release criteria, it's time to gain consensus on what you've developed. If people react negatively to your draft criteria ("No, we couldn't possibly do that"), then learn why they are concerned. Generating release criteria reveals assumptions and fears about the project and the product. 've found these questions helpful when generating or gaining consensus on release criteria:
These questions generally help the entire project team stay focused on what's needed for this release.
As the test lead, I've gained consensus on release criteria in either of two ways: 1) drafting the release criteria, discussing them, and coming to an agreement at a project team meeting, or 2) drafting them with the project manager and discussing them at a project team meeting. I prefer generating release criteria with the entire project team, because then the team (not just the project manager or me) owns the release criteria. However, if you're working on a large project, or have never used release criteria before and want people to understand what they're working toward, you may be better off generating release criteria with the project manager in advance.
No matter how you gain consensus, make sure you evaluate the release criteria and the product's readiness for release—not how the developers and tester achieved the release criteria.
Using Release Criteria
I use release criteria to evaluate the state of the product throughout the duration of the project. I ask my project managers to use the release criteria as part of the project team meeting, so that the entire team is able to assess the project's state of doneness during the entire project. When we get to the formal system test part of the project, I use the criteria as part of the testing status report.
I also find that release criteria are an early warning sign that you're not going to make the release. Manny was a project manager, halfway through a six month project. Manny looked at the progress to date and was concerned that the team was not going to make the ship date. He decided to make the release more real to the project team by generating release criteria with them.
Manny then used the release criteria each week during the project team meeting, to verify the project was making progress. It worked for a couple of weeks, until one week during the release criteria evaluation, one of the engineers said, "I'm not going to make it. I've tried and tried, and I’m just not going to make that criterion for our ship date." Manny said, "OK, I need to go back to management and see what we need to do. Before I do that, does anyone else think they’re going to have problems meeting any of the release criteria?" Another engineer said, "I can’t get performance that good between now and the time we have to release. When we discussed the release criteria, I thought it was possible, but now I realize it's not going to happen."
Manny was able to get a much earlier heads-up about his project's progress with release criteria. For this team, realizing they weren't going to meet the release criteria two-thirds of the way through the project instead of at the end was a relief. The project manager knew what the project reality was and could work with management to see which tradeoffs made the most sense. In this case, the project manager was able to renegotiate the release date, so that the product could meet the release criteria.
If all goes well, you'll evaluate the release criteria as you go, and you'll meet the criteria when the project is supposed to end. However, projects don't always go well, and you won't always be able to meet the release criteria. When that happens, make sure you're honest about what's happening.
For me, release criteria are either met or not met. We aren't partway to meeting a criterion—we haven't met it. I find that this binary approach helps me more when I'm discussing the state of the software with senior management. If I say we're partway there, they hear that we're done. If I say we haven't met the criterion, they hear we're still working.
I generally don't change the release criteria during a project. But there are at least two circumstances when it's reasonable to change release criteria: 1) when we learn more about what "done" means for this project, and 2) if we realize we can't meet all the release criteria by the desired release date.
If you learn more about what "done" means, then ask yourself these questions about the release criteria: Must we meet this criterion? What happens if we don't meet the criterion?
If you're working on a project and can’t meet the release criteria, say so. Have your management say something like: "We thought these other criteria were important, but we realize now that the date is even more important than we thought. We're going to release the product, even though we haven't met all the release criteria." If this is true, you can have them add, "We're going to determine what prevented us from meeting our criteria this time, and create the next project so that we don’t miss our release criteria."
It's possible to create a software product your customers will want to use, and to know when you're done crecreating it. You don't have to play the "when-can-we-release-the-product" game. When you use release criteria to know when a project is done, you have taken potentially hidden decisions and made them public and clear. Make your release criteria objective and measurable, so everyone on the project knows what they're working toward. Use the criteria as you progress through the project and up to the final release. Then you can say, "Release it!" with pride.