probably will, discuss the outliers as a team. For instance, if I choose one and you choose three, we can discuss why you believe the depth of testing was moderate and why I believe it was shallow. After some discussion, we can vote again to see where we all are. If after some number of votes we still don't agree, we'll choose the most pessimistic assessment.
Use a similar approach to report quality, this time using your thumb-thumbs up for high quality, down for low, and sideways for moderate.
This quick, collaborative exercise also has the benefit of leaving everyone on the testing team with a common understanding of depth and quality.
Report the pair of assessments to the team as a one-to-five score for depth and high, medium, or low for your testing team's quality assessment. Will uses some number of stars for depth and a happy face, neutral face, or frown face for quality.
At Cycle-end Time, Report Quality of New Features
It's common in agile projects to stage a demo of the product at the end of a development cycle. During the demo, each new feature is demonstrated for the entire team to see. It's here that Will reports the depth of testing and quality ratings individually for each feature of the product.
In the past, the testing team may have complained that they didn't have enough time to test thoroughly. Now no complaint is necessary. The depth of testing is clearly reported. It's up to the team as a whole to decide what to do about it. After watching this practice in place, I've seen developers agree that they need to finish coding earlier in the sprint to give testers more time. Really, I'm not kidding.
Report Cumulative Quality on the Product as a Whole
As development continues to build onto the product, more and more features are added. Testers continue to work hard testing the new features that are added and the product as a whole. They're interested in how all these features fit together and in increasing their depth of testing across the product. To keep the team informed on their assessment of the quality of the product as they see it, the testers assess the entire product quality at the end of every cycle.
The product being built is big, so offering a single assessment for the entire product wouldn't be too valuable. Fortunately, this product is divided into a couple dozen functional areas. For each functional area, the test team offers a depth and quality assessment. The result is a "report card" of sorts composed of depth ratings and quality assessments. The goal of the entire team is to push both depth and quality as high as possible.
Keep Open Bug Counts Visible
The number of bugs found isn't always an assessment of quality. For example, low testing depth may return a low number of bugs, where high testing depth will likely return a higher number of bugs. Even with a low number of bugs, a subjective quality assessment could indicate low quality: "I didn't test much, but everything I tested was broken."
The team should know about all bugs and do their best to address them as quickly as they're found.
Will keeps the number of bugs found visible all the time by using a chart that looks similar to an agile burn chart. This two-dimensional chart uses number of known bugs for its vertical axis and time for its horizontal axis. Will publishes a new version of the chart every day and posts it on the wall in the team