Testing Your Worth

In this uncertain economy, testers are losing their jobs, and entire test groups are being laid off. I don't think testers as a whole are out of the job market, but I think there's a new trend in the testing world toward testers who can provide maximum value, either in reading and writing code, writing automated tests, being a subject-matter expert, or being an industry expert. 

Why do I think this is a trend? Many managers I know are working to justify their test group's (and their own) existence. Test managers need to explain the benefits of their current testing strategy, whether it's to senior software managers or to finance people. Managers may or may not know much about software, but they know when the services they're paying for (testing) don't seem to be providing enough bang for the corporate buck.

Amanda, a test manager, wanted to replace a manual black box tester, Ginger, who'd moved to another part of the country. Amanda's VP, Jim, asked what kind of a person Amanda was looking for. When Amanda said she wanted to replace Ginger with another manual black box tester, Jim asked what the cost-benefit analysis was for that kind of a tester versus other kinds of testers. Amanda was initially at a loss for the analysis. She then decided she could look at what people did, and compare their activities to the numbers and kinds of problems the testers found. (Amanda didn't need to compare salaries; salaries were close enough that they were not a consideration.)

Amanda came up with this table for the most recent release of the software, organized by who found the highest total percentage of defects:

Table 1: Total defects found by tester
There's not enough information here to make an informed decision about the value of each tester, but if you look at raw defect counts, the test developers look like they've found more overall defects.

Unfortunately, overall defect-find rate per tester is a particularly bad metric. We can all inflate defect counts with not-very-interesting defects. And what about Franny, our exploratory tester who only found 5 percent of the overall defects, but found a whopping 38 percent of the high-severity defects? We not only want to keep Franny, we might consider getting more Frannys in the group. So let's look at the table again, this time sorted by severity of defects found.

Table 2: Total high-severity defects found by tester
 Amanda used this data to say: "Why are we finding 87 high-severity defects (more than 10 percent of our total defects)