How Did I Miss That Bug? Managing Cognitive Bias in Testing: An Interview with Gerie Owen and Peter Varhol

[interview]

GO: Representative bias is similar to that. It's your ability or the way that you judge things based on other things. If you judge something that tastes like chicken, well, it may not be chicken. But this can happen in testing, and that "Well, it's always worked that way in the past, so it's still got to work this way." You can sometimes miss regression defects because you say, "Well, I've already tested with a new order, and I've tested with an order in process. Why bother to test with a completed order?"

Then curse of knowledge, that's when we become so knowledgeable about something that we fail to look at it from the perspective of somebody new to whatever the topic is. Of course, in testing, that would be the application. If you test the same application over and over again, you're going to get to know it so well that you're not going to necessarily approach it as a new user. A lot of times usability defects can be missed because of the curse of knowledge.

PV: That also happens to domain experts where they might think, "Well, we've always done it this way. So that's all you should care."

CP: OK. It seems like there's not having the right set of eyes on it, have a new set of eyes on it, having tunnel vision, seeing what you want to see. But are there any scenarios where those biases, or any other biases, actually help with testing?

PV: I can't think of any, to tell you the truth.

CP: OK.

PV: The important thing is to be able to catch yourself, because biases mean that we're looking at one specific thing to the exclusion of others. So we have one form of view to the exclusion of others. Gerie, we're welcome to disagree. I just don't see any scenario where being biased toward a particular approach or a particular set of features can help us.

GO: I tend to agree. The only way it could possible, maybe, is if you're biased to believe that requirements aren't everything, then maybe you'll test outside of them. But, yeah, I think in general, biases are something that are going to hurt rather than help.

PV: For an example, you may think that your development team is the best team on the planet. You may think that they're hot stuff. So you may—I'll say unintentionally, unconsciously—back off or evaluate more leniently than you might otherwise. On the other hand, if you think that you have a poor to average development team, you may find things that actually aren't there. It's possible to be biased both ways, and neither of them is particularly helpful.

CP: OK. You talk about the development team. A lot of times, or occasionally, development teams are made up of different types of developers or developers at different stages in their careers. Does the extent to which you are biased change depending on how many years . . . ?

GO: I don't think so. I think, maybe, having been in the testing industry now for quite a few years, I think, maybe, if anything, you might tend to become more biased. When I know they've got new a developer on the project: "Oh, cripe." Then, yeah, sure, I may test his code more, but any test effort, you have a limited amount of time. If I over-test the new guy's code, I'm going to be under-testing somebody who's been there for a long time. Potentially, yeah, that could hurt.

PV: Cameron, I think Gerie brings up a very good point here, that we will never rid ourselves of biases. But we can overcome them by reminding ourselves that they do, in fact, exist. Gerie just reminded herself that, "Gee, if I spend too much time with the new guy's code, if I'm biased against the new guy—or the new person, I should say—then I may not be doing my job."

CP: OK. Another thing, too, is you mentioned those three different types of biases, especially with inattentional blindness, where you just need to know what you're looking for. As you go through your career and you come up with the same problems over and over again, or you see the same type of defects happening over and over again, could that possibly cause you to become more blind, or more susceptible to inattentional blindness, as the years go by?

GO: I suppose it could, but I like to think that . . .

CP: Doesn't happen.

GO: This is how I think that way. I think you're right. It probably does. I think the best way to counteract that is have multiple testers on a project. Move testers around so that if you've got these verticals, especially when you test a center of excellence, you've got verticals where you've got the same testers.

They become domain experts. That's good, but that's going to lead to more of the bias. It's going to lead to the curse of knowledge. When they're seeing the same application, it's probably going to lead to the inattentional blindness. So it's probably a good thing to rotate some of your testers around. Give them new experiences. Put a fresh set of eyes on the same applications.

About the author

Cameron Philipp-Edmonds's picture Cameron Philipp-Edmonds

When not working on his theory of time travel, Cameron T. Philipp-Edmonds is writing for TechWell, StickyMinds, and AgileConnection. With a background in advertising and marketing, Cameron is partial to the ways that technology can enhance a company's brand equity. In his personal life, Cameron enjoys long walks on the beach, romantic dinners by candlelight, and playing practical jokes on his coworkers.

Upcoming Events

Sep 22
Sep 24
Oct 12
Nov 09