The Potential Pitfall of Ratings

[article]
Summary:

Responses to ratings-based surveys are particularly prone to misinterpretation if the surveys don’t allow space for open-ended comments. These comments offer insight into what respondents are really thinking, which may not be obvious from their ratings.

Responses to ratings-based surveys are particularly prone to misinterpretation if the surveys don’t allow space for open-ended comments. These comments offer insight into what respondents are really thinking, which may not be obvious from their ratings.

I gained some personal appreciation of the importance of comments after one of my seminars. The evaluation form my client asked me to use requested ratings on a 10-point scale. When I reviewed the feedback afterwards, the evaluations of two particular participants caught my attention. One rated me a 10 on every survey item. (Yes, 10 was the top score!) The other gave me a string of 7s.

Clearly, one participant thought I had done a great job and the other considered me maybe a C+. But which was which would not have been obvious without their comments.

In particular, the person who gave me the 10s listed several things he’d have liked me to do differently. More of this, less of that, longer, slower, etc. By contrast, the person who gave me 7s offered glowing comments, including that this was the best class he’d ever taken.

I don’t know, maybe the 10-rater felt that even when something is done well, there’s room for improvement; I certainly appreciated his feedback. And maybe the giver of 7s had never before given anyone better than a 6. Without their clarifying comments, I’d have drawn the wrong conclusions about their ratings. This sort of situation makes me question whether ratings alone have any value, except to satisfy management's demand to see numbers.

If you use a rating system to solicit feedback, provide a place for comments at the end. If feasible, also request a brief explanation for some of the items rated. The feedback you receive will help you draw more meaningful conclusions than ratings alone, and that information will help you target, with precision, the things you’d like to change.

(Truth in blog writing compels me to admit, however, that I still like receiving 10s more than 7s ...)

About the author

Naomi Karten's picture Naomi Karten

Naomi Karten is a highly experienced speaker and seminar leader who draws from her psychology and IT backgrounds to help organizations improve customer satisfaction, manage change, and strengthen teamwork. She has delivered seminars and keynotes to more than 100,000 people internationally. Naomi's newest books are Presentation Skills for Technical Professionals and Changing How You Manage and Communicate Change. Her other books and ebooks include Managing Expectations, Communication Gaps and How to Close Them, and How to Survive, Excel and Advance as an Introvert. Readers have described her newsletter, Perceptions & Realities, as lively, informative, and a breath of fresh air. She is a regular columnist for StickyMinds.com. When not working, Naomi's passion is skiing deep powder. Contact her at naomi@nkarten.com or via her Web site, www.nkarten.com.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Nov 09
Nov 09
Apr 13
May 03