Usability: Don’t Listen, Just Watch


Someone once said to me, "If you don't fail, you don't learn." Well, many people have said that to me. Essentially, it’s the driving philosophy behind agile development and user-centered design. It's one of those statements that appears to be very logical, but is not so easy to put into practice. The fear of failure is what drives the majority of our anxiety in the corporate world. If you weren't afraid of messing up your speech to one-hundred-plus people, would you still be nervous? I think not, well not as much anyway.

Overcoming the “Stupid Question” Fear
What worked for me was starting small. Taking advantage of times where failure didn’t matter a great deal; meetings with small numbers of attendees, delivering training to small groups, etc. The lesson from this is to make the participants fear of failure as small as possible. As you're demonstrating the product to them or preparing them for the evaluation, perhaps mention some of your own inadequacies in relation to the product; but of course be very careful not to “lead the witness;” you need their feedback, not yours through their words. This may help their fear move to a more manageable level.    

Balancing the need to observe with not making participants nervous to the point of being useless can be complex to achieve. It will of course depend on the participant and their previous experiences as to how much you need to adjust your approach, but being armed with some easy-to-deploy and flexible techniques will put you in a good starting position. Here are a couple of mine that have proven valuable in the past

  • A comfortable environment. Quite often this is the participant’s usual work space. They are surrounded by their work mates, they have their family and pet photos on their desk and you’re simply a visitor 'pulling up a chair'. Also, the need for filming cannot be a blocker to this anymore as setting up a smartphone to lean against the monitor is an easy and non-intrusive form of video capture.
  • A soft introduction to the product. You need to be careful with this. The last thing you want to end up doing is providing a training session and therefore negating the evaluation almost completely. An introduction is common practice for an evaluation; however, there is an observation opportunity that is often overlooked (pun intended). Make the participant aware that it’s an introduction, which will help them be more comfortable as they will assume you’ll be doing the work, and while introducing the product make sure you observe them at the same time. You can gleam some very valuable information just from this first period of the evaluation. “Why did you have to do that?” A powerful question from the participant; and you’re only two minutes in.
  • Understand their background. Talking to the participants about their experience is a valuable conversation. Asking them about their specific role in the company is great, but I also like to take it to a more personal level. Sharing a conversation about a favorite show or book adds an element of harmony to the situation. For those participants that are very much “work only” and are not willing to discuss personal details, you can be sneaky and build it into the evaluation. I have previously advised a participant that I needed to know some of these details so that we could more accurately assess the demographic of the user base. When she answered how many kids she had it was easy for me to ask their age and then to bring my child into the conversation, and away we went talking about schools and sport. The benefit to understanding their background is two-fold. First, you get to know what experience they have and how this may impact their use of the product. Second, it will assist in building a rapport with the participant, helping to ease those very important comfort levels. 

Getting participants to a state of mind where they are comfortable to honestly raise whatever issues they come across, including emotional responses, is the ultimate goal. If this does not work, observation may be the only avenue you have, so it’s important to keep your eyes open, and don’t always believe the participant when they say the product is easy to use.

Do you have any tips on how to ease the comfort levels of participants or examples of when observation was your only hope? Please let us know in the comments section below.


User Comments

1 comment
Duncan Nisbet's picture

Nice to read a post from the other side of the looking glass David

"Getting participants to a state of mind where they are comfortable to honestly raise whatever issues they come across, including emotional responses, is the ultimate goal."

That is such an important point & cannot be reiterated enough.

Formal UX evalutaions remind me of lie detector sessions :-)

I remember doing market research when I was younger - the sessions were invariably run in the back room of a bar / pub with snacks & a couple of beers. This really helped the group relax, loosened the tongues & hopefully provided some useful feedback. Probably not advisable in the more formal work setting though...




January 6, 2014 - 8:56am

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.