Usability: Don’t Listen, Just Watch

[article]
Summary:

When projects employ professional testers, their focus is generally “functional” testing, and usability quickly becomes an afterthought—if it's thought about at all. David Greenlees writes that getting participants to a state of mind where they are comfortable to honestly raise whatever issues they come across, including emotional responses, is the ultimate goal.

 When I’ve been undertaking usability testing, no matter how many times the facilitators tried to tell the participants they were evaluating the product and not them, the participants still got the sense they were the ones being “tested.” Their conversation and feedback are very much centered on their own abilities, not on the product. In general, this sense instills instant anxiety and is very difficult to get past, even if you are a master of dealing with nerves. It is a very rare occurrence for individuals to not be anxious when they feel as though they are being tested.

Granted, this doesn’t happen all the time. When engaged, professional user research participants will tell you exactly what they think of the product as they understand the importance of such information. However, how often do projects employ professional user-research participants? I’ve only seen one in my career, which spans over hundreds of projects. More often, projects employ professional testers, but their focus is generally “functional” testing and usability quickly becomes an afterthought—if at all.

My Anxious First Time Participating
Before running evaluations myself and undertaking usability testing as a professional, I experienced a usability evaluation as a participant.

It was an internal product evaluation, and I was new to the company. The test took place in a “usability lab” that had one-sided glass adjoining the observation room. There were three or four cameras recording me, and I knew there were also several people in the observation room as I had seen them all moving around before we started. I was a nervous wreck! It was like I was auditioning for some type of acting role. I do believe that it was more distressing for me than public speaking, and at the time that was one of my greatest fears.

I ran through the various scenarios as directed; however, to this day I question the value of any observations they may have made and any information they gained through feedback from me. My nerves impacted my reactions and product navigation, of that I have no doubt.

The facilitators offered the usual assurances, and told me that there was no need to be nervous; however nerves are not something that simply vanish on request. 

What Could They Have Done?
The best way forward in that scenario, from my personal perspective, would have been for one of the observers to sit next to me at a computer in a location that was familiar to me (my usual workstation) and taken simple notes as I moved through the scenarios; an approach that has become more common in UX design more recently.

I would not have been anywhere near as nervous and my reactions would have been less tainted. Yes, they would not have had their video recording, but as I mentioned, this footage is of no real use anyway. My emotional responses were too impacted by my nerves. It was not a total loss; I do recall discussing certain points of confusion. However, would those points have been as confusing if I was not so nervous and therefore clearer minded?  

After that event, it took years to build up the courage to participate in more evaluations. The only reason I did was due to the product being far more familiar to me by that time. It’s important to gauge how the participants are feeling before, during, and after an evaluation. Usability is subjective, and feelings impact decisions. As feelings change, so to do the associated decisions.  

No Feedback at All?
There are also times when you can’t worry about what the participants are saying, as they won’t say anything at all. When it comes to usability testing, there are no stupid questions. Make sure your participants are aware of that fact, and don’t be afraid to ask questions in order to prompt open responses from them. 

The Participant’s Point of View
Imagine you’re a participant in a usability evaluation of some description, sitting in a room in front of a computer with two, maybe more, people watching your every move; not unlike my first experience. This would be somewhat nerve-wracking if you were an expert user of the product, let alone not knowing it well (which most participants won’t). The potential to feel very inadequate as a user is quite high, and therefore questions would be easy to avoid asking.

How many times have you stopped yourself from asking a question as you were afraid of looking stupid? Earlier in my career I was guilty of this often. I was concerned of what the people around me would think if the question I asked had an obvious answer that I had not seen. 

As a software tester I have learned that questions are what we do; they are the core of our profession, so I have left that problem far behind. For many others though, particularly when faced with an anxious situation (i.e. participating in a usability evaluation), this is an all too common scenario. 

User Comments

1 comment
duncan nisbet's picture

Nice to read a post from the other side of the looking glass David

"Getting participants to a state of mind where they are comfortable to honestly raise whatever issues they come across, including emotional responses, is the ultimate goal."

That is such an important point & cannot be reiterated enough.

Formal UX evalutaions remind me of lie detector sessions :-)

I remember doing market research when I was younger - the sessions were invariably run in the back room of a bar / pub with snacks & a couple of beers. This really helped the group relax, loosened the tongues & hopefully provided some useful feedback. Probably not advisable in the more formal work setting though...

 

Duncs

 

January 6, 2014 - 8:56am

About the author

David Greenlees's picture David Greenlees

David Greenlees has been testing software for over ten years. Many of these spent in one of Australia's largest government departments, while more recently undertaking a consultant role in multiple organizations. He is a vocal and valued member of the Context-Driven Testing Community and is extremely passionate about the betterment of the software testing craft. He has published several articles and blogs regularly at http://martialtester.wordpress.com/ and http://hellotestworld.com/. In 2012, David founded the Australian Workshop on Software Testing (http://ozwst.wordpress.com/), Australia's only Peer Conference. Currently authoring a book on the subject of software testing and martial arts, his passion outside of work, you can follow David on Twitter @MartialTester.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Oct 12
Oct 15
Nov 09
Nov 09