Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

5 Ways to Maximize your Usability Testing Efforts

Jeff Sauro • February 26, 2013

When conducting a usability test, gathering input on a design, or testing new features, we strongly recommend the 1:1 moderated session over group sessions.

However, we usually aren't convincing others to run 1:1 sessions.

Instead, we spend more effort convincing others of combining the traditional in-person 1:1 session with other methods.

We find we can generate more insights at the same or less cost using a mix of remote moderated and unmoderated testing, surveys, and inspection methods.

Gathering input from a user in a face-to-face session provides rich feedback without the biases introduced by multi-person sessions. A skilled moderator with the right tasks can gather immediate feedback on what the users says and what they don't say or do.

Slight hesitations, long pauses and puzzled looks provide cues for the moderator to probe deeper into potential problem areas. However, those benefits come with some costs.

One-on-one in person sessions can be:

  • Time consuming. For every user, you need at least a facilitator's time and possibly a note-taker as well. With typical sessions lasting an hour along with transitions and late arrivals, it's hard to run more than five users in a day.  Running concurrent sessions means higher costs and the problem of merging notes.

  • Harder to recruit for. It's not easy to get people to come to a facility during the workweek in a particular city. This is especially difficult when you're recruiting specific profiles like small business owners, accountants or lawyers.

  • Biased. Sample bias is ubiquitous in research (even academic research) so there's no avoiding it. However, participants who have the time and are able to come to a lab in the middle of a workweek may be less representative than those who might have a few minutes in the evening.

  • More expensive.  Getting people to come in person means higher honorariums costs ($100+) for one hour of time.

  • Harder to schedule. You've usually got three schedules to work with: the user's, the facilitator's, and the client's, not to mention fixed lab times.  Scheduling can be a thankless job.
Despite the drawbacks of in-person moderated testing, we still use it and love it. We simply don't rely solely on it for gathering insights.  Instead, we use a mix of surveys, moderated (remote and in-person) and unmoderated testing to maximize our usability testing budgets.

Here are five ways to maximize usability testing budgets:
  1. Facilitate fewer sessions. If you have the budget and time to conduct thirty 1:1 sessions, run half the users and apply the remaining funds and time to using additional research methods to triangulate on your questions. This will also increase the confidence in your findings.

  2. Survey. Surveys are one of the most cost-effective ways of gathering information from users. They aren't a substitute for a usability test, but they can quickly gather data on general preferences and behavior. We often use a survey to screen participants for a usability test AND to gather data on general behaviors. The in-person time is too valuable to waste on things such as the websites that users have been to, demographic questions, or even attitudes about product features. Gather those ahead of time, follow up with the participant if necessary, and summarize those surveys in the report.

  3. Use remote moderated sessions. We can usually recruit more easily, with less lead time and have a lower honorarium for participants who can participate from their home or office. As an added bonus we've found we can recruit more difficult to find users like doctors and lawyers and such.

  4. Conduct unmoderated sessions. With the same tasks and interface you can test dozens or hundreds of users for the same recruiting costs as 20-30 in person sessions. This is especially the case when the interface is a website or any prototype that can be hosted remotely.  What's more, services like Usertesting.com and now UserZoom, offer the ability to record the screens of users in unmoderated sessions. Having a video record of where the users get stuck or distracted is just about as good as the lab itself.

  5. Use Heuristic Evaluations. It's often not requested by clients or specified in project documents, but having even a single expert review an interface with a set of tasks can reveal about 30% of the issues users encounter in a usability test.  This can be especially helpful when users are hard to find. What's more, Heuristic Evaluations tend to uncover legitimate problems that aren't encountered during a session because of time constraints, but are nonetheless valuable for development teams.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Usability Testing
.

Posted Comments

There are 1 Comments

March 21, 2013 | ramesh wrote:

Hi,Jeff very informative content on maximizing usability testing.I think the idea of surveys and remote moderated session is great to improve our testing.By the way you can get some great information at http://www.techved.com on usability testing.Check out this site as well for great information 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 1 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[3796 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Usertesting.com

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.