Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

The Right Tool for the User Research Method

Jeff Sauro • November 8, 2011

There are many great methods for gathering insights from users. And there are definitely many software programs and services to help.

Instead of making a long list of them I've presented the few I use the most when conducting user research.

Lab Based Usability Test

The classic method of usability testing involves bringing users into a lab (usually with a one way mirror) and observing them as they attempt tasks.

I typically use a screen recorder like Camtasia and a custom web-application called Usability Logger for recording notes and collecting metrics for stand alone tests or more complex mult-product within subjects tests. 

There are also free screen recorders like CamStudio which have worked for me in the past and the more sophisticated Morae which includes more recording control and data collection. You can go low budget on this too and use a conference room and pad of paper.

Moderated Remote Usability Test

Inviting participants to attempt tasks in a usability test while they share their screen from their home or work is a growing method in User Research. I use GoTo Meeting for the screen sharing and recording the audio and video.

There are several screen recording solutions out there including WebEx and Adobe Connect but I've found that GoTo Meeting tends to work on the most users most of the time and the recording is relatively easy.  If you don't have the budget then Mikogo is free, offers a free conference service and it's worked for me when GoTo meeting has not. The main downfall is that you cannot record audio with the screen (yet).

Unmoderated Remote Usability Test

Having users complete tasks without the need for a moderator's time or expensive lab costs continues to grow in popularity.  Good software can make the process even easier.

In most studies I use Userzoom which is like a Suisse Army Knife of User Research Tools including task based navigation testing, surveys, clicks-maps and card sorting. It also allows me to customize studies on the fly and works with either a plugin the user downloads or a line of Javascript we insert in the website.

For a low price Loop11 also offers a reliable task-based navigation solution and core reporting. Disclosure: Loop11 and Userzoom are both sponsors of MeasuringUsability.com

Video of Users

When you want a qualified panel of users to test a website and you want to actually see what they're doing then both Usertesting.com and Userlytics are good solutions. Userlyrtics also offers recordings of the user's facial expressions which can be helpful for branding and awareness studies.  

I've often combined both Usertesting.com and Userzoom to have recordings of users interacting with the site, detailed heat maps and a rich metrics to analyze. Disclosure: Usertesting.com and Userlytics are both sponsors of MeasuringUsability.com.

Standardized Questionnaires

Nothing makes survey data quite as meaningful as comparing it to a standardized benchmark. For a standardized measure of website usability, credibility, loyalty and appearance I use the SUPR-Q. For a standardized measure of usability for software, cell-phones, hardware and IVR systems I use the System Usability Scale (SUS).

Predicting Task Times

When shaving seconds or minutes off a task is paramount (such as in software that's used in call centers) getting an estimate of skilled users is one of the best ways to measure improvements. Keystroke level modeling is a method that's been refined over the last 3 decades and provides a quick and reliable way for estimating task times and assessing productivity improvements.

I use a method called composite KLM and the software I developed available at www.compositeKLM.com to generate quick and reliable estimates. Cogtool is a free tool developed at Carnegie Mellon. It requires more of an investment to learn and is especially suited for mocking up new interfaces.

Card Sorting

When you need to know how to categorize your navigation or what to name the labels then card sorting is a great method. You can go low tech and actually use index cards but there's software that makes both delivery and analysis easier.

Web-based solutions like Optimal Workshop's Optimal Sort and Userzoom's card sorting tool also allow you to collect data from users located around the world. Another Disclosure: Optimal Workshop is a sponsor of MeasuringUsability.com.

Surveys

Surveys continue to be a popular and efficient way of collecting feedback quickly. I created my own survey platform UsableSurveys.com which allows me to collect both simple survey data (multipl-choice questions and open-text) as well as more complex A/B concept testing and top-task ratings.

For complex branching and tracking SurveyGizmo and Userzoom offer good solutions. The ubiquitous Survey Monkey is also an option when there's $0 budget.

Recruiting Users

When I need users recruited and sent to an unmoderated evaluation I rely on panel agencies to recruit and deliver users.  I've had good experiences with Toluna for the general Internet population, car buyers and IT decision makers.  For hard to find users (like hardware engineers) Userlytics offers a competitive service and even Usertesting.com has come through for me for a retail study.


 
 

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More



You Might Also Be Interested In:

Related Topics

User Research, Methods, Remote Usability Testing, Usability Testing
.

Posted Comments

There are 2 Comments

February 23, 2013 | Joanne Locascio wrote:

Thanks. Very helpful.  


November 11, 2011 | Catalina Butnaru wrote:

It's a rich summary of usability tools, but there's something missing from the lab testing methodology, and that is eye tracking. It seems to be overlooked probably because it's not an affordable and accessible method. It requires training in the field of cognitive psychology and very good understanding of the different metrics used in this case. 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 3 + 2: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4239 Subscribers]

Connect With Us

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.