Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

What UX Methods to Use and When to Use Them

Jeff Sauro • March 12, 2013

There are a number of popular methods used in improving the user experience at all phases of research and design. 

The following are some of the more popular methods we use and when we use them. 

We will focus on most of these methods in detail at the UX Bootcamp in Denver.


Requirements

Who are the Users and Customers?

  • Survey:  The cheapest way to find out who your users are, what they want, what they do, what they purchase, where they shop, and what they own is to survey them. Use internal and external contact lists to get a less biased view on your customers. Survey software is free, so there's no excuse.

  • Persona / Market Segmentation: Turn survey data into meaningful clusters.  What functions do certain segments want, and when in the buying decision do they care the most?  Think beyond gender, income and age, and look to tasks and domain experience as key differentiators. 

  • Competitive Analysis: Rarely does a product or website do something that NO ONE else does. Understand the market, find out what similar companies do in your market and look to similar industries. What features are common? What delights customers?  Use industry benchmarks like the Net Promoter Score for word-of-mouth and the System Usability Scale for usability.

  • Contextual Inquiry: Users can't always articulate what they need or what they want. Through the process of observing users in their workplace or home attempting to solve problems and accomplish goals, look to identify unmet needs and understand the tasks they perform.

  • Stakeholder Interviews: An amazing amount of information already exists in different departments across companies. Don't simply interview the HiPPO's. Use a structured interview to ask customer support, QA, development, marketing and sales to find out what to build, what to fix, and what to cut.
  • Quality Function Deployment: Structure the ideas from internal stakeholders with data from users and customer into a matrix to understand what functions will meet the most internal and external requirements. Incorporate the competitive information to really impress your colleagues and make more informed decisions.

What are users trying to do?

  • Task Analysis: Decompose what users are trying to accomplish to understand how the application makes tasks more efficient and effective. See the Bible on Task Analysis: Contextual Design by Beyer and Holtzblatt.

  • Top Tasks Analysis: Your application can't do everything for everyone all the time. Most people use applications (software or websites) for just a handful of tasks. Survey users and find out which "vital few" tasks satisfy the majority of the needs most of the time. Be sure the application does these tasks well. See Gerry McGovern's book "The Stranger's Long Neck."

Design & Development

What will the interface look like?
  • Wireframing : Sketching the major elements of an interface early using paper, Visio or PowerPoint is sufficient for understanding functionality, flow and opportunities for improvement.  It allows you to get designs in front of stakeholders.

  • Prototyping : Add a level of fidelity to your designs and get them tested early and often. Lean UX means generating the minimum viable prototype. Be ready to throw it out and start over. There are a plethora of prototyping tools for non-coders that turn images and sketches into hosted websites and clickable screens. Try Justinmind, Axure or Balsamiq.

Testing & Evaluation Methods

How will it be organized?

What problems are users having?

  • Moderated In Person Testing: Ideal for mobile device testing or when it's tough to put prototypes up remotely, test users in a lab, conference room or even a hallway to get an idea about what tasks are problems and what needs to be fixed.

  • Moderated Remote Testing:  Using cheap and ubiquitous services like GoTo Meeting or WebEx, you can recruit users from anywhere in the world to attempt tasks. You can even record facial expressions with webcams.  Don't simply ask what users think of the design, have them attempt tasks and probe on difficulties and collect metrics.

  • Unmoderated Remote Usability Testing: If your designs and tasks are well defined enough and you can host your prototypes online, users can attempt tasks remotely without having to be there. You can even test images with hotspots. With structured tasks and specific questions using services like UserZoom, Usertesting.com and Loop11 can get you results from dozens to hundreds of users the same day.  Act on the insights and test again.

Deployment and Release

  • Usability Benchmark Study: Now that everything is functioning, understand how usable your website or software is by having a representative set of users attempt tasks. Collect metrics and use confidence intervals to generate a reliable benchmark. Use standardized questionnaires (after the task and at the end of the study) where possible. This can be done in a lab-environment or remotely.

  • Unmoderated Remote Usability Testing : With a live website you can have users attempt the same tasks you identified early in the Top Task analysis and in the formative design stage. You can record clicks and even have an entire video that shows where users are getting stuck without you being there.

  • Comparative Benchmark Study : How difficult are the same tasks on the competitive applications you defined in the requirement stages? Recruit users, use core metrics like completion rates, time and task-difficulty and see the strengths and weaknesses of your website.  Sometimes the best comparable is a best in class website that provides a similar service in a different industry. If you're selling mobile-service plans, consider comparing the checkout experience to DirecTV or Zappos.

  • A/B Testing : Don't guess, test. Design and improvements don't stop once you're released; this is much easier in the web-based application word. Test forms, buttons, copy, images and prices. Don't be afraid to test wild-card ideas.

  • Multivariate Testing: One-variable-at-a-time testing helps tweak the website but can take a long time if you want to test a lot and you will have no idea how two elements interact. For example, surprising things happen when you couple a lower price with a different product package (two variable interactions). You can multivariate test on a live website or simulate the experience in a development environment using attitudinal data instead of actual purchases.

  • Survey: Are users recommending your website or product? Do they trust it and find it appealing? Compare your scores to industry benchmarks and use standardized questions. Ask what users would improve and associate open-ended comments to quantitative data.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More



You Might Also Be Interested In:

Related Topics

Methods, A/B Testing, Usability Testing, Survey
.

Posted Comments

There are 5 Comments

June 9, 2014 | Charles wrote:

Good list, but would point out that personas and segmentations are two very different things, made in different ways and for different purposes.  


December 11, 2013 | Jessica Miller wrote:

Great article, you pretty much got everything!rnHere are my three best usability analysis methods to use.rnrnhttp://usabilitylab.walkme.com/main-usability-analysis-methods-to-use/, I'd love to hear what you think of my post! 


March 15, 2013 | Len wrote:

This is a great read - thank you for sharing. Our developers here at Seamgen use similar methods when it comes to designing and building custom apps for our clients and to create a great user experience when finally utilizing their app for their business workflow. We also test our applications to make sure it meets our benchmark as well as our client's unique needs.

http://www.seamgen.com/mobile-software-development/ 


March 15, 2013 | Marc Resnick wrote:

Your timing for this article is perfect. We are organizing a real time design charrette at User Experience Day this year (held during the Human Factors Annual Meeting in October in San Diego). The plan is to divide the room into four quadrants, have four different UX methods going on simultaneously evaluating the same application, and having attendees circulate - checking out the ones they are most interested in. Then at the end we have a wrap up where we compare/contrast the results from each UX method.

If anyone is interested in participating, or just attending, please let me know. mresnick at bentley dot edu. 


March 14, 2013 | LM wrote:

Very nice summary of methods here!
We know recruiting can be a hassle... so ethn.io offers a service for live-recruiting, as well as the option to send out direct links via email, Twitter, Craigslist, etc... and integrates with loop11, Usertesting, OptimalSort, and Usabilla... Just sayin. ;) Hope it helps! Let me know if you have any questions. 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 5 + 2: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4278 Subscribers]

Connect With Us

Our Supporters

Usertesting.com

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.