Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Prioritizing Problems in the User Experience: The FMEA

Jeff Sauro • January 22, 2013

Traveling is hard. Traveling with children is especially hard.

A number of things can go wrong, making the trip difficult or even nonexistent.

Some problems are nuisances (sick and/or hungry kids, delayed flights, or the wrong sized rental car), while other problems will lead to failure, which in this case means not reaching your destination on time (forgetting your passport or sleeping in and missing your flight).

The prevention of failure involves planning in order to avoid problems and having a backup plan when things go wrong. 

You can reflect on previous trips and their mishaps, and you mentally walk through all the logistics and work out details with your spouse to be sure that things aren't forgotten.

The process of thinking through potential problems and knowing how to detect and prevent problems before they create a catastrophe is what the Failure Modes Effect Analysis (FMEA) is all about.

Problems in the User Experience

Things go wrong quite frequently in the user experience.  Users mistype email addresses, forget their password, and experience long wait times on customer support. They are unable to find products, they take too long to make a purchase, and their information can be stolen.  While we ultimately want a great user experience, measuring poor experiences helps us understand how to prevent things from going wrong and generate better experiences.

In any user interface, some portion of users will have difficulty accomplishing their intended goals. Usability problems are inevitable and ubiquitous.  Even after several rounds of testing and improvements, some users will still have some problems.  The goal is to minimize the frequency and severity of problems, errors, failures, and of course those UX catastrophes.

Intuition will get you only so far. Refining the user experience further requires more systematic thinking. Like many of the Six Sigma tools, the FMEA helps facilitate a more critical examination of the tasks users attempt and helps to uncover problems and prioritize fixes.

How much is your car worth?

Let's image you're interested in buying a new car. Should you trade it in or try and sell it yourself? Either way, you want the most money for your car, and knowing its true value is essential information.  Many websites (e.g., Edmunds, Kelley Blue Book, Autotrader and Cars.com) offer the ability to tell you how much you car is worth.

The task seems simple: Enter your make and model, condition and mileage, and like the Magic 8 ball, you get a value. Of course, there are a number of details that complicate the process and things can go wrong. You can pick the wrong make, be unable to describe the condition or understand automotive jargon, leave off features, or the entire process can take so long that you finally give up. 

Applying the FMEA in finding the value of your car can help the designers of these websites prevent nuisances and failures (which would be not getting a correct value or the wrong value for a car).

Step 1: Identify the steps in the process. Decompose a typical scenario into small steps to identify failure points or bottle necks. Ideally, you can do this by watching the path users go down from a usability test. If not, pick the most common paths users would traverse from log-data and perform what is done in Keystroke Level Modeling.

Step 2: What can go wrong? Here's where recording poor experiences comes in handy. Are users taking too long, getting the wrong car value, making errors or not recommending the website or product because of pricing or policies? Usability problem lists with severity ratings can be especially helpful in identifying the sort of failures you want to avoid. Are users entering the wrong make, picking the wrong condition, getting slowed down by too many pages or intrusive ads, or are they stopping at a preliminary estimate before getting the right value? Use any and all data sources to get a good picture about problems. These sources include call center data, surveys, 3rd party reports, former customers and competitive data.

Step 3: How common are the problems ? Not all problems affect all users. Watch just a handful of users in a usability test and you'll see that some users struggle with one step while others breeze through all of them. For example, in our test of the Enterprise Rental Car Website we saw that most users had a problem finding the total price for their rental car, while only a few users had difficulty picking the correct car type. For the FMEA, rank problems from least common (assigned a 1) to most common (assigned a 10). Usability problem occurrence has a critical role in estimating sample sizes as well as the FMEA. Even if a problem isn't terribly harmful, if most users encounter it, it can be worth addressing.

Step 4: How severe are the problems ? Just because a problem affects a small percentage of users doesn't mean that it's not a big deal. If a few users are getting the wrong car value and losing money on their trade-in, the negative experience can snowball after the dissatisfied users tell their friends. In the FMEA, prioritize the severity of the problem from cosmetic/minor (assigned a 1) to leading to task failure, loss of money or loss of life as a 10.  The rankings are largely context dependent so don't get bent out of shape arguing over whether a problem is a 4 or 5. Spending time understanding the consequences of failures helps make the rankings more meaningful. This step also emphasizes the importance of separating problem frequency and severity.

Step 5: How hard are the problems to detect?  Some of the most pernicious problems are the ones that are hard to detect.  If a problem is harder to detect, it's harder to diagnose and prevent. It's like when you take your car to the mechanic because of the noise it's making, but when the mechanic goes to listen, it's fine.  In usability testing, this is usually a function of only some users having a problem with an interface element some of the time.

For example, the wording of a condition of a car might only confuse users trading in a certain model and year as well as those who have never traded a car in before.  Or, the layout of an interface might only affect users on old versions of Internet Explorer with low resolution screens.  Like the previous two steps, assign problems that are easiest to detect as 1's and the hardest to detect as 10's.  In our experience, it can be difficult to know if a problem is easy or hard to detect.  When in doubt use a 5, and then change the rating when you feel more confident about how easy or difficult it is to detect.  Using a 5 has the effect of neutralizing the detection aspect of the FMEA, and this gives more weight to frequency and severity.

Step 6: Calculate the Risk Priority Number:  The RPN is found by multiplying the ranking of occurrence by severity by detection (RPN = F*S*D).  This is similar to the criticality score that Rubin & Chisnell described in their book on usability testing and provides the weighted ranks to help prioritize and fix usability problems.  The higher the Risk Priority Number, the higher the priority the feature has to be addressed. 

The figure below shows three examples of possible problems in the car appraisal and purchasing process. Like the QFD, the raw value is less important than the relative rank or the magnitude of the difference. In the example data, the "Estimated Car Value" function has a higher problem possibility than the "Finding if a car is available in a zip code" function.


Figure 1: FMEA worksheet example (click to enlarge) and download a template.

Step 7: Identify Root Causes of the failures: Ranking what problems to address is one thing, finding ways to avoid them is another. For each of the undesirable outcomes, think of the root causes of the problem.  Sometimes the causes are obvious, while other times it can be helpful to use a Fishbone and answer "Why?" until you uncover root causes instead of symptoms:
Why are users getting the wrong car value?
They don't scroll down to see the additional features section.
Why? There options take up more than single page.
Why?  The screens are smaller to allow for more ads?
Why?  These pages are among the highest visited on the site and ads generate a significant portion of revenue.

You can download the FMEA spreadsheet to help conduct an FMEA in your next UX project.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Six Sigma, Usability Problems
.

Posted Comments

Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 2 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4106 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Use Card Sorting to improve your IA

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.