Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

How common are usability problems?

Jeff Sauro • September 29, 2010

Just how common are usability problems in websites and software? 

Surprisingly there is very little out there on the frequency of usability problems. Part of the reason is that most usability testing happens early in the development phase and is at best documented for an internal audience. 

Once a website is launched or product released what little usability testing is done is typically more on benchmarking than on finding and fixing problems. 

I wanted to get an idea of how likely a user is to encounter a problem on completed software so I reviewed usability publications and a collection of usability reports from companies.I only included tests on completed applications and live websites and excluded those that were in the design phase and didn't have current users at the time of testing.

24 Usability Tests with Problem Frequencies

My investigation turned up a wide range of products and websites from 24 usability tests. Examples include rental car websites, business applications (financial and HR) and consumer productivity software (calendars, spreadsheets and word-processors).   I didn't include data from Heuristic Evaluations and Cognitive Walk-Throughs. While these methods are effective for identifying usability problems I wanted to focus just on problems that users actually experienced. 

Finding the Average Usability Problem Frequency

To get an idea of how common problems are I divided the number of users that encountered each problem reported in each of the 24 tests. I then averaged all the problems frequencies for each test.  For example, in one usability test with 8 users, 9 total problems were found.  The number of users that encountered each problem is shown in the table below.

 Problem  P1   P2   P3   P4 P5  P6 
  P7  P8 
  P9

 Frequency  7  7  8  4  3  2  8  4  3
 Total Users
 8  8  8  8  8  8  8  8  8
 Proportion  .875 .875
 1  .5 .375  .25  1  .5 .375Avg =.639

The average problem frequency for this test is .639 or there's about a 64% chance a user will encounter a problem with this application and set of tasks. 

Average Problem Frequencies are Inflated

One problem with this calculation is that it is a bit inflated. More obvious problems tend to show up with small sample sizes. It takes a much larger sample size to detect problems that affect a small percentage of users. If more users were tested, more idiosyncratic problems would turn up and thus lower the average problem frequency (a sort-of long tail of usability problems).

Fortunately there is a way to adjust for this bias to provide a more accurate picture of the average problem frequency (Good Turing & Normalization Methods[pdf]). As a short-cut approach, multiplying the average frequency by .9 and subtracting .04 tends to work just as well. So for the example problem set above we get an average problem frequency of .9*(.639) - .04 = .54. In other words, there's a 54% chance a user will encounter at least one usability problem with this interface. For this application, a usability problem is unfortunately common.  

Business Application have more Usability Problems than Consumer Software and Websites

The list of problem frequencies for the 24 studies are shown in the table below. If the data came from a publication it is noted with a superscript reference number.

  Business Applications
 Consumer Software
 Websites
.65
.54
.49
.46
.43
.40
.40
.31
.30
.09
.02
       
 .35 6
 .34 5
 .28 5
 .25 5
 .21 3
 .10 2
 .09 1
 
       
.07
.06
.04
  .03 4
.03
.03
Mean (SD)
 .37 (.19)
 .23 (.11)
 .04 (.02)

While there are a couple exceptions, this data tells us something that may seem obvious: On average users are more likely to encounter usability problems in business software than with consumer software and websites [F(2,23) =10.74; p <.01].

What wasn't obvious was the magnitude of the difference. Usability problems are almost ten-times more common on business applications than on websites. The ratio is around 2 to 1 for business applications and consumer software.

Business applications are typically customized to integrate into enterprise systems. They are often purchased to reduce costs and improve productivity and users often receive some training or have specialized skills. Business software typically contains a lot of complex functionality and so it makes sense that there are more things that can impede a good user experience.

Consumer software and especially websites on the other hand are typically self-service. If users can't walk-up and use a website, they're gone. Switching costs for consumer software and websites are low so there is a low tolerance for a poor user-experience. They often have a fraction of the functionality of large-scale business applications.

Not All Usability Problems Have the Same Impact

Of course usability problems aren't all created equal. Their impact can range from trivial to tragic and in this analysis I didn't take into account problem severity. With that said there's good reason to believe that problem frequency and severity are independent—meaning a usability problem is just as likely to have a major or minor impact on the user experience.

Sample Sizes for Usability Testing

Much of the controversy around how many users to test stems from the variability in problem frequency. If users are less likely to encounter a problem, then you need to test more users to find more problems. Even with this limited sample of usability tests, you can see that you'll need a larger sample size when testing live websites than testing most live business applications.

The public facing websites in this sample have tens of thousands of daily users and most of the usability problems have been addressed. If you wanted to find more problems that affect at least 4% of users than you'll need to plan on testing 46 users. In comparison, for many of the business applications in the sample, testing only 4 users would uncover most of the problems that affect at least 37% of users.

In other words, the rule of thumb for testing around 5 users is likely pretty close for business applications, but eight is not enough for websites and most consumer software.

Track Problem Frequency

It's a good idea to keep track of how many users encountered each problem, and then keep track of that information for internal benchmarking efforts and sample size planning. At the very least you can compare the problem frequencies of your next usability test to the benchmarks in this article. For example, if your website has an average usability problem frequency above 20% then you'll know there's still a lot of improvements than can be made.

In addition to reductions in task time and higher satisfaction ratings, reductions in the frequency of usability problems can be a great way to show how usability resources improved the user-experience.

References

  1. Law, L. -C. and Hvannberg, E.T(2004), Analysis of combinatorial user effect in international usability tests. In CHI Conference on Human Factors in Computing Systems, ACM,  9–16.
  2. Lewis, J. R., Henry, S. C., and Mack, R. L. (1990). Integrated office software benchmarks: A case study. In HCI - INTERACT '90, Proc, (pp. 337-343). Cambridge, England:
  3. Nielsen, J (1994) Estimating the number of subjects needed for a thinking aloud test. International Journal of Human–Computer Studies 41,, 385–397.
  4. Spool, J. & Schroeder, W. (2001). Testing web sites: five users is nowhere near enough. In: CHI '01: CHI '01 extended abstracts on Human factors in computing systems, ACM.
  5. Virzi, R.A.(1992) Refining the test phase of usability evaluation: How many subjects is enough? HumanFactors 34, , 457-468.
  6. Woolrych, A. & Cockton, G. (2001). Why and when five test users aren't enough.. In: Vanderdonckt, J.; Blandford, A. & Derycke, A. (Ed.), Proceedings of IHM-HCI 2001Conference, Cepadeus.



About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More



You Might Also Be Interested In:

Related Topics

Problem Discovery, Sample Size, Usability Testing
.

Posted Comments

There are 1 Comments

February 23, 2011 | Chris wrote:

There is no mention of what business application are web based or desktop based, also there is no mention of any technical constraints or delivery constraints. 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 1 + 1: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4207 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Use Card Sorting to improve your IA

Usertesting.com

Loop11 Online Usabilty Testing

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.