Jeff Sauro • June 6, 2012

A lot happens when you observe a user during a usability test.There's the interface, the utterances, the body language and the metrics.

This rich experience can result in a long list of usability issues.

Issues can range from cosmetic (a user finds the type in ALL CAPS a bit much) to catastrophic (data loss), suggestions (a new feature) or positives (finding defaults helpful).

These usability issues should be described and usually accompanied by a screen shot or short video clip to illustrate their impact. But don't stop there. Usability issues should be converted into a user by problem matrix.

An example of a user by problem matrix is shown below from an evaluation of the Enterprise Rental Car website.

Figure 1: A User by Problem matrix for renting a car on Enterprise.com. This is not a QR code.

The users are numbered on the Y axis and the issues have each been given a number. Each black square represents which user encountered which problem. You can compute such matrices by task or across tasks for the whole usability test.

For this particular task, we observed 30 users encountering 28 unique issues while attempting to rent a car. We sort our problem matrices with the most frequent occurring problems on the left and users having the most problems on top. This gives the most problem density in the upper left corner.

A User by Problem matrix provides :

- Frequency by Problem: For example, the most common problem (far left) involved users having trouble adding a GPS navigation to their rental (they had to do it after they entered their personal details). This problem impacted 24 out of 30 users (80%). We can be 95% confident between 62% and 91% of ALL users would also have this problem.
- Frequency by User: The first user in the matrix encountered 9 of the 28 problems (32%) while user 28 encountered only 1 problem (3%) and users 29 and 30 encountered no problems while renting the car.
- Problems that affected only one user (the long tail of usability problems): The last nine problems were encountered by only a single user (3%).
- The average problem frequency: By averaging together the problem frequencies we get a sense of how common problems are. For this task, the average adjusted problem frequency is 12%, meaning users have about a 12% chance of encountering a usability problem given this task scenario. Technical Note: See the discussion on why and how to adjust Average Problem Frequencies.
- The percent of problems likely discovered: Given the total number of problems and the total number of problems encountered only once after observing 30 users we've seen about 97% of problems for this task, interface and type of user. See Chapter 7 of Quantifying the User Experience and the calculator for how to compute this figure as well as discussion on the limitations of this approach.

You can also compare problem matrices. The following image shows four matrices for two tasks on two different rental car websites: Enterprise and Budget.

Figure 2: User by Problem matrices and average problem occurrences (P) for two tasks and two rental car websites.

For renting a car, Enterprise had an adjusted average problem frequency of 12% compared to Budget's 9% for the same task. For finding the nearest rental location for a specified address the average problem frequency was 15% for Enterprise and 7% for Budget.

For both tasks, Enterprise had a higher frequency of usability problems and this was reflected in the lower task completion rates, longer task times and lower satisfaction scores we recorded in the comparative usability evaluation.

Taking that additional step of converting long lists of usability problems into a more numeric and visually digestible format provides both more clarity and credibility to usability testing.

The Experiment Requires That You Continue: On The Ethical Treatment of Users

28 Resources for Getting Started In UX

How to Conduct a Usability test on a Mobile Device

Confidence Interval Calculator for a Completion Rate

5 Examples of Quantifying Qualitative Data

97 Things to Know about Usability

The Five Most Influential Papers in Usability

Should you use 5 or 7 point scales?

What five users can tell you that 5000 cannot

How common are usability problems?

Nine misconceptions about statistics and usability

8 Ways to Show Design Changes Improved the User Experience

10 Things to Know about Usability Problems

A Brief History of the Magic Number 5 in Usability Testing

.

Quantifying the User Experience: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.