Sample Size Calculator for Discovering Problems in a User Interface

Posted Comments

There are 7 Comments

February 22, 2012 | Jeff Sauro wrote:

Great question. If we assume 30% of users actually would have a problem with part of the interface, then you'll have a good chance of seeing that AT LEAST once in your usability test of 5 users.

A problem is generally considered a problem if it happens at least one time and that's how the calculator works. Determining if a problem is more than minor is an issue of judgment and not math.

For example, in a recent benchmarking study of a rental car website, we watched as one user was ready to rent the car, but then switched the payment option on the checkout screen to "Pay Now" instead of "Pay at the Counter." In making this change, the price didn't change (as it did on the previous page).

This only happened to 1 out of 5 users. Perhaps this is a bug or it's working as intended. It could take a lot to fix this issue and maybe the development team will only fix it if they know enough users will actually do this (so it's not just a fluke). Given just 5 total tested, we can be 95% confident between 2% and 64% of users will have the problem. Definite uncertainty, but highly unlikely that say only 1% of users would have the problem. If the problem impacts 5 out of 5, it's likely at least 60% of the user population would also have the problem.

This touches on the issues of problem frequency and severity. A problem can happen to 5/5 and it seems more severe simply because of its frequency, but its impact can be minor (like a small typo) or a problem can impact 1/5 users and its impact can be major (loss of a sale).

As a matter of strategy you can certainly have a threshold where something is only considered a problem if it happens to AT LEAST 2 users, instead of AT LEAST 1 user. This increases the sample size needed and you can see Table 1 here: http://www.measuringusability.com/qualitative_sidebar.htm

 

---

February 22, 2012 | Lucas wrote:

Hey Jeff,
I am a big fan of your blog and after reading this post I kept wondering... in the first example "You would need to test on average 5.3189045575676 users to discover 85% of UI problems given the occurrence of a problem is 30%." How many users out of those 5 (.3189...) would have experience the same problem so that you may consider it a "30%'s problem"?
I mean, if the same problem occurs to 5 out of 5 of your users I guess that's not the same as if that problem occured to only 1 out of 5, am I right?
Does that problem have to occur to ALL the 5 users or if it happened to just 1 that´s considered enough?
Thank you in advance! 

---

February 2, 2012 | Ivan wrote:

Jeff,
the "Estimate Problem Occurrence (p) then Sample Size" calculator doesn't work now.
Uncaught TypeError: Object [object Object] has no method 'prettyGallery' 

---

June 17, 2009 | Davant G Bryant wrote:

6. Why is population shape of concern when estimating a mean? What does sample size have to do
with it? 

---

April 30, 2008 | Raul R Wells wrote:

Very useful, I could not move the bar, so it reported 50% against a 100% I wished to mark 

---

January 4, 2008 | John Romadka wrote:

It might help if you could give a real world example of how these calculators could be used. Because I think I get it, but an example would make it more concrete. 

---

January 4, 2008 | John Romadka wrote:

Jeff,
I wonder if you could add a little contextual help for the Problem Occurance field (".30"). Specifically, when would I use a number closer to 0 or a number closer to 1. 

---

Post a Comment

Comment:

Your Name:

Your Email Address:

To prevent comment spam, please answer the following :
What is 1 + 5: (enter the number)