Jeff Sauro • July 15, 2010

Will users purchase an upgrade? What features are most desired? Will they recommend the product to a friend? Part of measuring the user experience involves directly asking users what they think via surveys. The Web has made surveys easy to administer and deliver. It hasn't made the question of how many people you need to survey any easier though. One common question is "How many people do I need to survey to have a representative sample?" This question mixes two concepts: representativeness and sample size.- Do your respondents have the authority to make purchasing decisions?
- Are you surveying a user or system administrator?
- Are only people from North America responding to your survey about international issues?

Is it OK if your response averages could be off by +/-1%?

How about +/- 5% ?

How about +/- 10% ?

How about +/- 20% ?

The point just before you are no longer comfortable with the level of error in your estimate is where you stop. Once you have the largest margin of error you can tolerate, just use the following table to approximate the sample size you need. For more precise estimates you download an Excel Calculator and 23-page guide on how to perform the calculations.

Margin of Error (+/-) | Rating Scale Sample Size | Binary (50%) Sample Size |

1% | 6073 | 9600 |

3% | 686 | 1064 |

5% | 249 | 381 |

10% | 64 | 93 |

15% | 30 | 39 |

20% | 18 | 21 |

If your questions are a mix of both rating scales and binary response choices (yes/no agree/disagree), then use the binary sample size because it is always larger than the rating scale sample size.

Sample size calculations are all about balancing precision with cost. When you need to limit your cost you'll need to deal with more uncertainly in your estimate.

The graph below plots the relationship between the margin of error and sample size for both binary and rating scale responses. You can use it to approximate any sample size between margins of error of 1 and 25% or download the excel calculator and PDF guide to get more precise calculations.If obtaining additional respondents is costly then you'll need to revisit the question of the highest margin of error you or the sponsor is comfortable with or find ways to increase the response rate, such as offering an incentive. Sample size calculations are all about balancing precision with cost. When you need to limit your cost you'll need to deal with more uncertainly in your estimate.

The New Face of Usability Testing

7 Techniques for Prioritizing Customer Requirements

How to Compute a Confidence Interval in 5 Easy Steps

The Experiment Requires That You Continue: On The Ethical Treatment of Users

97 Things to Know about Usability

Nine misconceptions about statistics and usability

Confidence Interval Calculator for a Completion Rate

8 Ways to Show Design Changes Improved the User Experience

What five users can tell you that 5000 cannot

Does better usability increase customer loyalty?

Should you use 5 or 7 point scales?

10 Things to Know about Usability Problems

A Brief History of the Magic Number 5 in Usability Testing

5 Examples of Quantifying Qualitative Data

The Five Most Influential Papers in Usability

Why you only need to test with five users (explained)

.

Quantifying the User Experience: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.