Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

What is a Representative Sample Size for a Survey?

Jeff Sauro • July 15, 2010

Will users purchase an upgrade? What features are most desired?  Will they recommend the product to a friend? Part of measuring the user experience involves directly asking users what they think via surveys. The Web has made surveys easy to administer and deliver. It hasn't made the question of how many people you need to survey any easier though. One common question is "How many people do I need to survey to have a representative sample?" This question mixes two concepts: representativeness and sample size.

Representative: Are you asking the right people?

It is much better to ask a few of the right people what they think than a lot of the wrong people. Some things to consider are:
  • Do your respondents have the authority to make purchasing decisions?
  • Are you surveying a user or system administrator?
  • Are only people from North America responding to your survey about international issues?
Representative samples mean asking the people in your population of interest. It has less to do with the right sample size than with the right target. Spend more of your time on identifying, finding and asking the right people than on finding the right sample size. Ideally you can select your respondents from the appropriate population in some random way, although even randomness is less important than representativeness. For example, you could select every third member on a mailing list of representative users.

Sample Size: How Precise do you need to be?

Once you're fairly confident you're talking to the right people, you can ask how many you need to survey. For single-shot surveys when no comparisons are being made (e.g., to prior surveys or benchmarks) sample size is all about how precise you need to be. The more precise you need the estimate the more people you need. The calculation is basically a backwards confidence interval. Confidence intervals are made up of a confidence level (typically 95%) and a margin of error. The margin of error is how precise you need to be.

You can't be 100% precise

The only way to be 100% accurate (a 0% margin of error) with your survey results is to survey everyone. Unless you're the government this is probably not an option. Instead you have to deal with some margin of error. You need to find the highest margin of error you or the survey sponsor is comfortable with. To do this answer the following question:

Is it OK if your response averages could be off by +/-1%?
How about +/- 5% ?
How about +/- 10% ?
How about +/- 20% ?

The point just before you are no longer comfortable with the level of error in your estimate is where you stop. Once you have the largest margin of error you can tolerate, just use the following table to approximate the sample size you need. For more precise estimates you download an Excel Calculator and 23-page guide on how to perform the calculations.

 Margin of Error
(+/-)
 Rating Scale
Sample Size
 Binary (50%)
Sample Size
 1%  6073  9600
 3%  686  1064
 5%  249  381
 10%  64  93
 15%  30  39
 20%  18  21

Example 1 : Rating Scale

For example, if you're asking respondents to rate how satisfied they are with a service on a 7-point scale, and want no more than a 5% margin of error, then you'll need to plan on obtaining 249 responses. Of course not everyone who is asked to fill out a survey responds. A response rate of 10% would mean you should plan on sending the survey to around 2500 people. You can certainly achieve a higher response rate through a more targeted list. In a recent survey I conducted where participants opted-in to a list, I achieved a 60% response rate.

Example 2: Binary Response

If you were asking users whether they would use your service in the future (yes or no) and want a 3% margin of error around the actual percentage, then plan on getting around 1064 responses (meaning you'd need to send it to close to 10,000 people if you have only a 10% response rate).

If your questions are a mix of both rating scales and binary response choices (yes/no agree/disagree), then use the binary sample size because it is always larger than the rating scale sample size.
Sample size calculations are all about balancing precision with cost. When you need to limit your cost you'll need to deal with more uncertainly in your estimate.
The graph below plots the relationship between the margin of error and sample size for both binary and rating scale responses. You can use it to approximate any sample size between margins of error of 1 and 25% or download the excel calculator and PDF guide to get more precise calculations.




If obtaining additional respondents is costly then you'll need to revisit the question of the highest margin of error you or the sponsor is comfortable with or find ways to increase the response rate, such as offering an incentive. Sample size calculations are all about balancing precision with cost. When you need to limit your cost you'll need to deal with more uncertainly in your estimate.

More about Sample Sizes for Surveys

The graph and tables above were calculated using a common formula for finding the sample size. A few assumptions were made based on the typical variability observed with rating scale response patterns such as the confidence level (95%) and the typical amount of variability in a population.  If you'd like more information on how to calculate your own sample size you can download the Survey Sample Size package. It contains a 23-page step-by-step calculation guide with plenty of examples and practice questions--plus an  Excel calculator which find the sample size for both binary responses and rating scales.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Sample Size, Margin of Error
.

Posted Comments

There are 3 Comments

March 30, 2013 | Mark Tonkin wrote:

This is a very interesting and informative web site. Well done!

I especially like the material on the Net Promoter Score. 


June 6, 2011 | anonymous wrote:

Great article, as usual. I wish I didn't have to go to the 'download' page for the 'Survey Sample Size Package' only to discover I have to 'buy' it.
 


January 19, 2011 | Gary Boykin wrote:

Jeff your articles are excellent -- Keep up the good work, thank you 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 5 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[3812 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.