Jeff Sauro • September 17, 2004
We already saw how a manageable sample of users can provide meaningful data for discretebinary data like task completion. With continuous data like task times, the sample size can be even smaller.Time (in seconds)  
You  101 
Colleague 1  132 
Colleague 2  125 
Colleague 3  145 
Mean  125.75 
St Deviation  18.46 
Range  44 
To arrive at the elusive "significant" sample size, you need to try a few reasonable sample sizes and see which ones fall within the limits of the confidence interval. The values (n) you choose will affect the the critical value for t and the Standard Error since both use n in their equation. We'll use 25, 20, 15, 10 and 5 and which ever value has a confidence interval at about 10 seconds we'll use as the ideal sample. (Again all this assumes that our internal sample did a good job of determining the standard deviation of the larger population).
Sample  95% CI  SE  SQRT N  Stdev  t * 
25  7.61  3.692  5  18.46  2.063 
20  8.63  4.12  4.47  18.46  2.093 
15  10.22  4.76  3.87  18.46  2.144 
10  13.20  5.83  3.16  18.46  2.262 
5  22.92  8.25  2.23  18.46  2.776 
At about 15 users, the conifdence interval narrows close enough to ten seconds that it will probably be sufficient. I'd use this 15 as the approximate number of users you'd need to sample and know that to get more precise, you'd need to sample more than 15 users. This result is much better than thinking you need to test 100 or 1000 in order to get "statistically significant results. If +/ 10 seconds isn't precise enough you can:
Sample Sizes in the Real World of Usability Testing
If you've run enough usability tests, in many cases your sample size is usually determined ahead of timethat is, you know your budget and time frame and therefore approximately how many users you'll be samplingusually somewhere between 10 and 30. I then approach sampling as getting as many users as I can within that range and then compute the statistics later.


mean time of your sample (126.6)  
true mean time of the entire population of users  
n  number of users in the sample (15) 
s  the standard deviation of the sample (16.33) 
t*  t statistic = (2.144789) or use the excel function =TINV(.05,14) [confidence level(.05) and degrees of freedom n1 (14) ] 
Plugging in the numbers, for the estimated mean of the total population of users on this task we get:
= 126.6 + or  9.08
So when reporting the mean time for this task we would say, "We are 95% confident the mean time is between 117.5 seconds and 135.6 seconds." In this example, our original sample turned out to be a good estimate of the mean time and standard deviation but don't expect that to usually work out so well.The Experiment Requires That You Continue: On The Ethical Treatment of Users
28 Resources for Getting Started In UX
8 Ways to Show Design Changes Improved the User Experience
97 Things to Know about Usability
How to Conduct a Usability test on a Mobile Device
How common are usability problems?
A Brief History of the Magic Number 5 in Usability Testing
The Five Most Influential Papers in Usability
Should you use 5 or 7 point scales?
Why you only need to test with five users (explained)
Does better usability increase customer loyalty?
10 Things to Know about Usability Problems
5 Examples of Quantifying Qualitative Data
Confidence Interval Calculator for a Completion Rate
Quantifying the User Experience: Practical Statistics for User Research The most comprehensive statistical resource for UX Professionals Buy on Amazon  
Excel & R Companion to Quantifying the User Experience Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon  Download  
A Practical Guide to the System Usability Scale Background, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon  Download  
A Practical Guide to Measuring Usability 72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon  Download 