Measuring Usability: Sample Size Calculator for Discovering Problems in a User Interface

Sample Size Calculator for Discovering Problems in a User Interface
by Jeff Sauro | October 1, 2006 :: 19 Related Questions How useful was this calculator?
Avg. Rating: 64 ( 15 ) | 3 Comments

Page Tags

Tag Name	#	Vote
Calculator	10
Binomial Probability	8
Sample Size	7
UI Problems	7
binomial	2

New Tag:

Sample Size from an estimate of Problem Occurrence (p)

If the probability of detecting a UI problem is known in advance, use this portion of the calculator to estimate the total number of users needed to uncover on average the specified percentage of problems (e.g. 90%). The calculator is based on the binomial probability formula.

Input

Discover of all Problems.

Problem Occurrence (values between 0 and 1)

Results

Estimate Problem Occurrence (p) then Sample Size
This portion of the calculator first builds an estimate of the probability of detecting a UI problem (from sample data). It then produces an estimate of the number of users needed to discover the specified percent of total problems. It uses the Good-Turing and Normalization procedure as outlined by Lewis (2001) and further discussed in (Turner, Lewis & Nielsen 2006).

Input

Results

References
Lewis, James (2001) "Evaluation of Procedures for Adjusting Problm-Discovery Rates Estimated from Small Samples" in The International Journal of Human-Computer Interaction 13(4) p. 445-479

Turner, C. W., Lewis, J. R., and Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (Ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084-3088). Boca Raton, FL: CRC Press.

If you'd like an email when a new article or calculator is posted sign up for Email Updates.

Related Questions

Ask a Question

	How do you determine the sample size for data for which the mean and standard deviation are not known?
	In 1992, the FAA conducted 86,991 pre-employment drug tests on job applicants who were to be engaged in safety and security relatd jobs and found that 1,143 were positive a. construct a 95% confidence interval for the population proportion of positive drug test. b. why is the normality assumption not a problem, despite the very small value?
	A central university has a student population of 60,000. The university is interested in determining what proportion of them is in favour of a new grading system. Determine a sample size with confidence level of 95% that will show the true proportion of population in favour of the new system within plus and minus 0.02.
	The Web-based company Oh Baby! Gifts has a goal of processing 95 percent of its orders on the same day they are received. If 485 out of the next 500 orders are processed on the same day, would this prove that they are exceeding their goal, using � = .025? (See story.news.yahoo.com accessed June 25, 2004.)
	A machine cuts circular filters from large rolls of material. If 7.3% of the filters fail to meet specifications, use the normal approximation to the binomial to compute the probability that a sample of 100 of the filters will contain 5 or fewer that fail to meet specifications.
	My question is really about sample size. Say you work at a facility and want to perform an assessment of your safety culture � this would involve multiple topics of questions expecting answers like agree, tend to agree, not sure, tend to disagree, disagree. How would you estimate the sample size if your total facility population is only 80 persons? Would the manner in which you estimate the sample size be different if you used any combination of the following methods to conduct the assessment: a written survey, individual interviews, one on one observations, or focus groups? What if these assessments must be performed at eight different facilities that have no relationship to each other and each of their total populations range from 15-1000, with a mean of 153? To make it even more complicated, would the manner in which you estimate sample size change if you really wanted each of the facilities to assess eight work groups or divisions in the workforce (e.g., management, operations, maintenance, engineering, etc.) at each of their facilities? Greatly appreciate your input.
	Dear Stats Person, Here is my question. If a person were to guess a number from 1-10, the chance of guessing a 4 would be 0.1. If 5 people were asked to guess a number from 1-10, the chance of guessing a 4 would be 0.5. The chance of 2 people guessing a 4 would be (0.1)(01.)=(0.01), correct? Shouldn't the total number of people also be accounted for? That is what is the chance of 2 people guessing a 4 out of 8 people? or 2 people guessing a 4 out of 20 people guessing? What about the chance of 3 people guessing 4 out of 50 people guessing? Is there a formula for this? Thank you. Diane
	I am charged with sampling expense report submissions for accuracy. We get 8000 T&E claims, and want to sample a subset making inferences about the population. I can probably get a good estimate of the populations' SD, how would i calculate required sample size? I think i would be measuring the delta between actual and claimed - the majority be 0. Is this a one sided issue? Thanks in advance - FT Also, would i need the acceptable level first? ie. we would accept an average difference of $5? How can i work backwards if the first sample size yields an average of $1? (if that made sense)
	How to determine the sample size for comparing mulltiple parameters like Height , weight, Blood pressure , Blood parameters like blood glucose, total cholesterol, etc in two different populations?
	When you calculate CIs for time on task, I notice that you use the critical value from the t-distribution (TINV(0.05, degrees_of_freedom)). But when you calculate CIs for task completion rate (using the Adjusted Wald Method), you use the critical value from the z-distribution (1.96). Why don't you use the same value in both?
	Suppose that in NY State, 35% of voters are Democrats, 30% are Republicans, and 11% are independent. You are conducting a poll by randomly calling registered voters. In your first four calls, what is the probability that you talk to: a)all Republicans? b)exactly 2 Republicans? c)no Independents? d)all Independents? e)at least one independent?
	in a random sample of 200 person of a town 120 are found to the tea drinker.In a random sample of 500 person of another town 240 are found to be tea drinker.Is the proprtion of tea drinker in the two towns are equal? use .001 level significance.
	A doctor knows from experience that 20% of the patients to whom he gives a high blood pressure drug will have undesirable side effects. Assume the doctor gives that drug to ten of his patients. Find the probability that among the ten patients to whom he gives the drug, at most two will have undesirable side effects.
	State the main points of the Central Limit Theorem for a mean. B. Why is population shape of concern when estimating a mean? What does sample size have to do with it?
	A surgical technique is performed on eight patients. You are told that there is an 80% chance of success. a) Construct a binomial distribution. b) Find the mean, variance, and standard deviation of the probability distribution. c) Find the probability that the surgury is successful for exactly six patients. d) Find the probability that the surgury is successful for fewer than six patients
	Popcorn kernels take between 100 and 200 seconds to pop. What sample size (number of kernels) would be needed to estimate the true mean seconds to pop with and error of 5 seconds and 95% confidence level?
	In Dallas, some fire trucks were painted yellow (instead of red) to heighten their visibility. During a test period, the fleet of red fire trucks made 153,348 runs and had 20 accidents, while the fleet of yellow fire trucks made 135,035 runs and had 4 accidents. At � = .01, did the yellow fire trucks have a significantly lower accident rate? (a) State the hypotheses. (b) State the decision rule and sketch it. (c) Find the sample proportions and z test statistic. (d) Make a decision. (e) Find the p-value and interpret it. (f) If statistically significant, do you think the difference is large enough to be important? If so, to whom, and why? (g) Is the normality assumption fulfilled? Explain.
	A local bakery determines that the probability of a customer ordering a blueberry muffin is 50%. 5 customers come into the bakery. A. What is the probability that 3 out of the 5 order a blueberry muffin? B. Explain why this scenario would meet the criteria of a binomial distribution. A random sample of 100 students who took a statistics exam were asked their score on the exam. The mean score on the test was 50; the standard deviation was 10. The scores are normally distributed. A) What percent of the students scored below 30? B) What percent scored between 30 and 55?
	A random sample of 10 miniature Tootsie Rolls was taken from a bag. Each piece was weighed on very accurate scale. The results in grams were 3.087 3.131 3.241 3.241 3.270 3.353 3.400 3.411 3.437 3.477 (a)Construct a 90% confidence intervalfor the true mean weight. (b)What sample size would be necessary to estimate the true weight an error of +/- 0.03 gram with 90% confidence?

Ask a Question

April 30, 2008 | Raul R Wells wrote:

Very useful, I could not move the bar, so it reported 50% against a 100% I wished to mark

January 4, 2008 | John Romadka wrote:

It might help if you could give a real world example of how these calculators could be used. Because I think I get it, but an example would make it more concrete.

January 4, 2008 | John Romadka wrote:

Jeff, I wonder if you could add a little contextual help for the Problem Occurance field (".30"). Specifically, when would I use a number closer to 0 or a number closer to 1.

Comments
Name
Email Address
To prevent comment spam, please answer the following question before submitting (tags not permitted) : What is 1 + 1: (enter the number)