Measuring Usability: Sample Size Calculator for Discovering Problems in a User Interface

Sample Size Calculator for Discovering Problems in a User Interface
by Jeff Sauro | October 1, 2006 :: 36 Related Questions How useful was this calculator?
Avg. Rating: 61 ( 44 ) | 4 Comments

Page Tags

Tag Name	#	Vote
Calculator	15
UI Problems	14
Binomial Probability	13
Sample Size	12
binomial	9

New Tag:

Sample Size from an estimate of Problem Occurrence (p)

If the probability of detecting a UI problem is known in advance, use this portion of the calculator to estimate the total number of users needed to uncover on average the specified percentage of problems (e.g. 90%). The calculator is based on the binomial probability formula.

Input

Discover of all Problems.

Problem Occurrence (values between 0 and 1)

Results

Estimate Problem Occurrence (p) then Sample Size
This portion of the calculator first builds an estimate of the probability of detecting a UI problem (from sample data). It then produces an estimate of the number of users needed to discover the specified percent of total problems. It uses the Good-Turing and Normalization procedure as outlined by Lewis (2001) and further discussed in (Turner, Lewis & Nielsen 2006).

Input

Results

References
Lewis, James (2001) "Evaluation of Procedures for Adjusting Problm-Discovery Rates Estimated from Small Samples" in The International Journal of Human-Computer Interaction 13(4) p. 445-479

Turner, C. W., Lewis, J. R., and Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (Ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084-3088). Boca Raton, FL: CRC Press.

If you'd like an email when a new article or calculator is posted sign up for Email Updates.

Related Questions

Ask a Question

	I am charged with sampling expense report submissions for accuracy. We get 8000 T&E claims, and want to sample a subset making inferences about the population. I can probably get a good estimate of the populations' SD, how would i calculate required sample size? I think i would be measuring the delta between actual and claimed - the majority be 0. Is this a one sided issue? Thanks in advance - FT Also, would i need the acceptable level first? ie. we would accept an average difference of $5? How can i work backwards if the first sample size yields an average of $1? (if that made sense)
	A local bakery determines that the probability of a customer ordering a blueberry muffin is 50%. 5 customers come into the bakery. A. What is the probability that 3 out of the 5 order a blueberry muffin? B. Explain why this scenario would meet the criteria of a binomial distribution. A random sample of 100 students who took a statistics exam were asked their score on the exam. The mean score on the test was 50; the standard deviation was 10. The scores are normally distributed. A) What percent of the students scored below 30? B) What percent scored between 30 and 55?
	Popcorn kernels take between 100 and 200 seconds to pop. What sample size (number of kernels) would be needed to estimate the true mean seconds to pop with and error of 5 seconds and 95% confidence level?
	State the main points of the Central Limit Theorem for a mean. B. Why is population shape of concern when estimating a mean? What does sample size have to do with it?
	A sample of 20 pages was taken without replacement from the 1,591-page phone directory Ameritech Pages Plus Yellow Pages. On each page, the mean area devoted to display ads was measured (a display ad is a large block of multicolored illustrations, maps, and text). The data (in square millimeters) are shown below: 0 260 356 403 536 0 268 369 428 536 268 396 469 536 162 338 403 536 536 130 (a) Construct a 95 percent confidence interval for the true mean. (b) Why might normality be an issue here? (c) What sample size would be needed to obtain an error of �10 square millimeters with 99 percent confidence? (d) If this is not a reasonable requirement, suggest one that is. I am new at this and it would help if you could give me the formula and break it down step by step so I can understand. Thanks
	A machine cuts circular filters from large rolls of material. If 7.3% of the filters fail to meet specifications, use the normal approximation to the binomial to compute the probability that a sample of 100 of the filters will contain 5 or fewer that fail to meet specifications.
	In a national wide survey a researcher expects 30 percent of the population will agree with an value statement. He wishes to have less than 2% error and 95% confident. What sample size is needed??
	Dear Stats Person, Here is my question. If a person were to guess a number from 1-10, the chance of guessing a 4 would be 0.1. If 5 people were asked to guess a number from 1-10, the chance of guessing a 4 would be 0.5. The chance of 2 people guessing a 4 would be (0.1)(01.)=(0.01), correct? Shouldn't the total number of people also be accounted for? That is what is the chance of 2 people guessing a 4 out of 8 people? or 2 people guessing a 4 out of 20 people guessing? What about the chance of 3 people guessing 4 out of 50 people guessing? Is there a formula for this? Thank you. Diane
	My question is really about sample size. Say you work at a facility and want to perform an assessment of your safety culture � this would involve multiple topics of questions expecting answers like agree, tend to agree, not sure, tend to disagree, disagree. How would you estimate the sample size if your total facility population is only 80 persons? Would the manner in which you estimate the sample size be different if you used any combination of the following methods to conduct the assessment: a written survey, individual interviews, one on one observations, or focus groups? What if these assessments must be performed at eight different facilities that have no relationship to each other and each of their total populations range from 15-1000, with a mean of 153? To make it even more complicated, would the manner in which you estimate sample size change if you really wanted each of the facilities to assess eight work groups or divisions in the workforce (e.g., management, operations, maintenance, engineering, etc.) at each of their facilities? Greatly appreciate your input.
	A central university has a student population of 60,000. The university is interested in determining what proportion of them is in favour of a new grading system. Determine a sample size with confidence level of 95% that will show the true proportion of population in favour of the new system within plus and minus 0.02.
	In 1992, the FAA conducted 86,991 pre-employment drug tests on job applicants who were to be engaged in safety and security relatd jobs and found that 1,143 were positive a. construct a 95% confidence interval for the population proportion of positive drug test. b. why is the normality assumption not a problem, despite the very small value?
	Can you give me a formula to calculate sample size?
	A robot, when working properly, produces 5% or less defective items. whenever the robot produces more than 5% defective items, i needs repair. A random sample of 300 items taken from the production line contained 27 defective items. Test at 1% significance level whether or not the robot needs repairing
	The Web-based company Oh Baby! Gifts has a goal of processing 95 percent of its orders on the same day they are received. If 485 out of the next 500 orders are processed on the same day, would this prove that they are exceeding their goal, using � = .025? (See story.news.yahoo.com accessed June 25, 2004.)
	A researcher is interested in estimating the average salary of fire fighters in a large city. He wants to be 95% confident that his estimate is correct. If the standard deviation is $1050, how large a sample is needed to get the desired information and to be accurate within $200?
	How to determine the sample size for comparing mulltiple parameters like Height , weight, Blood pressure , Blood parameters like blood glucose, total cholesterol, etc in two different populations?
	in a random sample of 200 person of a town 120 are found to the tea drinker.In a random sample of 500 person of another town 240 are found to be tea drinker.Is the proprtion of tea drinker in the two towns are equal? use .001 level significance.
	Suppose you are planning a sample of employees to determine the monthly average # of vacation days. Standards set: Confidence level of 99% and an error of less than 5 units. Standard deviation be 6 units. What would be the required sample size?
	When estimating the mean of a population, how large must the sample be in order that the 95% error margin is 1/8 the standard deviation?
	A surgical technique is performed on eight patients. You are told that there is an 80% chance of success. a) Construct a binomial distribution. b) Find the mean, variance, and standard deviation of the probability distribution. c) Find the probability that the surgury is successful for exactly six patients. d) Find the probability that the surgury is successful for fewer than six patients
	A random sample of 10 miniature Tootsie Rolls was taken from a bag. Each piece was weighed on very accurate scale. The results in grams were 3.087 3.131 3.241 3.241 3.270 3.353 3.400 3.411 3.437 3.477 (a)Construct a 90% confidence intervalfor the true mean weight. (b)What sample size would be necessary to estimate the true weight an error of +/- 0.03 gram with 90% confidence?
	In a survey of 500, 60% responded positively to an value question. Calculate a confidence level at 95% to get an interval estimate for proportion?
	A classical experiment in extrasensory perception (ESP) consists of asking a subject to tell, without looking, the suit of each card in a deck of Zener cards. There are five suits in the deck so the probability of a chance match between the guess and the card is 1/5 or .20. If only chance is operating, you would expect a subject to get 20 matches in 100 guesses (.20 X 100 =20). The standard deviation for this mean of 20 is 4. What is the probability of a subject making 23 or more matches in 100 guesses? What is the probability of 27 or more matches in 100 guesses? Suppose a friend of a friend claimed to have ESP and agreed to sit just on time and guess at 100 cards. Suppose she made 36 matches. Calculate the probability of this many, or more, matches if only chance is at work, and carefully write a conclusion.
	in a study of 11000 car crashes it was found that 5720 of them occurred within 5 miles of home. use a 0.01 significance level to test tha claim that more than 50% of car crashes occur within 5 miles of home.
	An automobile manufacturer wants to estimate the mean gasoline mileage that its customers will obtain with its new compact model. How many sample runs must be performed in order that the estimate be accurate to within 0.25 mpg at 90% confidence? (Assume that � = 2.0.)
	A researcher expects the population proportion of the Cubs Fans in Chicago to be 80%. Error of less than 5% confident of an estimate to be made from a mail survey. What is the sample size required?
	In a survey of adults, 68% thought that DNA test for identifying an individual were very reliable. You randomly select 24 adults and ask each if he or she thinks DNA test for identifying an individual are very reliable. a. Decide whether or not you can use the normal distribution to approximate the binomial distribution in this case. If so, find the mean and standard deviation, If not, explain why. b. Find the probability that at most 15 people say DNA tests for identifying an individual are very reliable.
	How do you determine the sample size for data for which the mean and standard deviation are not known?
	In Dallas, some fire trucks were painted yellow (instead of red) to heighten their visibility. During a test period, the fleet of red fire trucks made 153,348 runs and had 20 accidents, while the fleet of yellow fire trucks made 135,035 runs and had 4 accidents. At � = .01, did the yellow fire trucks have a significantly lower accident rate? (a) State the hypotheses. (b) State the decision rule and sketch it. (c) Find the sample proportions and z test statistic. (d) Make a decision. (e) Find the p-value and interpret it. (f) If statistically significant, do you think the difference is large enough to be important? If so, to whom, and why? (g) Is the normality assumption fulfilled? Explain.
	Survey to determine what proportion of new car buyers continue to have their car serviced at the dealership after warranty ends. Estimates 30% of customer do so. Results should be accurate within 5%. Also 95% confident of the results. What sample size is necessary?
	Annual starting salaries for college graduates with degrees in business administration are generally expected to be between $30,000 and $45,000. Assume that a 95% confidence interval estimate of the population mean annual starting salary is desired. What is the planning value for the population standard deviation (0 decimals)? __________ How large a sample should be taken if the desired margin of error is as shown below (0 decimals)? a. $500? __________ b. $200? __________ c. $100? __________ d. Would you recommend trying to obtain the $100 margin of error? _________________
	When you calculate CIs for time on task, I notice that you use the critical value from the t-distribution (TINV(0.05, degrees_of_freedom)). But when you calculate CIs for task completion rate (using the Adjusted Wald Method), you use the critical value from the z-distribution (1.96). Why don't you use the same value in both?
	A central university has a student population of 60,000. The university is interested in determining what proportion of them is in favour of a new grading system. Determine a sample size with confidence level of 95% that will show the true proportion of population in favour of the new system within plus and minus 0.02.
	Philadelphia is conducting a study on the characteristics of tourists who drive to Eagles football games. Previous studies indicate that approximately 70% of all game attendees are people who decided to drive from out of town. If the researcher leading the study desires a 99% confidence level and an interval range of plus or minus 10%, what size should the sample be?
	Suppose that in NY State, 35% of voters are Democrats, 30% are Republicans, and 11% are independent. You are conducting a poll by randomly calling registered voters. In your first four calls, what is the probability that you talk to: a)all Republicans? b)exactly 2 Republicans? c)no Independents? d)all Independents? e)at least one independent?
	A doctor knows from experience that 20% of the patients to whom he gives a high blood pressure drug will have undesirable side effects. Assume the doctor gives that drug to ten of his patients. Find the probability that among the ten patients to whom he gives the drug, at most two will have undesirable side effects.

Ask a Question

June 17, 2009 | Davant G Bryant wrote:

6. Why is population shape of concern when estimating a mean? What does sample size have to do with it?

April 30, 2008 | Raul R Wells wrote:

Very useful, I could not move the bar, so it reported 50% against a 100% I wished to mark

January 4, 2008 | John Romadka wrote:

It might help if you could give a real world example of how these calculators could be used. Because I think I get it, but an example would make it more concrete.

January 4, 2008 | John Romadka wrote:

Jeff, I wonder if you could add a little contextual help for the Problem Occurance field (".30"). Specifically, when would I use a number closer to 0 or a number closer to 1.

Comments
Name
Email Address Not Published
To prevent comment spam, please answer the following question before submitting (tags not permitted) : What is 5 + 2: (enter the number)