Measuring Usability: Graph and Calculator for Confidence Intervals for Task Times

Graph and Calculator for Confidence Intervals for Task Times
by Jeff Sauro | February 6, 2006 :: 1 Related Articles:: 65 Related Questions How useful was this calculator?
Avg. Rating: 50 ( 18 ) | 3 Comments

Page Tags

Tag Name	#	Vote
Confidence Intervals	31
Graph	25
Log Transformation	24
Calculator	23
Geometric Mean	23
Median	23
Non-Normal Data	23
Task Time	23
Normal Curve	12
skewness	12
Hypothesis Test	8
Geometric Standard Deviation	7

New Tag:

This calculator takes raw task times, transforms them using the Natural Logarithm and computes a confidence interval. The values are displayed in the dot-plots graph below.

Enter Times
(One Per Line)

Confidence Level

Load Sample Data

Results
95% Confidence Interval:	( , )	Geometric Mean
		Median:
Arithmetic Mean		Observations
Arithmetic StDev

Reporting Results

Use the low and high values in the Results section as the confidence intervals for the task times. These values correspond to the green-dashed lines in the the graphs. They are the boundaries of the confidence interval. For example, if you entered the raw task times: 55, 65, 75, 62, 45, 135 (in any order) with a 95% confidence level, you would report the following:

Average Time: 68 seconds, 95% CI (46,101)

The arithmetic mean is provided as a point of reference. You'll notice how the geometric mean is lower than the arithmetic mean, this is a symptom of a positively skewed distribution: the more your data is positively skewed the bigger the difference between the two means. If your data is normally distributed, the two values would be almost identical.

One consequence of using the transformed values to derive the confidence interval is that the intervals are not symmetric around the mean. You'll notice in the example above that the margin of error, is 68 + 33 seconds and 68 - 22 seconds. This asymmetry is caused by the nonlinear log transformation.

Raw Values Format
Raw values should be the task times in seconds format (e.g. two minutes 15 seconds = 135). Minutes and seconds separated by a colon will not work (e.g. 2:30 will return an error). You can enter up to 20 values in any order.

Geometric Mean
The geometric mean should be used instead of the arithmetic mean since values from time data are almost always positively skewed. The geometric mean is the exponentiated value of the arithmetic mean of the natural logged values. In English, if you take the raw task times, convert them into logged values, take the average of these values then convert this log average back into raw form(called exponentiate), you have the Geometric Mean. Exponentiate is like taking the anti-log of a logged value. You anti-log a value when the logarithm is base 10, you exponentiate a value when the logarithm is natural or base e (approx 2.71828). The Geometric Standard deviation is a misleading figure, so I don't recommend reporting that with the Geometric Mean, instead, report the confidence intervals.

Why do I need to Transform?

It's nothing personal, but your task time data probably isn't normally distributed. Don't worry, it's normal for task time data not-to be normal (ok, enough with the double use of "normal"). Task time data, like most time data is positively skewed. This skewness comes from two major elements:

There is a natural lower boundary in times (it's physically impossible to complete a task faster than some minimum time).
Some users will take an exceptionally long amount of time to complete the task.

You can see the skewness by plotting your data. Figure 1 below shows raw task times from a large usability study with 49 users. Notice how the times have a lower boundary at around 70 seconds and the long times above 400 seconds. Figure 2 shows what happens with the log-transformation. This transformation takes the longer task times and pulls them in. Also notice how the distribution looks more like a normal "bell-curve."

Most confidence interval formulas need the data to be approximately normally distributed for their endpoints to be accurate. The log-transformation has been found to be one of the best transformations for time data (e.g. Howell p.346) since the means and standard deviations are correlated.

Figure 1: Raw Time

Figure 2: Log-Transformed Times

Should I report the Geometric Mean or Median?

The short answer is report the Geometric Mean. The more complicated and technical answer is : With small samples there is evidence that the Median is less accurate than the mean, since the sample median is not considered an unbiased estimate of the population median (Cordes 1993), (Eisenhart et al 1948). Now, when data are transformed as they are in the above calculations, an interesting situation occurs. The log of the skewed values will create a more symmetric distribution as Figures 1 and 2 show. If the distribution becomes symmetric the mean and the median are identical. Therefore, when we take the mean of the logged values this should roughly equal the raw median. This isn't always the case due to odd or even values affecting the median. Since the Geomtric Mean will rarely be identical to the Median, it's better to use the Geometric Mean, for the same reason the Arithmetic mean is better than the Median with small samples. There is an excellent discussion of this issue here: Summary Statistics:Location & Spread

References

Howell, David C. (2002) "Statistical Methods for Psychology" Fifth Edition. Thomas Learning.

Cordes, Richard (1993) "The Effects of Running Fewer Subjects on Time-on-Task Measures" Internation Journal of Human Computer Interaction 5(4) 393-403.

Eisenhart, C, Deming L, & Martin CS (1948) "On the Aritmetic Mean and Median in Small Samples from the Normal and Certain Non-Normal Populations." Annals of Mathematical Statistics p599

If you'd like an email when a new article or calculator is posted sign up for Email Updates.

Related Articles
Confidence Interval Calculator for a Completion Rate

Related Questions

Ask a Question

	Briefly describe the concept of a confidence interval and provide an example.
	Popcorn kernels take between 100 and 200 seconds to pop. What sample size (number of kernels) would be needed to estimate the true mean seconds to pop with and error of 5 seconds and 95% confidence level?
	If your hypothesis is not correct, can the data collected still be used within a subsequent analysis.
	How would I figure out a 90% and 95% confidence interval for the below informaion? What would be the formula? Can I figure this out in Excel or Megastat? How would I do that? Men Female count 53 47 mean 36,492.92 24,451.51 sample variance 340,313,003.72 154,893,232.30 sample standard deviation 18,447.57 12,445.61 standard error of the mean 2,533.97 1,815.38
	what is the value of z score required for a 70% confidence interval?
	Can you please direct me to a site that would have an illustration of the normal curve that also includes cumulative percentiles, Z-score equivalents, T-score equivalents, SAT scaled score equivalents, etc. thanks
	Annual starting salaries for college graduates with degrees in business administration are generally expected to be between $30,000 and $45,000. Assume that a 95% confidence interval estimate of the population mean annual starting salary is desired. What is the planning value for the population standard deviation (0 decimals)?
	A researcher expects the population proportion of the Cubs Fans in Chicago to be 80%. Error of less than 5% confident of an estimate to be made from a mail survey. What is the sample size required?
	A sample of PVC pipes coming off a production line were tested for pipe diameter size. The statistical results (in millimeters) were: Mean 300 Median 300 Mode 300 Standard Deviation 15 Range 90 Number in Sample 100 a) According to the Normal Rule, what percent of the pipes had diameters between 285 and 315? b) What would the range of pipe diameters need to be in order to capture 95% of the pipe diameters?
	In 1992, the FAA conducted 86,991 pre-employment drug tests on job applicants who were to be engaged in safety and security-related jobs, and found that 1,143 were positive. (a) Construct a 95 percent confidence interval for the population proportion of positive drug tests. (b) Why is the normality assumption not a problem, despite the very small value of p?
	Philadelphia is conducting a study on the characteristics of tourists who drive to Eagles football games. Previous studies indicate that approximately 70% of all game attendees are people who decided to drive from out of town. If the researcher leading the study desires a 99% confidence level and an interval range of plus or minus 10%, what size should the sample be?
	A random sample of 10 miniature Tootsie Rolls was taken from a bag. Each piece was weighed on very accurate scale. The results in grams were 3.087 3.131 3.241 3.241 3.270 3.353 3.400 3.411 3.437 3.477 (a)Construct a 90% confidence intervalfor the true mean weight. (b)What sample size would be necessary to estimate the true weight an error of +/- 0.03 gram with 90% confidence?
	An engineer in an automotive factory wishes to know what the tire pressure is on all cars leaving the factory. She measures the tire pressure on a sample of 10 randomly selected cars as they are about to leave the plant, in psi. The results are: 32.1 32.3 32.0 30.9 31.5 32.4 32.9 33.1 32.2 31.4 Calculate a 95% Confidence Interval on these numbers
	A central university has a student population of 60,000. The university is interested in determining what proportion of them is in favour of a new grading system. Determine a sample size with confidence level of 95% that will show the true proportion of population in favour of the new system within plus and minus 0.02.
	What are the 5 steps involved in hypothesis testing using the traditional/classical method; Can you use a simple real world example to explain it to me, becaue I am really not getting it.
	Why don't statisticians calculate 100% confidence intervals?
	What percent of the time should this occur? (Z-Score) ?
	The width of a confidence estimate for a proportion will be: a. Narrower for a 99% confidence interval than for a 95% confidence interval. b. Wider for a sample size of 100 than for a sample size of 50. c. Narrower for 90% confidence than for 95% confidence d. Narrower when the sample proportion is .50 than when the sample proportion is .20.
	How do you make data normally distributed?
	out of a random sample 167 students pass an exam out of the 300. how do you calculate an exact 99% confidence interval for the proportion of sudents who passed the exam?
	what are the key terms in a verbal hypothesis that signify whether you are conducting a one-tailed or two-test? Give an example and explain your answer.
	For each of the following situations, indicate whether an error has occurred and, if so, indicate what kind of error (Type I or Type II) has occurred. a We do not reject H0 and H0 is true. b We reject H0 and H0 is true. c We do not reject H0 and H0 is false. d We reject H0 and H0 is false.
	Jeff: Can you possibly share the actual formula you are using in this application to produce the Adjusted Wald calculations? Thanks
	Use the given degree of confidence and sample data to construct a confidence interval for the population mean x. Assume that the population has a normal distribution. The principal randomly selected six students to take an aptitude test. Their scores were: 77.9 89.1 80.7 78.6 74.4 82.0 Determine a 90 percent confidence interval for the mean score for all students. 76.36 < x < 84.54 84.64 < x < 76.26 76.26 < x < 84.64 84.54 < x < 76.36
	You survey a sample of 10 college students to measure their racial prejudice using a scale of 0 (no prejudice) to 100 (extreme prejudice). Here are the scores you obtained. 10 43 30 30 45 82 12 26 24 35 Compute the mode, median, mean, and standard deviation for this sample. Is there a skew; and if so, is it positive or negative? What does this skew reveal about the dispersion in the sample.
	As an experiment, a self-confessed connoisseur of cheap popcorn carefully counted 773 kernels and put them in a popper. After popping, the unpopped kernels were counted. There were 86. a)Construct a 90% confidence interval for the proportion of all kernels that would not pop. b)Check the normality assumption. c) Try the Very Quick Rule. Does it work well here? Why, or why not? d) Why might this sample not be typical?
	When calculating confidence interval estimates...how is the x and standard deviation calculated?
	Which type of data set can the mean not be calculated?
	A random sample of 25 households finds that an average of 2.3 people reside in each house (the standard deviation is 0.35). With a 95% confidence level, what is your estimation of the population average?
	If 350 respondents out of a random sample of 1,000 Americans reported that they did not trust their government, what is your estimation at a 99% confidence level of the proportion of the American population who do not trust their government?
	What is the Z score of a 98% confidence interval?
	What is the Trimmed Mean For?
	Calculate the 95% confidence interval on the following GPA's from 30 randomly selected students. 0.979 0.891 0.962 0.858 0.909 0.936 0.963 0.903 0.914 0.925 0.867 0.888 0.735 0.897 0.851 0.776 0.999 0.967 0.503 0.711 0.963 0.943 0.396 0.951 0.747 0.933 0.909 0.583 0.95 0.756
	Find the margin or error for the 95% confidence interval used to estimate the population proportion. In a survey of 7100 TV viewers, 38% said they watch network news programs.
	How can I compute a one-sided 97.5% confidence interval using SPSS for this ? IN a cohort of 121 eyes treated with drug A, 3 eyes experience a drug related side effect, i.e 3/121. Thanks
	Can the bell curve probability value be over the value of 1?
	IQ tests generally have scores in which the mean is 100, and the standard deviation is 16. If your IQ score was 134, what percentage of the population would have a higher IQ than you?
	A sample of the math test scores of 35 fourth-graders has a mean of 82 with a standard deviation of 15. Find the 95% confidence interval of the mean math test scores of all fourth-graders. Find the 99% confidence interval of the mean math test scores of all fourth-graders. Which interval is larger? Explain why.
	Biting an unpopped kernel of popcorn hurts! As an experiment, a self-confessed connoisseur of cheap popcorn carefully counted 773 kernels and put them in a popper. After popping, the unpopped kernels were counted. There were 86. (a) Construct a 90 percent confidence interval for the proportion of all kernels that would not pop. (b) Check the normality assumption. (c) Try the Very Quick Rule. Does it work well here? Why, or why not? (d) Why might this sample not be typical?
	My data set is non-normal (cycle time). Can I convert the data into "Good and Bad" based on a specification and then find a Z value to estimate my sigma level ?
	Suppose you are planning a sample of employees to determine the monthly average # of vacation days. Standards set: Confidence level of 99% and an error of less than 5 units. Standard deviation be 6 units. What would be the required sample size?
	A telescope manufacturer wants its telescopes to have standard deviations in resolution to be significantly below 2 when focusing on objects 500 light-years away. When a telescope is used to focus on an object 500 light years away 30 times, the sample standard deviation turns out to be 1.46. a.State explicit null and alternate hypotheses b.Test your hypothesis at the �=0.01 level.
	I am trying to do A/B testing onweb page displays. One is the test the other is the control. Could you help me determine the formula for sample size at a 95% confidence level? My historical Conversion Rate (success): 11% Visitors: 280 per day I would like to see a 2% improvement in my conversion rate.
	A sample of 20 pages was taken without replacement from the 1,591-page phone directory Ameritech Pages Plus Yellow Pages. On each page, the mean area devoted to display ads was measured (a display ad is a large block of multicolored illustrations, maps, and text). The data (in square millimeters) are shown below: 0 260 356 403 536 0 268 369 428 536 268 396 469 536 162 338 403 536 536 130 (a) Construct a 95 percent confidence interval for the true mean. (b) Why might normality be an issue here? (c) What sample size would be needed to obtain an error of �10 square millimeters with 99 percent confidence? (d) If this is not a reasonable requirement, suggest one that is. I am new at this and it would help if you could give me the formula and break it down step by step so I can understand. Thanks
	When would you use a hypothesis test for the difference in two population proportions at your place of employment, in your education, or in politics?
	The Web-based company Oh Baby! Gifts has a goal of processing 95 percent of its orders on the same day they are received. If 485 out of the next 500 orders are processed on the same day, would this prove that they are exceeding their goal, using � = .025? (See story.news.yahoo.com accessed June 25, 2004.)
	Calculate a 98% confidence interval for the following data 15.7,15.7,15.5,15.2,15.2,15.1,15.3.
	I am comparing 2 proportions of patients (before and after the introduction of a new quality-of-care project). The project is expected to reduce the proportion of patients with a particular disease. Is a one-sided test of significance enough to prove that?
	Twenty students randomly assigned to an experimental group receive an instructional program; 30 in a second group do not. After 6 months, both groups are tested on their knowledge. The experimental group has a mean of 38 on the test (with an estimated population standard deviation of 3); the control group has a mean of 35 (with an estimated population standard deviation of 5). Using the .05 level, what should the experimenter conclude? (a)Use the steps of hypothesis testing, (b) sketch the distributions involved, and (c) explain your answer to someone who is familiar with the t test for a single sample, but not with the t test for independent means.
	In a survey of 500, 60% responded positively to an value question. Calculate a confidence level at 95% to get an interval estimate for proportion?
	I am charged with sampling expense report submissions for accuracy. We get 8000 T&E claims, and want to sample a subset making inferences about the population. I can probably get a good estimate of the populations' SD, how would i calculate required sample size? I think i would be measuring the delta between actual and claimed - the majority be 0. Is this a one sided issue? Thanks in advance - FT Also, would i need the acceptable level first? ie. we would accept an average difference of $5? How can i work backwards if the first sample size yields an average of $1? (if that made sense)
	An engineer in an automotive factory wishes to know what the tire pressure is on all cars leaving the factory. She measures the tire pressure on a sample of 10 randomly selected cars as they are about to leave the plant, in psi. The results are: 32.1 32.3 32.0 30.9 31.5 32.4 32.9 33.1 32.2 31.4 Calculate a 95% Confidence Interval on these numbers
	Based on information obtained from a sample of 54, a 98% confidence interval for the average profit level of regional banks is given by 67.4 million to 87.78 million. Determine the sample standard deviation of profit
	Assume that the heights of 5 year old boys are normally distribued with a mean of 100cm and a standard deviation of 60. What is the sampling distribution of the mean for a sample size of 900 and its confidence interval at the 99% level?
	Find a confidence interval for the mean assuming that each sample is from a normal population. Mean = 127, s = 27, n = 16. Find the 90% Confidence Interval.
	Your company asks you to compare a new advertising campaign and the old one. Sample data on the accounts as follows: Old: 40, 28, 35, 38, 31, 42, 26, 44, 29, 43 New: 29, 26, 31, 26, 28, 31, 19, 21, 27, 30 Calculate the mean, median, and mode. Calculate variance and standard deviation for each set. Calculate a 95% Confidence Interval for the two sets.
	in a sample of 1000 tv vieweres 330 watched a particlar programme. find 99 pecent confidence limits for tv viewers who watched this programme
	Determine the critical value Za/2 that corresponds to 94% level of confidence Compute the 90% confidence interval about m if the sample size, n, is 55. How does does increasing the sample size affect the margin of error, E?
	Administrative staff based at a business school in the UK are advised to take a 15 minute break from their personal computer after working on it for 90 minutes continuously. A random sample of 36 staff revealed that, one average a break was taken after 97.4 minutes of continuous use. The corresponding standard deviation was measured at 5.1 minutes. a. Provide a 99% confidence interval for the population mean time for working on a PC continuously, prior to taking a break. b. Conduct a suitable test at the 1% level of significance, to see whether staff, on average, are working longer than advised.
	A machine produces 3 inch nails. A sample of 100 nails is selected, and it is found that 25 are shorter than 3 inches. Find the 95% confidence interval on the proportion of all such nails that are shorter than 3 inches.
	how to calculate one sided 95% confidence limits for a proportion.could you provide a formula
	A random sample of n=64 children of working mothers showed that they were absent from school an average of 5.3 days per term, with a standard deviation of 1.8 days. Provide a 96% confidence interval for the average number of days absent for all students.
	If you have a normal distribution of a random variable x, with a mean of 56 and standard deviation of 8: What is the probability that the random variable x will be within plus or minus 1 standard deviation of the mean?
	A sample of households that subscribe to the United Bell Phone Company revealed the following numbers of calls received by each household last week. Determine the mean, median, and standard deviation of the number of calls received. 52, 43, 30, 38, 30, 42, 12, 46, 39, 37, 34, 46, 32, 18, 41, 5
	In 1992, the FAA conducted 86,991 pre-employment drug tests on job applicants who were to be engaged in safety and security relatd jobs and found that 1,143 were positive a. construct a 95% confidence interval for the population proportion of positive drug test. b. why is the normality assumption not a problem, despite the very small value?

Ask a Question

April 30, 2008 | Wanda Brantley wrote:

Great calculator. I use it to double check my answers at work all the time

January 28, 2008 | Jeff Sauro wrote:

Good question. The Confidence Level Tutorial should provide some guidance. In general, if you're not sure, leave it at 95%. For exploratory data analysis where you're looking for patterns, feel free to go to 90 or 85%.

December 4, 2007 | John Romadka wrote:

It would be nice to have a little more info about when/why to change the confidence level from the default 95%.

Comments
Name
Email Address Not Published
To prevent comment spam, please answer the following question before submitting (tags not permitted) : What is 5 + 3: (enter the number)