Measuring Usability: Graph and Calculator for Confidence Intervals for Task Times

Graph and Calculator for Confidence Intervals for Task Times
by Jeff Sauro | February 6, 2006 :: 1 Related Articles:: 20 Related Questions How useful was this calculator?
Avg. Rating: 43 ( 12 ) | 3 Comments

Page Tags

Tag Name	#	Vote
Confidence Intervals	20
Graph	15
Log Transformation	15
Calculator	14
Geometric Mean	14
Median	14
Non-Normal Data	14
Task Time	14
Normal Curve	3
skewness	3

New Tag:

This calculator takes raw task times, transforms them using the Natural Logarithm and computes a confidence interval. The values are displayed in the dot-plots graph below.

Enter Times
(One Per Line)

Confidence Level

Load Sample Data

Results
95% Confidence Interval:	( , )	Geometric Mean
		Median:
Arithmetic Mean		Observations
Arithmetic StDev

Reporting Results

Use the low and high values in the Results section as the confidence intervals for the task times. These values correspond to the green-dashed lines in the the graphs. They are the boundaries of the confidence interval. For example, if you entered the raw task times: 55, 65, 75, 62, 45, 135 (in any order) with a 95% confidence level, you would report the following:

Average Time: 68 seconds, 95% CI (46,101)

The arithmetic mean is provided as a point of reference. You'll notice how the geometric mean is lower than the arithmetic mean, this is a symptom of a positively skewed distribution: the more your data is positively skewed the bigger the difference between the two means. If your data is normally distributed, the two values would be almost identical.

One consequence of using the transformed values to derive the confidence interval is that the intervals are not symmetric around the mean. You'll notice in the example above that the margin of error, is 68 + 33 seconds and 68 - 22 seconds. This asymmetry is caused by the nonlinear log transformation.

Raw Values Format
Raw values should be the task times in seconds format (e.g. two minutes 15 seconds = 135). Minutes and seconds separated by a colon will not work (e.g. 2:30 will return an error). You can enter up to 20 values in any order.

Geometric Mean
The geometric mean should be used instead of the arithmetic mean since values from time data are almost always positively skewed. The geometric mean is the exponentiated value of the arithmetic mean of the natural logged values. In English, if you take the raw task times, convert them into logged values, take the average of these values then convert this log average back into raw form(called exponentiate), you have the Geometric Mean. Exponentiate is like taking the anti-log of a logged value. You anti-log a value when the logarithm is base 10, you exponentiate a value when the logarithm is natural or base e (approx 2.71828). The Geometric Standard deviation is a misleading figure, so I don't recommend reporting that with the Geometric Mean, instead, report the confidence intervals.

Why do I need to Transform?

It's nothing personal, but your task time data probably isn't normally distributed. Don't worry, it's normal for task time data not-to be normal (ok, enough with the double use of "normal"). Task time data, like most time data is positively skewed. This skewness comes from two major elements:

There is a natural lower boundary in times (it's physically impossible to complete a task faster than some minimum time).
Some users will take an exceptionally long amount of time to complete the task.

You can see the skewness by plotting your data. Figure 1 below shows raw task times from a large usability study with 49 users. Notice how the times have a lower boundary at around 70 seconds and the long times above 400 seconds. Figure 2 shows what happens with the log-transformation. This transformation takes the longer task times and pulls them in. Also notice how the distribution looks more like a normal "bell-curve."

Most confidence interval formulas need the data to be approximately normally distributed for their endpoints to be accurate. The log-transformation has been found to be one of the best transformations for time data (e.g. Howell p.346) since the means and standard deviations are correlated.

Figure 1: Raw Time

Figure 2: Log-Transformed Times

Should I report the Geometric Mean or Median?

The short answer is report the Geometric Mean. The more complicated and technical answer is : With small samples there is evidence that the Median is less accurate than the mean, since the sample median is not considered an unbiased estimate of the population median (Cordes 1993), (Eisenhart et al 1948). Now, when data are transformed as they are in the above calculations, an interesting situation occurs. The log of the skewed values will create a more symmetric distribution as Figures 1 and 2 show. If the distribution becomes symmetric the mean and the median are identical. Therefore, when we take the mean of the logged values this should roughly equal the raw median. This isn't always the case due to odd or even values affecting the median. Since the Geomtric Mean will rarely be identical to the Median, it's better to use the Geometric Mean, for the same reason the Arithmetic mean is better than the Median with small samples. There is an excellent discussion of this issue here: Summary Statistics:Location & Spread

References

Howell, David C. (2002) "Statistical Methods for Psychology" Fifth Edition. Thomas Learning.

Cordes, Richard (1993) "The Effects of Running Fewer Subjects on Time-on-Task Measures" Internation Journal of Human Computer Interaction 5(4) 393-403.

Eisenhart, C, Deming L, & Martin CS (1948) "On the Aritmetic Mean and Median in Small Samples from the Normal and Certain Non-Normal Populations." Annals of Mathematical Statistics p599

If you'd like an email when a new article or calculator is posted sign up for Email Updates.

Related Articles
Confidence Interval Calculator for a Completion Rate

Related Questions

Ask a Question

	When calculating confidence interval estimates...how is the x and standard deviation calculated?
	Annual starting salaries for college graduates with degrees in business administration are generally expected to be between $30,000 and $45,000. Assume that a 95% confidence interval estimate of the population mean annual starting salary is desired. What is the planning value for the population standard deviation (0 decimals)?
	What is the Trimmed Mean For?
	I am charged with sampling expense report submissions for accuracy. We get 8000 T&E claims, and want to sample a subset making inferences about the population. I can probably get a good estimate of the populations' SD, how would i calculate required sample size? I think i would be measuring the delta between actual and claimed - the majority be 0. Is this a one sided issue? Thanks in advance - FT Also, would i need the acceptable level first? ie. we would accept an average difference of $5? How can i work backwards if the first sample size yields an average of $1? (if that made sense)
	How do you make data normally distributed?
	Why don't statisticians calculate 100% confidence intervals?
	If you have a normal distribution of a random variable x, with a mean of 56 and standard deviation of 8: What is the probability that the random variable x will be within plus or minus 1 standard deviation of the mean?
	I am trying to do A/B testing onweb page displays. One is the test the other is the control. Could you help me determine the formula for sample size at a 95% confidence level? My historical Conversion Rate (success): 11% Visitors: 280 per day I would like to see a 2% improvement in my conversion rate.
	What is the Z score of a 98% confidence interval?
	How would I figure out a 90% and 95% confidence interval for the below informaion? What would be the formula? Can I figure this out in Excel or Megastat? How would I do that? Men Female count 53 47 mean 36,492.92 24,451.51 sample variance 340,313,003.72 154,893,232.30 sample standard deviation 18,447.57 12,445.61 standard error of the mean 2,533.97 1,815.38
	IQ tests generally have scores in which the mean is 100, and the standard deviation is 16. If your IQ score was 134, what percentage of the population would have a higher IQ than you?
	what is the value of z score required for a 70% confidence interval?
	If 350 respondents out of a random sample of 1,000 Americans reported that they did not trust their government, what is your estimation at a 99% confidence level of the proportion of the American population who do not trust their government?
	A sample of PVC pipes coming off a production line were tested for pipe diameter size. The statistical results (in millimeters) were: Mean 300 Median 300 Mode 300 Standard Deviation 15 Range 90 Number in Sample 100 a) According to the Normal Rule, what percent of the pipes had diameters between 285 and 315? b) What would the range of pipe diameters need to be in order to capture 95% of the pipe diameters?
	in a sample of 1000 tv vieweres 330 watched a particlar programme. find 99 pecent confidence limits for tv viewers who watched this programme
	A random sample of 25 households finds that an average of 2.3 people reside in each house (the standard deviation is 0.35). With a 95% confidence level, what is your estimation of the population average?
	You survey a sample of 10 college students to measure their racial prejudice using a scale of 0 (no prejudice) to 100 (extreme prejudice). Here are the scores you obtained. 10 43 30 30 45 82 12 26 24 35 Compute the mode, median, mean, and standard deviation for this sample. Is there a skew; and if so, is it positive or negative? What does this skew reveal about the dispersion in the sample.
	In 1992, the FAA conducted 86,991 pre-employment drug tests on job applicants who were to be engaged in safety and security relatd jobs and found that 1,143 were positive a. construct a 95% confidence interval for the population proportion of positive drug test. b. why is the normality assumption not a problem, despite the very small value?
	What percent of the time should this occur? (Z-Score) ?
	how to calculate one sided 95% confidence limits for a proportion.could you provide a formula

Ask a Question

April 30, 2008 | Wanda Brantley wrote:

Great calculator. I use it to double check my answers at work all the time

January 28, 2008 | Jeff Sauro wrote:

Good question. The Confidence Level Tutorial should provide some guidance. In general, if you're not sure, leave it at 95%. For exploratory data analysis where you're looking for patterns, feel free to go to 90 or 85%.

December 4, 2007 | John Romadka wrote:

It would be nice to have a little more info about when/why to change the confidence level from the default 95%.

Comments
Name
Email Address
To prevent comment spam, please answer the following question before submitting (tags not permitted) : What is 2 + 3: (enter the number)