Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Graph and Calculator for Confidence Intervals for Task Times

Jeff Sauro • February 6, 2006

This calculator takes raw task times, transforms them using the Natural Logarithm and computes a confidence interval. The values are displayed in the dot-plots graph below. You can also download an excel version of this calculator.

Enter Times
(One Per Line)

Confidence Level



Load Sample Data
 
Results
95% Confidence Interval: ( , ) Geometric Mean
    Median:
Arithmetic Mean Observations
Arithmetic StDev    

Download an excel version of this calculator.

Reporting Results

Use the low and high values in the Results section as the confidence intervals for the task times. These values correspond to the green-dashed lines in the the graphs. They are the boundaries of the confidence interval. For example, if you entered the raw task times: 55, 65, 75, 62, 45, 135 (in any order) with a 95% confidence level, you would report the following:
  • Average Time: 68 seconds, 95% CI (46,101)
The arithmetic mean is provided as a point of reference. You'll notice how the geometric mean is lower than the arithmetic mean, this is a symptom of a positively skewed distribution: the more your data is positively skewed the bigger the difference between the two means. If your data is normally distributed, the two values would be almost identical.

One consequence of using the transformed values to derive the confidence interval is that the intervals are not symmetric around the mean. You'll notice in the example above that the margin of error, is 68 + 33 seconds and 68 - 22 seconds. This asymmetry is caused by the nonlinear log transformation.

Raw Values Format
Raw values should be the task times in seconds format (e.g. two minutes 15 seconds = 135). Minutes and seconds separated by a colon will not work (e.g. 2:30 will return an error). You can enter up to 20 values in any order.


Geometric Mean
The geometric mean should be used instead of the arithmetic mean since values from time data are almost always positively skewed. The geometric mean is the exponentiated value of the arithmetic mean of the natural logged values. In English, if you take the raw task times, convert them into logged values, take the average of these values then convert this log average back into raw form(called exponentiate), you have the Geometric Mean. Exponentiate is like taking the anti-log of a logged value. You anti-log a value when the logarithm is base 10, you exponentiate a value when the logarithm is natural or base e (approx 2.71828). The Geometric Standard deviation is a misleading figure, so I don't recommend reporting that with the Geometric Mean, instead, report the confidence intervals.

Why do I need to Transform?

It's nothing personal, but your task time data probably isn't normally distributed. Don't worry, it's normal for task time data not-to be normal (ok, enough with the double use of "normal"). Task time data, like most time data is positively skewed. This skewness comes from two major elements:

  1. There is a natural lower boundary in times (it's physically impossible to complete a task faster than some minimum time).
  2. Some users will take an exceptionally long amount of time to complete the task.
You can see the skewness by plotting your data. Figure 1 below shows raw task times from a large usability study with 49 users. Notice how the times have a lower boundary at around 70 seconds and the long times above 400 seconds. Figure 2 shows what happens with the log-transformation. This transformation takes the longer task times and pulls them in. Also notice how the distribution looks more like a normal "bell-curve."

Most confidence interval formulas need the data to be approximately normally distributed for their endpoints to be accurate. The log-transformation has been found to be one of the best transformations for time data (e.g. Howell p.346) since the means and standard deviations are correlated.

Figure 1: Raw Time



Figure 2: Log-Transformed Times



Should I report the Geometric Mean or Median?

The short answer is report the Geometric Mean. The more complicated and technical answer is : With small samples there is evidence that the Median is less accurate than the mean, since the sample median is not considered an unbiased estimate of the population median (Cordes 1993), (Eisenhart et al 1948). Now, when data are transformed as they are in the above calculations, an interesting situation occurs. The log of the skewed values will create a more symmetric distribution as Figures 1 and 2 show. If the distribution becomes symmetric the mean and the median are identical. Therefore, when we take the mean of the logged values this should roughly equal the raw median. This isn't always the case due to odd or even values affecting the median. Since the Geomtric Mean will rarely be identical to the Median, it's better to use the Geometric Mean, for the same reason the Arithmetic mean is better than the Median with small samples. There is an excellent discussion of this issue here: Summary Statistics:Location & Spread

References

Howell, David C. (2002) "Statistical Methods for Psychology" Fifth Edition. Thomas Learning.

Cordes, Richard (1993) "The Effects of Running Fewer Subjects on Time-on-Task Measures" Internation Journal of Human Computer Interaction 5(4) 393-403.

Eisenhart, C, Deming L, & Martin CS (1948) "On the Aritmetic Mean and Median in Small Samples from the Normal and Certain Non-Normal Populations." Annals of Mathematical Statistics p599

 

Rate this Calculator

Avg. Rating 55.97 (45)

Poor         Excellent
012345678910

.

Posted Comments

There are 7 Comments

November 11, 2010 | anonomous wrote:

Horrible 

May 27, 2009 | Siddharth Mahapatra wrote:

If I have used a JOhnson transformation, how do I compute the Confidence interval of the original data. How do I take the inverse of a JOhnson Transformation? 

March 30, 2009 | diego wrote:

very useful indeed. Even if the impression is that the calculation of 95% CI is not really precise. Or is it my mistake? 

February 24, 2009 | Eli D Ehrenpreis MD wrote:

This was a life saver!
I forgot my laptop with my statistics program and needed to recalculate some data to send in a revised manuscript. Thanks for making this available! 

April 30, 2008 | Wanda Brantley wrote:

Great calculator. I use it to double check my answers at work all the time 

January 28, 2008 | Jeff Sauro wrote:

Good question. The Confidence Level Tutorial should provide some guidance. In general, if you're not sure, leave it at 95%. For exploratory data analysis where you're looking for patterns, feel free to go to 90 or 85%. 

December 4, 2007 | John Romadka wrote:

It would be nice to have a little more info about when/why to change the confidence level from the default 95%. 

Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 1 + 2: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[1761 Subscribers]

Connect With Us

Subscribe to RSS

Our Supporters

Use Tree Testing to improve your IA
Use Tree Testing to improve your IA

Loop11 Online Usabilty Testing

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 15 journal articles and 3 books on statistics and the user-experience.
More about Jeff...

Distrust in Social Networks: Google+, Twitter, Facebook 2 days ago

5 Examples of Quantifying Qualitative Data 9 days ago

Comparison of Usability Testing Methods 16 days ago

Salaries for UX Researchers, Designers, Managers and IAs 23 days ago

97 Things to Know about Usability

10 Things to Know about Usability Problems

What five users can tell you that 5000 cannot

25 Resources for Measuring Usability

Confidence Interval Calculator for a Completion Rate

8 Ways to Show Design Changes Improved the User Experience

Top 10 Research-Based Usability Findings of 2010

How common are usability problems?

The Five Most Influential Papers in Usability

Why you only need to test with five users (explained)

Does better usability increase customer loyalty?

Should you use 5 or 7 point scales?

5 Second Usability Tests

How much is a PhD Worth?

A Brief History of the Magic Number 5 in Usability Testing

.

Featured Product

SUS Guide & Calculator Package

System Usability Scale (SUS)
Guide & Calculator Package

.
.
.
Home | Blog | Calculators | Products | Services | Contact

(303) 578-2801

© 2012 Measuring Usability LLC All Rights Reserved.