Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Graph and Calculator for Confidence Intervals for Task Times

Jeff Sauro • February 6, 2006

This calculator takes raw task times, transforms them using the Natural Logarithm and computes a confidence interval. The values are displayed in the dot-plots graph below. You can also download an excel version of this calculator.

Enter Times
(One Per Line)

Confidence Level



Load Sample Data
 
Results
95% Confidence Interval: ( , ) Geometric Mean
    Median:
Arithmetic Mean Observations
Arithmetic StDev    

Download an excel version of this calculator.

Reporting Results

Use the low and high values in the Results section as the confidence intervals for the task times. These values correspond to the green-dashed lines in the the graphs. They are the boundaries of the confidence interval. For example, if you entered the raw task times: 55, 65, 75, 62, 45, 135 (in any order) with a 95% confidence level, you would report the following:
  • Average Time: 68 seconds, 95% CI (46,101)
The arithmetic mean is provided as a point of reference. You'll notice how the geometric mean is lower than the arithmetic mean, this is a symptom of a positively skewed distribution: the more your data is positively skewed the bigger the difference between the two means. If your data is normally distributed, the two values would be almost identical.

One consequence of using the transformed values to derive the confidence interval is that the intervals are not symmetric around the mean. You'll notice in the example above that the margin of error, is 68 + 33 seconds and 68 - 22 seconds. This asymmetry is caused by the nonlinear log transformation.

Raw Values Format
Raw values should be the task times in seconds format (e.g. two minutes 15 seconds = 135). Minutes and seconds separated by a colon will not work (e.g. 2:30 will return an error). You can enter up to 20 values in any order.


Geometric Mean
The geometric mean should be used instead of the arithmetic mean since values from time data are almost always positively skewed. The geometric mean is the exponentiated value of the arithmetic mean of the natural logged values. In English, if you take the raw task times, convert them into logged values, take the average of these values then convert this log average back into raw form(called exponentiate), you have the Geometric Mean. Exponentiate is like taking the anti-log of a logged value. You anti-log a value when the logarithm is base 10, you exponentiate a value when the logarithm is natural or base e (approx 2.71828). The Geometric Standard deviation is a misleading figure, so I don't recommend reporting that with the Geometric Mean, instead, report the confidence intervals.

Why do I need to Transform?

It's nothing personal, but your task time data probably isn't normally distributed. Don't worry, it's normal for task time data not-to be normal (ok, enough with the double use of "normal"). Task time data, like most time data is positively skewed. This skewness comes from two major elements:

  1. There is a natural lower boundary in times (it's physically impossible to complete a task faster than some minimum time).
  2. Some users will take an exceptionally long amount of time to complete the task.
You can see the skewness by plotting your data. Figure 1 below shows raw task times from a large usability study with 49 users. Notice how the times have a lower boundary at around 70 seconds and the long times above 400 seconds. Figure 2 shows what happens with the log-transformation. This transformation takes the longer task times and pulls them in. Also notice how the distribution looks more like a normal "bell-curve."

Most confidence interval formulas need the data to be approximately normally distributed for their endpoints to be accurate. The log-transformation has been found to be one of the best transformations for time data (e.g. Howell p.346) since the means and standard deviations are correlated.

Figure 1: Raw Time



Figure 2: Log-Transformed Times



Should I report the Geometric Mean or Median?

The short answer is report the Geometric Mean. The more complicated and technical answer is : With small samples there is evidence that the Median is less accurate than the mean, since the sample median is not considered an unbiased estimate of the population median (Cordes 1993), (Eisenhart et al 1948). Now, when data are transformed as they are in the above calculations, an interesting situation occurs. The log of the skewed values will create a more symmetric distribution as Figures 1 and 2 show. If the distribution becomes symmetric the mean and the median are identical. Therefore, when we take the mean of the logged values this should roughly equal the raw median. This isn't always the case due to odd or even values affecting the median. Since the Geomtric Mean will rarely be identical to the Median, it's better to use the Geometric Mean, for the same reason the Arithmetic mean is better than the Median with small samples. There is an excellent discussion of this issue here: Summary Statistics:Location & Spread

References

Howell, David C. (2002) "Statistical Methods for Psychology" Fifth Edition. Thomas Learning.

Cordes, Richard (1993) "The Effects of Running Fewer Subjects on Time-on-Task Measures" Internation Journal of Human Computer Interaction 5(4) 393-403.

Eisenhart, C, Deming L, & Martin CS (1948) "On the Aritmetic Mean and Median in Small Samples from the Normal and Certain Non-Normal Populations." Annals of Mathematical Statistics p599

 

Learn More


UX Bootcamp: Sept 25th-27th in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar

Rate this Calculator

Avg. Rating 55.97 (45)

Poor         Excellent
012345678910

.

Posted Comments

There are 8 Comments

December 2, 2012 | g¨šnstige brautkleider wrote:

Invaluable info and excellent design you got here! I wish to thank you for sharing your ideas and putting the time into the stuff you post! Nice work! 

November 11, 2010 | anonymous wrote:

Horrible 

May 27, 2009 | Siddharth Mahapatra wrote:

If I have used a JOhnson transformation, how do I compute the Confidence interval of the original data. How do I take the inverse of a JOhnson Transformation? 

March 30, 2009 | diego wrote:

very useful indeed. Even if the impression is that the calculation of 95% CI is not really precise. Or is it my mistake? 

February 24, 2009 | Eli D Ehrenpreis MD wrote:

This was a life saver!
I forgot my laptop with my statistics program and needed to recalculate some data to send in a revised manuscript. Thanks for making this available! 

April 30, 2008 | Wanda Brantley wrote:

Great calculator. I use it to double check my answers at work all the time 

January 28, 2008 | Jeff Sauro wrote:

Good question. The Confidence Level Tutorial should provide some guidance. In general, if you're not sure, leave it at 95%. For exploratory data analysis where you're looking for patterns, feel free to go to 90 or 85%. 

December 4, 2007 | John Romadka wrote:

It would be nice to have a little more info about when/why to change the confidence level from the default 95%. 

Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 5 + 1: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[2861 Subscribers]

Connect With Us

Subscribe to RSS

Our Supporters

Use Card Sorting to improve your IA

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 15 journal articles and 4 books on statistics and the user-experience.
More about Jeff...

Measuring The Visual Appeal of Websites 6 days ago

Associating UX Changes to the Net Promoter Score 13 days ago

7 Ways to Find Users for Usability Testing 20 days ago

10 Metrics for Testing Website Navigation 27 days ago

10 Things to Know about Usability Problems

8 Ways to Show Design Changes Improved the User Experience

Does better usability increase customer loyalty?

How to Conduct a Usability test on a Mobile Device

97 Things to Know about Usability

Confidence Interval Calculator for a Completion Rate

Why you only need to test with five users (explained)

How common are usability problems?

5 Examples of Quantifying Qualitative Data

Nine misconceptions about statistics and usability

Should you use 5 or 7 point scales?

A Brief History of the Magic Number 5 in Usability Testing

5 Second Usability Tests

What five users can tell you that 5000 cannot

The Five Most Influential Papers in Usability

How much is a PhD Worth?

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals (JUST RELEASED)

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.
Home | Blog | Calculators | Products | Services | Contact

(303) 578-2801

© 2013 Measuring Usability LLC All Rights Reserved.