Now, If you've ever heard the news say a race is too close to call or there's a "statistical tie" it's because the width of both confidence intervals are overlapping enough that there's no clear leader. What does that mean? Imagine that a lot fewer people were surveyed for the poll taken by the Star-Tribune Newspaper and the margin of error was now +/- 6%. This new relationship is displayed in the figure below.

Notice how part of the confidence intervals overlap? The + 6% Margin of Error on the top of John Daniel's 45 % overlaps with the - 6% of Jim Bean's 55%. This means that if the poll were to be taken again, there's a reasonable chance that John Daniels might be leading over Jim Beam in the polls. How great of a chance is there? To know that you need to understand the confidence level.

The Confidence Level

While you may hear the margin of error reported on TV or in the paper, the confidence level is a crucial component that's often left out. Quite simply, the confidence level represents the likelihood that another sample will provide the same results. It is the percent likelihood statement that accompanies the width of the confidence interval. It is often set to the 95% level by convention but can be adjusted (see the confidence level in the figure below).

A confidence level of 95% means that 95 out of 100 times the sample percentages will fall within the confidence intervals. Or 5 times out of 100 the percentages will NOT fall within the confidence intervals. U

So if the Star Tribune took the same poll 100 times with a margin of error of 6% at 95% confidence, we'd expect that about 5 of those polls would show Jim Bean to have more than 61% of the vote or less than 49% of the vote.

Why 95%?
95% is the most frequent value of the confidence level and it is set that way mostly by convention (5% seemed like a reasonable amount of risk I suppose). You would want to lower it to 90% or 85% or raise it to 99% depending on the impact of being wrong.

In other words, if you were betting a lot of money on where the 100 sample results would fall, then you'd want to use a 99% confidence level, not an 80% level (unless you don't mind a 20% chance of having to pay up). But, there is a price to pay for having a more precise estimate, and that's a wider confidence interval. U

So as your confidence level increase, your confidence interval gets wider. By the way, there's nothing stopping you from having 96%, 91% or even an 83.5% confidence level. Here's another example.

You ask 12 users to register for a newsletter on a website you're testing for usability. You need to report the average time to the marketing team who's thinking about redesigning the website. After timing 12 users you report an average of 80 seconds with confidence intervals. The figure below shows how the confidence intervals change depending on the confidence level chosen.

So what intervals do you provide to the marketing group? If they want high confidence, you need to use a higher confidence level perhaps 95% or 99%(you need to be more confident of your results). If they just want a ballpark, then use a lower confidence level.

Let's assume they want to be moderately confident. In that case, I think a 90% confidence level will suffice. So here are a few ways you could report the range you observed in your sample of 12 users.

• The mean time was 80 seconds and we'd expect future samples to fall between 69 seconds and 91 seconds 90% of the time.
• The mean time was 80 seconds +/- 11 seconds (with 90% Confidence).
• The mean time was 80 seconds with a 90% confidence interval of 69 seconds to 91 seconds

As you can see, there are a few ways to phrase things. Notice how all three (implicitly or explicitly) contain the confidence interval, the margin of error and the confidence level? All three are critical in assessing statements using confidence intervals.