6 Proportions to Compare when Improving the User Experience

Posted Comments

There are 6 Comments

May 29, 2012 | Aniko Sandor wrote:

Jeff,

I meant p=0.4 in the example.
Aniko 

---

May 29, 2012 | Aniko Sandor wrote:

Jeff,
I understand your point that sometimes the audience is not trained in statistics and you still need to explain them your results. However, misinterpreting the p value may not be the best way to go. Perhaps p values need not to be presented in that case. Maybe the results could be explained the way you put it at the end of your comment: the difference found is not due to chance, therefore one design is better than the other. Think about a situation when the p value is .04. Would you still say you are 60% confident when the chance of a real difference is really small?
I admit I don’t have a better approach to this question, perhaps it is audience-dependent and sometimes you have to compromise.
Aniko 

---

May 10, 2012 | Jeff Sauro wrote:

Aniko

Thanks for the comment. Strictly speaking you are correct. The p-value speaks to the probability of obtaining a difference that large or larger if there really was no difference. However, I'm afraid this subtlety gets lost on a less technical audience, or worse, gets ignored entirely. Using confidence in this sense while less precise I find is both more interpretable and actionable. It also happens to be the lexicon adapted by the A/B testing and SEO communities (e.g. Google Analytics).

Saying you can be (1-p)*100% confident in a difference is something like short hand for saying "now that I know the observed significance level, I can say that if I had selected (1-p)*100% for the confidence level, the interval would have excluded 0"). For the more statistically inclined we might wince a little but the gist of the exercise has been communicated and digested: it's unlikely that the difference is due to chance.
 

---

May 10, 2012 | Aniko Sandor wrote:

Jeff,
On example 1 the interpretation of the p value seems inappropriate. A p value (or 1 - p) cannot be interpreted as confidence (97% in this case) in results. It only means that a difference like the one you found between your samples is likely to happen only in 3% of the cases if they are from the same population. Therefore, it is more likely that they are from different populations, thus your results are significant. 

---

April 24, 2012 | Jeff Sauro wrote:

John,

No, you're absolutely right! Thanks for the catch, I just fixed it.
 

---

April 23, 2012 | John wrote:

Jeff - Please correct me if I'm wrong here, but your statement in example #1 seems to be mis-worded. "If 11 out 12 users complete a task on Design A and only 5 out of 10 can complete the same task on Design B, then we can be 97% confident more users can complete the task on Design B." Shouldn't it be, "more users can complete the task on Design A."  

---

Post a Comment

Comment:

Your Name:

Your Email Address:

To prevent comment spam, please answer the following :
What is 1 + 4: (enter the number)