A/B Test Calculator

Posted Comments

There are 6 Comments

April 22, 2013 | Jeff Sauro wrote:

Richard,

Thanks for your comment. You are exactly right in your definition of the p-value. Unfortunately, the subtlety of your statement gets lost in the applied world of A/B testing. I understand the distinction but it usually confuses not informs the recipient of the results. See for example an earlier comment on this page as proof. We provide this context in Chapter 5 of the book.

Using 1 minus the p-value is a crude shorthand for interpreting the p-value for execs and those interpreting A/B test results. It is often communicated simply as confidence in many A/B testing packages (e.g. 95.7% confidence). So while this is not something I recommend reporting in scientific publications, in the world of picking the better alternative (version A or version B) using this nomenclature likely does more good than harm.
 

---

April 21, 2013 | Richard. wrote:

95.474% is not the chance the proportions are different. 4.526% is the chance that if the population (i.e. true) proportions were the same you would get a result as or more extreme than what you observe. 

---

February 11, 2013 | Zeeshan wrote:

very useful 

---

November 6, 2012 | Chuck wrote:

The Chi-square test assumes normal distribution of the variables. The results are too liberal [too low in this case]] The variables involved have a discrete distribution and a beta distribution is used for the test. Still, this is not too bad and is in line with many published simplifications. The results are not highly off, just a bit. 

---

September 9, 2012 | Zeeshan wrote:

I found this calculator very useful for non statistical persons 

---

May 16, 2012 | Bryan wrote:

A little help for those of us without stats degrees?

- What the hell are 1-tailed and 2-tailed p-values?
- In plain English, what does it mean that there's an N% chance the proportions are different, and why should I care?
- Ditto higher proportion

 

---

Post a Comment

Comment:

Your Name:

Your Email Address:

To prevent comment spam, please answer the following :
What is 3 + 1: (enter the number)