6 Proportions to Compare when Improving the User Experience
Jeff Sauro • April 17, 2012
One of the simplest ways to measure any event is a binary metric coded as a 1 or 0.
Such a metric represents the presence or absence of just about anything of interest: Yes/ No, Pass/ Fail, Purchase/No Purchase, On/Off.
Fundamentally the binary system is at the heart of computing as we know it.
It also plays a critical role in user research.
Binary measures can even apply to traditionally qualitative research such as noting whether a user mentions distrust of sales people or marketing materials
in an interview.
By adding up all the 1's and dividing by the total number observed you get a proportion. So if 5 out of 10 users mention distrusting marketing materials in an interview you get a proportion of 5/10 = .5 which is often expressed as a percentage 50%.
You can use proportions to help make data driven decisions just about anywhere: Which design converts more? Which product is preferred? Does the new interface have a higher completion rate? What proportion of users had a problem registering?
When it comes to comparing independent proportions you'll need to conduct a statistical test called the 2 proportion test (which is equivalent to the Chi-Square test).
We talk a lot about comparing two proportions in our book Quantifying the User Experience (Chapter 5)
and recommend a slight adjustment to the typical formulas you might be familiar with so it works for small and large sample sizes.
To make it easier to compare proportions, I've created a simple online web-calculator
which will do the statistical calculations for you. Just enter two proportions to see if the difference between them is more likely due to chance or more likely a legitimate difference.
Here are 6 examples of proportions and the statistical results to get you thinking:
- Completion Rates: If 11 out 12 users complete a task on Design A and only 5 out of 10 can complete the same task on Design B, then we can be 97% confident more users can complete the task on Design A.
- Conversion Rates: A large blue button was shown to 455 users and 37 (8%) purchased a product. A large red button was shown to 438 different users and 22 purchased the product (5%). There is a 94% chance the blue button will sell more products.
- Problem Occurrence: 4 out of 7 users received at least one error message when entering alerts and notifications into their profile on a credit card website. After a redesign, 1 out of 7 had at least one error. There is an 89% chance the number of errors has been reduced when setting account alerts.
- Proportion Recommending: 89 out of 100 (93%) customers said they recommended Smart Phone A to a friend in the last year compared to 67 out of 93 (72%) for Smart Phone B. There is a 99.7% chance this retroactive recommend rate is different.
- Proportion detracting: Prior to the change in the return policy, 49 out of 100 (49%) customers surveyed were detractors. After the change in policy 40 out of 96 (42%) were. There is about a 69% chance the difference is not due to chance (good, but not overwhelming evidence).
- Proportion that completed a task in less than 30 seconds: 4 out of 9 users could add a new contact in CRM application A in less than 30 seconds. Eleven out of 12 could on CRM application B. There is a 97% chance if we tested all users, more would complete the tasks on App B.