Jeff Sauro • April 17, 2012

One of the simplest ways to measure any event is a binary metric coded as a 1 or 0.Such a metric represents the presence or absence of just about anything of interest: Yes/ No, Pass/ Fail, Purchase/No Purchase, On/Off.

Fundamentally the binary system is at the heart of computing as we know it. It also plays a critical role in user research.

Binary measures can even apply to traditionally qualitative research such as noting whether a user mentions distrust of sales people or marketing materials in an interview.

By adding up all the 1's and dividing by the total number observed you get a proportion. So if 5 out of 10 users mention distrusting marketing materials in an interview you get a proportion of 5/10 = .5 which is often expressed as a percentage 50%.

You can use proportions to help make data driven decisions just about anywhere: Which design converts more? Which product is preferred? Does the new interface have a higher completion rate? What proportion of users had a problem registering?

When it comes to comparing independent proportions you'll need to conduct a statistical test called the 2 proportion test (which is equivalent to the Chi-Square test).

We talk a lot about comparing two proportions in our book Quantifying the User Experience (Chapter 5) and recommend a slight adjustment to the typical formulas you might be familiar with so it works for small and large sample sizes.

To make it easier to compare proportions, I've created a simple online web-calculator which will do the statistical calculations for you. Just enter two proportions to see if the difference between them is more likely due to chance or more likely a legitimate difference.

Here are 6 examples of proportions and the statistical results to get you thinking:

- Completion Rates: If 11 out 12 users complete a task on Design A and only 5 out of 10 can complete the same task on Design B, then we can be 97% confident more users can complete the task on Design A.
- Conversion Rates: A large blue button was shown to 455 users and 37 (8%) purchased a product. A large red button was shown to 438 different users and 22 purchased the product (5%). There is a 94% chance the blue button will sell more products.
- Problem Occurrence: 4 out of 7 users received at least one error message when entering alerts and notifications into their profile on a credit card website. After a redesign, 1 out of 7 had at least one error. There is an 89% chance the number of errors has been reduced when setting account alerts.
- Proportion Recommending: 89 out of 100 (93%) customers said they recommended Smart Phone A to a friend in the last year compared to 67 out of 93 (72%) for Smart Phone B. There is a 99.7% chance this retroactive recommend rate is different.
- Proportion detracting: Prior to the change in the return policy, 49 out of 100 (49%) customers surveyed were detractors. After the change in policy 40 out of 96 (42%) were. There is about a 69% chance the difference is not due to chance (good, but not overwhelming evidence).
- Proportion that completed a task in less than 30 seconds: 4 out of 9 users could add a new contact in CRM application A in less than 30 seconds. Eleven out of 12 could on CRM application B. There is a 97% chance if we tested all users, more would complete the tasks on App B.

The New Face of Usability Testing

7 Techniques for Prioritizing Customer Requirements

How to Compute a Confidence Interval in 5 Easy Steps

The Experiment Requires That You Continue: On The Ethical Treatment of Users

8 Ways to Show Design Changes Improved the User Experience

How to Conduct a Usability test on a Mobile Device

Why you only need to test with five users (explained)

97 Things to Know about Usability

Should you use 5 or 7 point scales?

Nine misconceptions about statistics and usability

What five users can tell you that 5000 cannot

A Brief History of the Magic Number 5 in Usability Testing

Does better usability increase customer loyalty?

How common are usability problems?

The Five Most Influential Papers in Usability

5 Examples of Quantifying Qualitative Data

.

Quantifying the User Experience: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.