Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

How many customers should you observe?

Jeff Sauro • February 8, 2011

Timing, luck and perseverance all play a role in making a successful product.

But so does observing and understanding your customers' problems. 

The number of customers you need to observe will depend on how common customer behaviors are and how certain you need to be.

Building a successful product means building something that customers want or need and are willing to pay for. It's not easy.

There are books written on successful and failed strategies for developing the right product.  

Asking Customers What they Want?

It's not as simple as asking customers what they want and building the product or adding features. Steve Jobs famously said in 1998:

It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them. 

I'd especially agree with the focus group part.

This does not mean it's never appropriate to ask customers what they want. Techniques like conjoint analysis work well when prioritizing features that are well understood and have a certain perceived value (e.g. features in a car). It's less effective for solving problems customers don't know they have.

Stalking Customers

When I worked at Intuit (makers of TurboTax, Quicken & QuickBooks) we used a method called "follow-me-home." It was as effective as it was simple in identifying customer problems and needs. 

We went to a customer's house or workplace and watched them do what they do and recorded their behavior and problems they encountered and how they solved them.  Data from follow-me-homes were used for new product ideas and improving existing products.

We tried not to interfere and instead took copious notes about how customers used the product--and we certainly didn't ask customer what features they wanted in the product. The notes from these sessions took many forms, but would contain information like this:
  • Customer used a calculator to figure out the sales-tax instead of using the product feature.
  • Customer exports data from their Point-of-Sale register into QuickBooks at the end of each day.
Designers, researchers and even managers took turns following customers home. The notes were categorized and grouped and sorted. After just a few customer-visits, some observations and behaviors were listed many times. For every new customer there would be at least one or two new observations. 

While we were seeing redundancy, we wanted to know how many customers we needed to follow-home before we had a reasonable picture of the problems so we could prioritize them, come up with solutions and build a better product. At best there were rules of thumb of between 5 and 20 of for each customer type.

While the idea of observing customers to generate innovation[pdf] is popular, there is little guidance on the number of customers you should plan on observing.

Sample Size Calculations for Discovering Product Opportunites

To answer this question on how many customers we need to observe to find opportunities for innovation, we can use the same approach that is used in finding the number of users you'd need to detect problems in an interface.  It's an approach Jim Lewis will explain in our forthcoming book and is also subject to vague and controversial sample size conventions.

To use it, we need to define two things.
  1. Issue frequency: Pick the minimum percentage of customers that will have the behavior or problem that you want to have a high likelihood of observing. For example, you might decide that you want to reliably detect (at least once) critical events that will happen to one out of five (20%) or more of your customers.
  2. Chance of observing the behavior:  Specify how certain you need to be of seeing (at least once) the issues you identified in Step 1. For example 90%, 85% or 80%.
For example, let's say you want to know how many customers you need to follow-home to be 90% sure you've seen, at least once, problems that 20% or more of users will exhibit (like not knowing a certain feature exists).

To find out you use a modification to the binomial probability formula.

Sample Size = log(1-.90)/log (1-.20) = 10.3 and to be safe we can round up to 11

So after following 11 customers home, you'll have at least a 90% chance of observing (at least once) behaviors and problems that 20% or more of customers will have.

The following calculator will do the math for you. Select 90% and 20% then click "Compute".

Sample Size Calculator


To observe of behaviors that of customers exhibit, plan on visiting
6
customers.


If you want to observe more behaviors that are less frequent, or need to be more certain, the number of customers you'll have to follow home will increase.

For example, if you want to have a 90% chance of observing (at least once) the behaviors that 1% of customers exhibit, you'll need to clear your calendar for 230 customer visits.

At any sample size, you are going to see repeated behaviors and problems. Counts of each behavior can be used to prioritize product ideas.

You can even generate confidence intervals around the percentage of customers exhibiting the behavior to have a solid estimate of how common they are in the total user-population.

Limitations

Keep in mind that observing 11 customers does not mean you'll see 90% of all possible behaviors. It only means you'll see 90% (or more) of the behaviors that 2 in 10 (or more) customers will exhibit within the constraints of the study.

This approach also only works with one type of customer (e.g. small retail businesses, home-builders or tax-preparers).  Change the type of customer and you need to compute a new sample size. Behaviors may change across customers, geography and age so you need to consider whether these attributes differentiate customer behaviors.

A Quantitative Approach to Support a Qualitative Activity

It may be that only one behavior will spark an idea that will generate a new product or feature that will radically increase sales for all customers. Quantity doesn't necessarily mean quality.

Whether your insights will generate the next Newton or the next iPad depends on a lot of factors. But when you want to know what percent of behaviors you'll likely see from your planned customer visits, this method is far more reliable than vague rules of thumb.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Sample Size, Usability Problems, Ethonographic Research
.

Posted Comments

Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 5 + 3: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[3777 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.