How many customers should you observe?
Jeff Sauro • February 8, 2011
Timing, luck and perseverance all play a role in making a successful product.
But so does observing and understanding your customers' problems.
The number of customers you need to observe will depend on how common customer behaviors are and how certain you need to be.
Building a successful product means building something that customers want or need and are willing to pay for. It's not easy.
There are books written on successful and failed strategies for developing the right product.
Asking Customers What they Want?
It's not as simple as asking customers what they want and building the product or adding features. Steve Jobs famously said
It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them.
I'd especially agree with the focus group part.
This does not mean it's never appropriate to ask customers what they want. Techniques like conjoint analysis
work well when prioritizing features that are well understood and have a certain perceived value (e.g. features in a car). It's less effective for solving problems customers don't know they have.
When I worked at Intuit
(makers of TurboTax, Quicken & QuickBooks) we used a method called "follow-me-home
." It was as effective as it was simple in identifying customer problems and needs.
We went to a customer's house or workplace and watched them do what they do and recorded their behavior and problems they encountered and how they solved them. Data from follow-me-homes were used for new product ideas and improving existing products.
We tried not to interfere and instead took copious notes about how customers used the product--and we certainly didn't ask customer what features they wanted in the product. The notes from these sessions took many forms, but would contain information like this:
- Customer used a calculator to figure out the sales-tax instead of using the product feature.
- Customer exports data from their Point-of-Sale register into QuickBooks at the end of each day.
Designers, researchers and even managers took turns following customers home. The notes were categorized and grouped and sorted. After just a few customer-visits, some observations and behaviors were listed many times. For every new customer there would be at least one or two new observations.
While we were seeing redundancy, we wanted to know how many customers we needed to follow-home before we had a reasonable picture of the problems so we could prioritize them, come up with solutions and build a better product. At best there were rules of thumb of between 5 and 20 of for each customer type.
While the idea of observing customers to generate innovation[pdf]
is popular, there is little guidance on the number of customers you should plan on observing.
Sample Size Calculations for Discovering Product Opportunites
To answer this question on how many customers we need to observe to find opportunities for innovation, we can use the same approach that is used in finding the number of users you'd need to detect problems in an interface. It's an approach Jim Lewis will explain in our forthcoming book
and is also subject to vague and controversial sample size conventions
To use it, we need to define two things.
- Issue frequency: Pick the minimum percentage of customers that will have the behavior or problem that you want to have a high likelihood of observing. For example, you might decide that you want to reliably detect (at least once) critical events that will happen to one out of five (20%) or more of your customers.
- Chance of observing the behavior: Specify how certain you need to be of seeing (at least once) the issues you identified in Step 1. For example 90%, 85% or 80%.
For example, let's say you want to know how many customers you need to follow-home to be 90% sure you've seen, at least once, problems that 20% or more of users will exhibit (like not knowing a certain feature exists).
To find out you use a modification to the binomial probability formula
Sample Size = log(1-.90)/log (1-.20) = 10.3 and to be safe we can round up to 11
So after following 11 customers home, you'll have at least a 90% chance of observing (at least once) behaviors and problems that 20% or more of customers will have.
The following calculator will do the math for you. Select 90% and 20% then click "Compute".
Sample Size Calculator
If you want to observe more behaviors that are less frequent, or need to be more certain, the number of customers you'll have to follow home will increase.
For example, if you want to have a 90% chance of observing (at least once) the behaviors that 1% of customers exhibit, you'll need to clear your calendar for 230 customer visits.
At any sample size, you are going to see repeated behaviors and problems. Counts of each behavior can be used to prioritize product ideas.
You can even generate confidence intervals
around the percentage of customers exhibiting the behavior to have a solid estimate of how common they are in the total user-population.
Keep in mind that observing 11 customers does not mean you'll see 90% of all possible behaviors. It only means you'll see 90% (or more) of the behaviors that 2 in 10 (or more) customers will exhibit within the constraints of the study.
This approach also only works with one type of customer
(e.g. small retail businesses, home-builders or tax-preparers). Change the type of customer and you need to compute a new sample size. Behaviors may change across customers, geography and age so you need to consider whether these attributes differentiate customer behaviors.
A Quantitative Approach to Support a Qualitative Activity
It may be that only one behavior will spark an idea that will generate a new product or feature that will radically increase sales for all customers. Quantity doesn't necessarily mean quality.
Whether your insights will generate the next Newton or the next iPad depends on a lot of factors. But when you want to know what percent of behaviors you'll likely see from your planned customer visits, this method is far more reliable than vague rules of thumb.