Jeff Sauro • February 8, 2011

Timing, luck and perseverance all play a role in making a successful product.But so does observing and understanding your customers' problems.

The number of customers you need to observe will depend on how common customer behaviors are and how certain you need to be.

Building a successful product means building something that customers want or need and are willing to pay for. It's not easy.

There are books written on successful and failed strategies for developing the right product.

It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them.

This does not mean it's never appropriate to ask customers what they want. Techniques like conjoint analysis work well when prioritizing features that are well understood and have a certain perceived value (e.g. features in a car). It's less effective for solving problems customers don't know they have.

We went to a customer's house or workplace and watched them do what they do and recorded their behavior and problems they encountered and how they solved them. Data from follow-me-homes were used for new product ideas and improving existing products.

We tried not to interfere and instead took copious notes about how customers used the product--and we certainly didn't ask customer what features they wanted in the product. The notes from these sessions took many forms, but would contain information like this:

- Customer used a calculator to figure out the sales-tax instead of using the product feature.
- Customer exports data from their Point-of-Sale register into QuickBooks at the end of each day.

While we were seeing redundancy, we wanted to know how many customers we needed to follow-home before we had a reasonable picture of the problems so we could prioritize them, come up with solutions and build a better product. At best there were rules of thumb of between 5 and 20 of for each customer type.

While the idea of observing customers to generate innovation[pdf] is popular, there is little guidance on the number of customers you should plan on observing.

To use it, we need to define two things.

- Issue frequency: Pick the minimum percentage of customers that will have the behavior or problem that you want to have a high likelihood of observing. For example, you might decide that you want to reliably detect (at least once) critical events that will happen to one out of five (20%) or more of your customers.
- Chance of observing the behavior: Specify how certain you need to be of seeing (at least once) the issues you identified in Step 1. For example 90%, 85% or 80%.

To find out you use a modification to the binomial probability formula.

Sample Size = log(1-.90)/log (1-.20) = 10.3 and to be safe we can round up to 11

The following calculator will do the math for you. Select 90% and 20% then click "Compute".

If you want to observe more behaviors that are less frequent, or need to be more certain, the number of customers you'll have to follow home will increase.

For example, if you want to have a 90% chance of observing (at least once) the behaviors that 1% of customers exhibit, you'll need to clear your calendar for 230 customer visits.

At any sample size, you are going to see repeated behaviors and problems. Counts of each behavior can be used to prioritize product ideas.

You can even generate confidence intervals around the percentage of customers exhibiting the behavior to have a solid estimate of how common they are in the total user-population.

This approach also only works with one type of customer (e.g. small retail businesses, home-builders or tax-preparers). Change the type of customer and you need to compute a new sample size. Behaviors may change across customers, geography and age so you need to consider whether these attributes differentiate customer behaviors.

Whether your insights will generate the next Newton or the next iPad depends on a lot of factors. But when you want to know what percent of behaviors you'll likely see from your planned customer visits, this method is far more reliable than vague rules of thumb.

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Identifying the 3 Types of Missing Data

10 Essential User Experience Methods

The Five Most Influential Papers in Usability

10 Things to Know about Usability Problems

What five users can tell you that 5000 cannot

How common are usability problems?

Why you only need to test with five users (explained)

A Brief History of the Magic Number 5 in Usability Testing

8 Ways to Show Design Changes Improved the User Experience

Nine misconceptions about statistics and usability

How to Conduct a Usability test on a Mobile Device

Confidence Interval Calculator for a Completion Rate

Does better usability increase customer loyalty?

97 Things to Know about Usability

Should you use 5 or 7 point scales?

.

Quantifying the User Experience: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.