Compared to What? Making Sense of Customer Experience Metrics

Jeff Sauro • September 28, 2011

These are three of the most important words for anyone trying to make better decisions with data.

I first heard them from Edward Tufte over a decade ago. 

You need a meaningful comparison to turn data into information.

We're often in such a hurry to get the survey out, start the usability test or conduct user interviews that we forget how we'll make sense of all the data. 

Even the best technology or statistical technique cannot substitute for information that puts responses, ratings and revenue in context.

For most measures of customer experience there are at least three good places to start to look for meaningful comparables:

1. A prior version
2. An industry average
3. A leading competitor

Here are some common metrics and some ideas for making them more meaningful:

Task Time: Time is money. But is a task taking too long or is it fast enough? You really need to have comparable data on the same task because subtle differences in task details can make your comparisons less helpful. For example, it takes Southwest Airlines on average 20 minutes to turn a plane around after it lands and have it ready for the next flight. For the same task, it takes United Airlines 35 minutes (see Gladwelll 2010).


Task Difficulty:  It's helpful to use a survey question that's been vetted psychometrically like the Single Ease Question (SEQ). Even with the right question and response options you'll want some comparable. 


Overall Ease of Use: What do users think of the overall usability of the application? The System Usability Scale (SUS) is a popular usability questionnaire that's free and short (at 10 items). It provides an overall estimate of usability.


Net Promoter Score: Are customers recommending your software to friends and colleagues? Asking the popular 11-point question is a good start. But is a 25% Net Promoter Score good? 



In your next analysis be sure to ask and answer "Compared to what?"

Learn More

Jeff Sauro hosted a live webinar on February 28th, 2012 on Best Practices for Remote Usability Testing. The event was overbooked so if you missed it you can now view a recording.

You Might Also Be Interested In:

What is a good task-completion rate?

SUS Guide & Calculator Package

Net Promoter & Usability Benchmark Report for Consumer Software

Will the real task-time please stand up?

Rate this Blog

Avg. Rating 4.8 (55)

Poor										Excellent
0	1	2	3	4	5	6	7	8	9	10

.

Posted Comments

Post a Comment

Comment:

Your Name:

Your Email Address:

.

To prevent comment spam, please answer the following :
What is 2 + 3: (enter the number)