Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

One thing you should fix on your website

Jeff Sauro • November 7, 2012

There's a long list of things to fix on any website.

But there are usually a few fixes which will have a disproportionate impact.

To help identify those, we ask users a simple question at the end of our studies: "If you could only fix one thing on this website, what would it be?"

We use an open ended question to collect responses. While this means we get a lot of "nothing" and "I don't know's" we also get a lot of valuable insights that are top-of-mind for the users. 

We perform a content-analysis and sift through hundreds to thousands of unique responses to group them into larger, more interpretable buckets.
 

What users would fix

As part of our ongoing data collection effort for the Standardized Universal Percentile Rank (SUPR-Q) we regularly survey users of over 200 websites on a rolling basis. We also ask what one thing each respondent would fix. I looked at a cross selection of 30 surveys we collected in the last year with over 2000 responses to our "what would you fix question."  The 30 websites are shown in the table below.

AmazonNew York Times
AppleOrbitz
BestBuyPayPal
CNNShutterfly
CraigslistSnapfish
DeltaTarget
eBayTwitter
eTradeUSA.gov
FacebookVistaprint
FlickrWall Street Journal
FoxnewsWalmart
GoDaddyYahoo
IMDBYelp
LinkedInYouTube
NetflixZappos

After eliminating the "I don't knows" and "nothing's" I put all the things users say they would fix into a word cloud.

 


Word clouds seem like a vestige of the Web 2.0 craze, but on occasion they can be helpful for taking a quick glance at the data.

Right away we see that "search" dominates along with the less helpful "better" and "easier" (imagine if you deliver a finding that said uses want things better and easier!).

Navigate also shows up, but certainly lower down in the list after things like videos and movies (which likely come from websites like Netflix and YouTube).

Content Analysis

Next we went through each response and categorized the phrases into groups to look for patterns.  The graph below shows the main groups of what users said they would improve across these 30 websites.
 

Figure 1: Categories derived from a content analysis of open ended responses to "What one thing would you fix" across 30 websites.

For example, we grouped 21% of the responses in to a category called "More Services/Information." Three comments and the websites they came from in this group are shown below:

What one thing would you improve on the website?

More  Services/Information Website
A wider variety of products   Zappos 
I would like more free layouts to choose  from.  Vistaprint 
More detailed descriptions of items   Target 

Again "search" shows up high in the list as we also saw in the word cloud with Yahoo, Amazon, Walmart and YouTube each generating 8% of the comments. Appearance is second with 12% of the responses mostly coming from LinkedIn, IMDB, Best Buy and Craigslist. While "appearance" didn't make the word cloud, we do see the appearance related words of "layout, color and design." 

This again is a reminder that word clouds provide a good first pass at data but are no substitute for deeper synthesis to put words into context.

The "Security/Anti-Spam" category which comprised 8% of the responses was largely driven by websites Craigslist (40%), Facebook (29%) and PayPal (8%). Three examples in this category are shown below.
   
Security/Anti SpamWebsite
Better security controls against spammers  Yahoo
It is a security issue, so I would like it to be known to be secure  Craigslist
Higher safe guards against scammers  PayPal


While we usually limit our analysis to within a single website, by combining 30 websites we are able to see some patterns that resonate with what we typically see in our usability studies. Problems with navigation and a poor search tend to plague the usability of websites, even highly visited ones as shown in this data.
 
Findability is a key component of overall website usability and we see it appearing in the list of fixes.   What's more, we also see some good evidence for why the SUPR-Q measures aspects of usability, credibility/trust and appearance.  All three have prominent roles in website quality and in the list of things to fix. 

So what one thing should you fix on your website? 

It's likely to be one of the items we found in our list but it will certainly depend on the website. A good first step is asking users what they'd fix. Next measure the key aspects of usability, credibility, appearance and loyalty before and after you make changes to quantify the better user experience.
 

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Findability, SUPR-Q
.

Posted Comments

There are 1 Comments

November 30, 2012 | Katie wrote:

Great post! In general, would you suggest turning an open-ended question into close-ended question after you've discovered the top trending categories? Pros and cons of doing so?

Also, what are your thoughts about the reliability of open-ended answers? We've all heard the phrase "users don't know what they want". In terms of credibility of data, is asking for difficulties encountered different then asking for suggestions or ways to improve your product? Interested in your thoughts about this, thanks! 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 1 + 3: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4124 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Usertesting.com

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.