Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

The Essential Elements of a Successful Website

Jeff Sauro • October 4, 2011

What makes a successful website?

There are some obvious metrics like revenue, traffic and repeat visitors.

But these are outcome measures. They don't tell you why revenue or traffic is higher or lower. 

Key drivers of these outcomes are how the users perceive and interact with your website. Selling a product that has demand or information that is valuable is of course essential. But it's rare to have a monopoly on products or information on the web.

What differentiates websites is the customer experience. 

Most people would agree that the customer experience is important, but what specifically about the experience helps or hinders a website? 

I examined the customer experience research in the Marketing and Usability literature and found some consistent themes. A successful website needs to be usable, credible and visually appealing. This will generate positive word of mouth, repeat visitors and ultimately a more successful website.

The trick is effectively measuring these concepts.

Measuring the Website Customer Experience

When people think of measuring website effectiveness they often think of analytics like click-through rates, purchases, bounce rates and time on site. These are important, but aren't giving us the whole customer experience picture. An effective way to know if users trust your website, think its usable and visually appealing is to ask them.

Unfortunately it's not as simple as just asking: "Is this website usable?" or "Do you trust us?"  There is a science to asking the right question in a way that generates reliable and valid conclusions. The process is called psychometric validation. It involves identifying different ways of asking users about the constructs of interest then refining statements (called items) to identify the ones that best discriminate good websites from bad.

There are many different questionnaires generating hundreds of items with different rating scales that measure different aspects of website usability, credibility, loyalty and appearance.

I picked 75 candidate items and asked several hundred users to respond to them regarding their recent usage of several websites. I then narrowed the list of items down to around 20 which tended to have the best internal reliability and ability to discriminate between good and bad websites. Finally the top 13 items were selected based on how well they clustered together in a Factor Analysis around the construct they were intended to measure: usability, credibility, loyalty and appearance.



The 13 items together create a new standardized questionnaire called the SUPR-Q. It stands for the Standardized Universal Percentile Rank-Questionnaire. Here are the four essential elements that make for a successfully website and how the 13 SUPR-Q items measure them.

Usability

Once you have a product or service that people want (utility), few things matter more than usability. Maybe the website offers a stellar service and looks really slick. If users can't accomplish what they want to do, find the product they're looking for, or complete their purchase--it's like it doesn't even exist. Especially for eCommerce websites, a usable experience means a profitable experience.

The construct of website usability is measured by having users state their level of agreement to these four items.

1.    This website is easy to use.
2.    I am able to find what I need quickly on this website.
3.    I enjoy using the website.
4.    It is easy to navigate within the website.

These four items account for 95% of the 10 item System Usability Scale (SUS) and provide an excellent measure of concurrent validity.  That means they provide a reliable measure of usability, specific to websites, more efficiently than SUS.

You'll notice that two of the items specifically reference findability and navigationósalient attributes of a usable website experience.  Item 3 taps into a measure of hedonic quality[pdf]. It's grouped with three more traditional usability items because it appears users tend to have similarly responses to usability and enjoyment. More technically, it means these items all load on the same factor as uncovered in a Varimax Rotated Factor Analysis.

Credibility (Trust, Value & Comfort)

Does the website sell products and collect credit card information? Are you gathering email addresses to build a subscriber base? If users don't trust your website, which for many companies is synonymous with their company and brand, they won't give up their information and website growth is impeded.

These five items measure the construct of credibility--which touches on aspects of trust, value, comfort and confidence.

5.    I feel comfortable purchasing from this website.
6.    This website keeps the promises it makes to me.
7.    I can count on the information I get on this website.
8.    I feel confident conducting business with this website.
9.    The information on this website is valuable.

Loyalty

Are users talking about your website favorably or are they telling their friends to avoid it like the plague? Will they return to the website and purchase more things or at least see what you have to say? These two items touch upon repeat usage from existing customers and net-new usage from new customers.

10.    How likely are you to recommend this website to a friend or colleague? (This is the same question used in the Net Promoter Score).
11.    I will likely visit this website in the future.

Appearance

Is your website looking circa 1998 and is the appearance hindering the experience? Users form impressions of your website based on the appearance in just a few seconds.

12.    I find the website to be attractive.
13.    The website has a clean and simple presentation.

Scoring

All the items (except #10) use a five point scale from strongly disagree to strongly agree. Item #10 uses the 11 point format familiar to those who ask it as the Net Promoter question.



Figure 1: The SUPR-Q's 13 items. All but item #10 are presented using a five point agreement scale.The name of the website can be used instead of "this website." 


Using the SUPR-Q

After the initial validation phase I commissioned several studies to gather data on websites. I surveyed current customers of hundreds of websites to compile a database of 4500 responses between summer 2010 and summer 2011.

The websites come from 18 industries including: Travel, Airlines, Wireless Carriers,  Retail, News/Information, Government and Automotive websites.  It contains a spectrum from highly usable and trustworthy to difficult to use and utterly chaotic websites.   See more details on the SUPR-Q and a list of more of the websites in the database.

The SUPR-Q can be administered after a usability test or to current users of a website retrospectively. In total there are over 200 websites in the SUPR-Q database.  Each website  contains data from between 30 and 400 users.

In addition to having a reliable and valid instrument for measuring websites, the database behind the SUPR-Q provides a relative percentile rank (this is what gives the questionnaire its name).  Instead of working with a raw mean, the global score and each component score are normalized against the database of websites and presented as a percentage.

This way, a score of 75% means the website scores higher than 75% of the websites in the database. Because I've commissioned the studies and own the data I can also reveal how each website scores on all the attributes.  It provides for an interesting analysis and provides many meaningful comparisons for any website across many industries.

For example, Facebook tends to score highly on loyalty (in the 85th percentile) but very poorly on trust (in the 10th percentile). Zappos, Apple and Amazon have the highest percentile ranks (the highest combination of all factors) while state government website and restaurants are at the bottom.

The database includes many well known brands, some lesser known ones and is refreshed on an annual basis so changes are reflected. For example, Netflix was measured in Q1 of 2011 and has one of the highest SUPR-Q scores (including very high loyalty ratings--corroborating other data). This was before its infamous price increase which has no doubt affected its scores on loyalty and credibility. It will be interesting to see if the scores rebound when I survey the customers again in a few months.

The process of scoring is easy. Collect the responses using any survey application (like Userzoom and Loop 11 or SurveyMonkey) then enter the raw responses into the coded Excel spreadsheet. Immediately you get the normalized percentile rank showing where the website scores relative to the 200 others in the database. You can also see up to 100 individual website scores for each of the attributes of usability, credibility, trust and appearance and even filter by industry.

The SUPR-Q is an efficient and valuable tool for benchmarking your website. It provides a sensitive and reliable measure in 13 easy to administer items. You can use the items freely on your next website evaluation (with attribution) but the real value comes from converting a raw score to a meaningful rank. More information can be found at www.SUPRQ.com.

Contact me to find out more about licensing the SUPR-Q for your next evaluation.



About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

.

Posted Comments

There are 7 Comments

May 7, 2012 | Erin wrote:

Hi Jeff, I was wonderingócan you monetize an increase in SUPRQ score somehow? Like an increase in brand equity? Same question with SUS. I see you said elsewhere on the site that NPS is affected by usability. 


April 23, 2012 | Jeff Sauro wrote:

Erin,

Even for sites that don't currently sell anything, knowing whether people "feel" comfortable purchasing is a helpful metric. Because it's aggregated into the Credibility subscale I've found it tends to work fine even on education or government websites (who often do have bill-pay type features).

You know, I have seen some participants make some comments about the "keeps the promises" question. Some sort of mumble, "hmm, not sure what promises they made" and even when participants have had some question they still seem to find a response that works (usually a 3). I would encourage them to answer it in regard to their attitude about the trustworthiness of the site in general.

With that said I'd be interested to see if that question continues to be an issue and I might drop it or use another one. It is also in the credibility factor which has the most questions so this would be the place to safely drop one. This question was picked originally because it did best discriminate and was more sensitive than others.  


April 18, 2012 | Erin wrote:

If the website is not an ecommerce website (purchasing is done offline) do you have an alternative suggestion for "I feel comfortable purchasing from this website?" How can you benchmark against other industry sites if your business is not ecommerce? Also, "This website keeps the promises it makes to me." is unclear to me. Do your participants ask for clarification, and if so, what do you tell them? 


November 28, 2011 | Rosalinda wrote:

Super jazzed about getntig that know-how. 


October 21, 2011 | Daman Anand wrote:

you are an expert in usability, how you forgot the "top" button at the bottom of your website. The button which takes the user to the top , so they can navigate. 


October 9, 2011 | Kailash Shinde wrote:

great article. i have satisfied with this article. Good for Website developer,
Online Marketers. 


October 6, 2011 | Cormac wrote:

I'll definitely be running this questionnaire on my site. Thanks very much.  


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 3 + 1: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4079 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Use Card Sorting to improve your IA

Usertesting.com

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.