Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Can you use the SUS for websites?

Jeff Sauro • January 18, 2010

The System Usability Scale (SUS) is the most popular standardized usability questionnaire. SUS was developed about 20 years ago at Digital Equipment Corporation by John Brooke. It's popular for two reasons: it's free and short (at only 10 questions). The process of taking a set of ordinary questions and making it into a psychometrically valid and reliable "standardized" questionnaire essentially involves having many users answer a large set of questions against many products and interfaces. You then analyze the responses to see which questions correlate well, tend to a cohesive structure and elicit about the same pattern of responses from people. You throw out questions with low correlations and unusual response patterns to whittle the set down to a manageable set that purports to measure something, in this case perceptions of usability.  This process of standardization was performed on the original super-set of SUS candidate questions some twenty years ago before the web (as we know it) existed. 

I've seen SUS data for desktop applications, printers, cell-phones, Interactive-Voice Response systems and most recently on websites and web-applications.  Some have argued that the 10 SUS questions just don't lend themselves well to effectively measuring websites. Websites are typically walk-up-and use customer facing interfaces. Does a SUS score on a website mean the same thing as a SUS score on a word-processor?
  1. I think that I would like to use this system frequently.
  2. I found the system unnecessarily complex.
  3. I thought the system was easy to use.
  4. I think that I would need the support of a technical person to be able to use this system.
  5. I found the various functions in this system were well integrated.
  6. I thought there was too much inconsistency in this system.
  7. I would imagine that most people would learn to use this system very quickly.
  8. I found the system very cumbersome to use.
  9. I felt very confident using the system.
  10. I needed to learn a lot of things before I could get going with this system.
Exhibit 1: The 10 SUS questions.

There are certainty some questions which don't seem well-suited for websites (e.g.Question 4, "I think I'd need the support of a technical person to be able to use this website.") and I'm sure some of the candidate questions which were thrown out in the original design phase of the questionnaire might be better. So there is good reason to be skeptical about the validity of SUS scores on website. With that said, websites are still interfaces and while some of the questions aren't the best, they seem to be close enough. Of the data I've seen so far, websites with high-perceived usability and applications with high-perceived usability tend to have similar high SUS scores. As I get more data on SUS scores from websites, I'll be interested to see how well this heavily used psychometric instrument holds up to the changing times.

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

SUS, Questionnaires
.

Posted Comments

There are 2 Comments

August 10, 2010 | Carolyn Snyder wrote:

I've used the SUS on a bunch of web sites, and in my opinion the scores are usually well aligned with the qualitative data. Your post makes me wonder if "I would need help from a customer support person" is the solution to #4, though even as is I think it's a useful question. Although it reads a bit strangely for a site that works well, I've seen instances where people have agreed with it, either because the site was buggy or required specialized domain knowledge.

The question I'd most like to overhaul is #5. Some people don't know the meaning of "integrated," and "functions" implies software. I've used a variant: "The various parts of this site fit together well." I have also, at times, replaced "cumbersome" with "awkward," for similar reasons.

I realize that tweaking validated questionnaires is the kind of thing that drives gurus like you nuts, but ultimately I don't really care about comparing a web site to a word processor. I care about comparing a web site with similar sites intended for similar audiences. I never use a SUS score alone as justification for a recommendation, and in the report I provide caveats for interpreting it.

It's like what they say about democracy - it's the worst form of government... except compared to everything else. When something better comes along, I'll switch. Until then, you can have my SUS when you pry it from my cold, dead hands :-). 


January 19, 2010 | Asbjorn Folstad wrote:

Thanks for an interesting and relevant post. It seems to be no easy solution to the challenge of finding user satisfaction measurements that are bot valid and relevant. It would be interesting to have your thoughts on recommendable measurements (if any) that are particularly tailored to measure the user satisfaction of websites. 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 2 + 1: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4081 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.