Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

SUM: Single Usability Metric

(Presented at CHI 2005)

Jeff Sauro • April 17, 2005

SUM is a standardized, summated and single usability metric. It was developed to represent the majority of variation in four common usability metrics used in summative usability tests: task completion rates, task time, satisfaction and error counts. The theoretical foundations of SUM are based on a paper presented at CHI 2005 entitled "A Method to Standardize Usability Metrics into a Single Score." Sauro and Kindlund.

Usability ScoreCard Added June. 2007
The UsabilityScorecard web-application will take raw usability metrics (completion, time, sat, errors and clicks) and calculate confidence intervals and graph the results automatically. You can also combine any combination of the metrics into a 2, 3 or 4 measure combined score. Data can be imported from Excel (.csv) and exported to Word(.rtf).

SUM Calculator

The SUM calculator will take raw usability metrics and convert them into a SUM score with confidence intervals. The analyst needs to provide the raw metrics on a task-by-task basis and know the opportunity for errors. SUM will automatically calculate the maximum acceptable task time, or it can be provided. This calculator is an Excel based version of the UsabilityScorecard except that it can only combine 4 measures (time, errors, sat and completion) and does not graph the results.

Download the SUM Calculator (Free Registration Required)

SUM FAQ's
  1. Why would you want a single usability metric?
If you agree that to truly know the usability of a product means measuring its usability, then you must necessarily ask: how do you measure usability? For us, that's SUM--a composite of multiple measures that all attempt to measure usability.

The bulk of usability activities are formative, that is, their major intent is to uncover and fix usability problems in a user interface. A single score would not be as beneficial here. It is most beneficial during a benchmarking test or summative assessment when you want to know how usable the application is, as opposed to what the usability problems are. Before you can say how usable something is, you need to be able to measure usability.

Why would you want to measure usability? If you cannot measure the construct of usability, or agree on what accounts for usability in some measurable way, then how will we know if any of the UE activities actually make the product more usable? This is a larger philosophical issue that has been asked in many contexts, here are some examples:

Measurement is at the heart of our scientific method. "Numerical Precision is the very soul of science" D'Arcy Wentworth Thompson, On Growth and Form (1917)
If you can't measure it, you cant manage it. (Old Management Saying)
>When you can measure what you are speaking about, and express it in numbers you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of and unsatisfactory kind: It may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of Science. Lord (William Thompson) Kelvin, pioneer in thermodynamics and electricity,1891.

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More

.

Posted Comments

There are 7 Comments

January 23, 2014 | Andrew Sweany wrote:

Hi again,
Please disregard my prior question - did a bit more research on the topic and answered my own question :)  


January 23, 2014 | Andrew Sweany wrote:

Hi Jeff,
Quick question: is it necessary to collect error rates to calculate the SUM or can it be done without? The reason I ask is that in the post on 10 essential metrics you state that the "SUM is a standardized average of measures of effectiveness, efficiency of satisfaction and is typically composed of 3 metrics: completion rates, task-level satisfaction and task time."

I love the idea of a single composite score but would like to keep it as simple as possible. Thanks for your help - absolutely love the site.  


September 3, 2009 | Mark Sheldon wrote:

If 100% of people can complete a task in 0 seconds it cannot be improved upon. How good usability is in reality is likely to be comparitive. A measure is useful as in peer group or competitive benchmarking you can identify best practice. You can also identify "verschlimbessurn" improvements that make things worse.

A possible complexity may be if the task is for pleasure. 


October 28, 2008 | Jeff Sauro wrote:

Interesting question. It depends on what you mean by user's experience. This measure, in its original 4 score form has a subjective component, task-level satisfaction. While these questions can vary, they usually ask the user how easy they thought the task was, how satisfied they were with the amount of time and ease in completing it. I believe that approaches what many consider the user's experience, perhaps not completely. Other ways to gauge user experience would be post-test questionnaires, such as the SUS, although we do not include that in the combined metric. 


October 28, 2008 | Jerry wrote:

Could this measure be used to gauge the user's experience with a product or it is more a measure of how usable the product actually is? 


June 5, 2008 | Joanne Locascion wrote:

I am trying to understand if there is a particular number by which something may be deemed or labeled as having "good" usability or "poor" usability. In other words, is a cut-off score. Trying to use this as part of a Master's project.
Thanks 


December 1, 2007 | anonymous wrote:

Single Usability Metric 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 4 + 5: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4331 Subscribers]

Connect With Us

Our Supporters

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.