Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

8 Ways to Show Design Changes Improved the User Experience

Jeff Sauro • March 1, 2011

A lot of effort goes into simplifying interactions, reducing bugs and enhancing features.

While these changes may be obvious to some, they can be taken for granted by others (especially those in charge of budgets).

It is valuable to document both the effort that goes into improving the user experience and the result of all that effort. You'll want to measure the interface before and after changes were made with benchmark tests or in a comparative test of both interfaces simultaneously. 

Here are eight ways to show that the design has improved.

Show:
  1. Improvement in perceived usability using a questionnaire like the System Usability Scale (SUS): Even if you double user-productivity, if users don't think an application is more usable, negative word-of-mouth can do great damage. A step-by-step guide and calculator are available for showing statistical improvements in SUS scores.

  2. Reduction in the time it takes to complete a task: Whether users are tracking customer information, logging sales orders or entering time-sheet information, showing a reduction in the time it takes users to complete tasks is a metric executives and users appreciate.

  3. Improvement in productivity for skilled users using Keystroke Level Modeling (KLM): If you don't have the time or budget to test actual users but have tasks that are done repeatedly, using a cognitive modeling technique like KLM provides a quick way to generate relative reductions in tasks times and improvements in worker productivity.

  4. Reduction in the number of UI problems encountered by users during a test: No one likes usability problems and even low budget discount usability efforts focus on finding and fixing problems. Simply document the number, severity and description of problems users are encountering before and after design changes. You have the opportunity to show reductions in both the number and severity of problems.

  5. Increases in completion rates for key tasks: Every application has great features that users just can't figure out. Increases in task completion rates on tasks around features that were heavily invested in can bring new life to a product without increasing feature bloat. People buy the benefits, not the features and usability is a key benefit.

  6. Reduction in the number of calls to customer support: User Interface problems and failed task attempts lead to expensive calls to customer support. Showing that design changes cut the number of support calls is an easily quantifiable savings for a company.

  7. Increases in conversion rates: Converting more browsers to members and more members to buyers is at the core of A/B testing. Simple changes in copy, layout and navigation translate into big numbers.

  8. Improvement in task-level satisfaction ratings using something like the Single Ease Question (SEQ): Overall satisfaction measures like SUS tell you what people think of the product in general but aren't terribly helpful for diagnosing what to fix. They also reflect attitudes about product features you may have no control over. Asking a simple question immediately after a task attempt provides laser-like precision. Eventually task-level satisfaction improvements lead to application level satisfaction improvements and will spread positively through word-of-mouth.
If you don't measure the impact of design changes and quantify the benefits, someone else will. Don't force people to guess. By providing simple quantitative measures of improvements in the user experience you have the data to both justify design efforts and get a better idea of what methods worked.

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Evidence Based Design, UX, Usability Metrics
.

Posted Comments

There are 7 Comments

December 14, 2011 | Jeff Sauro wrote:

Jeff

Good question and yes, in my experience for large enterprise software products that support different functions, for different roles, it's not uncommon to test a fraction of the functionality and use the SUS or other task-metrics to assess that --not just entire products or product suites. 


December 13, 2011 | Jeff Noyes wrote:

This seems like a viable solution for very small products. What about large ones with many many features. Are you breaking the SUS scores down by feature? 


March 21, 2011 | Jeff Sauro wrote:

José

You can use the SUS for any level of experience, it's not just for initial exposure. Experience matters, I've found on average long-term exposure increases SUS scores by 11% over initial use. But the effects of unusable software are typically so larger they over-ride differences in experience.

So there's nothing wrong with using experience users in late-stage analysis, just document their experience when making comparisons between groups. Ideally you're comparing users with similar experience with the application. 


March 21, 2011 | Jeff Sauro wrote:

Meredith,

Great question. It's a common misconception that sample size and quantitative/qualitative studies are related. The minimum sample size you need to use SUS or any quantitative metric is 2.

At a sample size of 8, you can show statistical differences between two iterations. The major difference is that you are limited to detecting large differences. But in early stage designs, you're largely concerned about those big differences that users notice.

As far as effective recruiting, you've really got the same issue whether you are quantifying your findings (as in a benchmark) or detecting problems. If you're using people who aren't representative of your user population, they you may be missing problems or finding false positives.

The notion that you need a large sample size for quantitative usability testing likely comes from the need for precision in benchmark testing. If you want to know how usable an application is, you can haul in some users and test them attempting representative tasks. Of course, any sample will fluctuate do to sampling error (just like a presidential poll does). To minimize the sampling error you need a larger sample size. As a general rule, just remember 20/20. At a sample size of 20, your metrics (completion rates, time and SUS) will have a margin or error of around +/- 20%.

The relationship between sampling error and sample size is an inverse square root relationship. So to cut your margin of error in half you need to quadruple your sample size. A margin of error of 10% would therefore need around 80 users (see http://www.measuringusability.com/test-margin.php).

So the number I test will depend on how much precision is needed or how large of a difference I'm hoping to detect. But for every test I collect metrics, you never know when they can help make better decisions. It's hard to show a small improvement with small sample sizes, but for large improvements, even very small sample sizes are statistically meaningful.
 


March 21, 2011 | Meredith Noble wrote:

Jeff, quick question – am I correct that any usage of the SUS, or the measurement of time completion, effectively changes a qualitative usability test to a quantitative one, and therefore requires a fairly large sample size (and extremely realistic recruiting) to be valuable? Normally we test 8 people at a time, but I can't imagine comparing SUS before and after without a much larger group of people. Not sure what would be required for statistical significance, but surely more than 8? How many do you test when using these measures? 


March 4, 2011 | José Tavares wrote:

Isn't the SUS questionnaire more suitable to use in the early stages of a product?
If I'm tracking experienced users that use the product for some years now, won't the users be biased by the previous usage of the product?  


March 4, 2011 | José Tavares wrote:

Thanks for this post.
I'll try immediately to sketch a plan to assess the improvements we are making in our product right now.
 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 2 + 1: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4112 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.