Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Does prior experience affect perceptions of usability?

Jeff Sauro • January 19, 2011

Are attitudes about usability constant?

If we think something is unusable today, will we think it's unusable tomorrow, next week or next year?

How much does prior experience affect how usable we think a websites or software is?

Enough to pay attention to. In a recent assessment, prior experience boosted usability ratings 11% for websites and consumer software. But the effect of experience paled in comparison with the effect of  "usable" and "unusable" websites and products.

In search of that conclusion, I looked to a large database of System Usability Scale (SUS) data that I maintain, which contains information on prior experience.  There is data from 62 websites and 16 consumer software products across almost 2000 users.

Website Experience

Over 1100 users attempted tasks across one of 62 websites (e.g. airlines, rental cars, retailers and government websites), answered the 10-item SUS and provided information on how often they'd been to the website prior to the usability test. 

I split the users into two groups: those who'd never been to the website (first-time users) and those who had been to the website before (repeat users).

On average I found that repeat users rated the websites as 11% more usable than first-time users. Generating a confidence interval around the difference tells us we can be 95% confident that repeat users think the website is between 6% and 15% more usable.


Figure 1: Website SUS scores by experience from 62 websites.

Consumer Software Experience

I next examined SUS scores from 800 users from 16 widely used consumer software products (e.g. Word, Photoshop and Quicken). Respondents provided information on how many years they'd owned and used the software (from less than a year to 10+ years).

I split the users into three categories: less than 3 years, 4-5 years and 5+ years of usage.

On average I found that each level of experience generated 5% higher SUS scores.  The average SUS score of the most experienced user group (those with 5+ years of experience) had 11% higher average SUS scores than those with the least experience (less than 3 years). 


Figure 2: Consumer Software SUS scores by years of experience from 16 products.

We can be 95% confident the most experienced users think the software is between 6% and 15% more usable.

Interestingly enough, both the average difference (11%) and the confidence interval around the difference (6 to 15%) are identical to the website data.

Familiarity Breeds Content

What this data tells us is that prior experience does impact our perceptions of usability as measured by SUS scores.  In general, a user with a lot of prior experience will rate an application as more usable. This will especially be the case between the users with the most experience and those with the least (or none at all).

Track Experience

When gathering SUS data (or any usability data) it's important to track the user's prior experience with a product or website.  It also likely makes sense to report the SUS scores against levels of experience.

Prior experience is often a concern when comparing a new unfamiliar system to an older familiar (and likely less usable system).  Researchers are concerned that the new system gets penalized because it isn't as familiar to the users even though it has bug fixes and simpler designs. They worry that improvements in usability might be masked by the effects of experience.

This data suggests that there is reason to be concerned. Even differences as large as 10% may be caused by differences in exposure.

Adjusting Scores

If a researcher can't mitigate the differences in prior experience, then an adjustment of between 6% and 15% in scores may be warranted.  This would especially be the case if there were large gaps in the years of experience, like the ones shown here, or when comparing new users with existing website users.

For example, imagine users with no prior experience with a new version of a software application generate an average SUS score of 70. Then the average SUS score from users who have been using the existing system for 7 or more years was 80. The new system score can be bumped by 11% to an adjusted SUS score of 77.7—suggesting a more similar perception in usability between systems.

Adjusting the scores in this way estimates what the mean SUS score would be like if the system had been in use for 5+ years.

Keep it in Perspective

In general, experience does matter, but it matters a lot less than differences between usable and unusable websites and software.  On average, experience with a website or software product accounts for only 3% of the difference in SUS scores. To put that into perspective, the difference in the usability of websites is much larger, accounting for around 30% of the variation in SUS scores (10 times greater).
 
The difference between consumer software products accounted for 11% of the variation (almost 4 times greater than experience). This lower amount of variation with the software sample I used is likely due to the more homogeneous nature of these mass-market products. 

Put another way, where we see differences of 6 to 7 points in SUS scores between the extreme levels of experience, we see differences in SUS scores of more than 60 points between the best and worst websites.

When there is little difference between the products being tested, however, experience can matter. You will want to track and report out on experience and likely bump the new version's SUS score for a more equitable comparison.

Editorial services courtesy of Marcia Riefer Johnston. See her "Word Power" blog.



About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

SUS
.

Posted Comments

Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 1 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4080 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Usertesting.com

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.