Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Measuring Usefulness

The Technology Acceptance Model (TAM)

Jeff Sauro • November 1, 2011

Is it easy to use? 

As important as that question is, there's one that's more important: Is it useful?

First and foremost, a product, website or application should solve a problem, fill a need or offer something people find useful. 

In fact, people are willing to put up with poor usability if a product delivers something of great perceived value. Consider how much time you would spend learning to use software if you knew you'd have a guaranteed way to double investments in the stock market? 

Conversely, it doesn't matter how easy to use a product is if people don't find it useful.  Usefulness is the holy-grail of product design, it's often even more important than revenue (think YouTube). How can we predict what people will use?

Predicting Usage

A good first step in predicting how much people will use a new product is to predict how how much they are using an existing product. This was one of the goals of the research done by Fred Davis in the early 1990's[pdf]

He developed what's come to be known as the Technology Acceptance Model (TAM) and a standardized questionnaire[pdf] that measures technology acceptance. The questionnaire consists of 2 parts, 10 items to measure usefulness and 10 items to measure ease of use. 

Participants are asked to provide their level of agreement on a 7 point scale (1=strongly disagree and 7 = strongly agree). Here are the 10 usefulness items:
  1. Using this product improves the quality of the work I do.
  2. Using this product gives me greater control over my work.
  3. This product enables me to accomplish tasks more quickly.
  4. This product supports critical aspects
  5. This product increases my productivity
  6. This product improves my job performance
  7. This product allows me to accomplish more work than would otherwise be possible.
  8. This product enhances my effectiveness on the job
  9. This product makes it easier to do my job
  10. Overall, I find this product useful in my job.
Davis had current users of software (from the late 80's and early 90's) answer the 20 TAM items as well as their current usage of the systems being examined (text editor and electronic mail) to see how well he could predict their usage.

Usefulness is 1.5 times more important than Ease of Use

Davis found the usefulness items were able to predict around 36% of actual reported usage.  The usability items had a more indirect effect on use by impacting attitudes towards usage.  Davis found that answers to the usefulness scores were about 1.5 times more important than usability scores in predicting actual use.

In short, you can explain a lot of whether people will accept or reject a technology based on how useful and usable it is.

Measuring Ease of Use

As it happens, several of the 10 ease of use items Davis used are the same or very similar to those used in the System Usability Scale (SUS). So when measuring product usability it's better to just use the SUS which has been studied refined and is widely used. For website usability use the SUPR-Q which has 4 items that effectively substitute for the SUS, are designed specifically for websites and are normalized against a database of 200 websites.

Caution

It would be great to ask people a few questions to see if they would both use and then purchase a product. Asking people what they want in a new product is notoriously unreliable--as Steve Jobs has famously noted.

We can and should be critical of data based on users' predictions of their future behavior from focus groups, surveys or even the most complex statistical analysis. However, being critical is different than dismissing data altogether. Behavioral metrics are fickle but they're often the only data we have and they are often quite valid predictors. I wonder what the usefulness scores of Netflix would have looked like prior to the recent maligned price and service changes

Here's how to use this information:

  • Consider measuring usefulness: by presenting the 10 TAM items to your users or potential users of a product

  • Measure over time:  See how changes in features correspond to changes in reported use.

  • Predict Usage and adoption: Ask the TAM prior to release of a product then after users have some time with it. See how well their usage was predicted by the TAM items.

  • Generate Benchmarks: Collect enough Technology Acceptance data to create an internal set of benchmarks when prioritizing products as I've done with usability and Net promoter benchmarks.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Questionnaires, SUS
.

Posted Comments

There are 6 Comments

March 20, 2014 | Sandeep wrote:

Hi Jeff, similar to SUS percentile scores, don't we have anything for TAM where we can know if 50 is bad or 60 is bad. Do you have any reference on this? 


February 13, 2014 | Andrew Sutton wrote:

You state "The questionnaire consists of 2 parts, 10 items to measure usefulness and 10 items to measure ease of use." Yet the paper by Fred Davis you link to shows 14 questions for each of these measures. Can you clarify? 


April 12, 2012 | Jennifer wrote:

How is the TAM scored? The SUS has a formula (for odd vs even questions...). Is there something simlar for the TAM? Or perhaps the same formula? 


November 28, 2011 | Jeff Sauro wrote:

The blog originally did not have the strongly disagree in the scale which @ reiner caught. I since corrected it so it is as intended. Sorry for the confusion it caused. 


November 23, 2011 | Bex T wrote:

@ reiner, how does a 7-point scale with 'strongly disagree' at one end and 'strongly agree' at the other only let you agree? Surely that allows you to either agree or to disagree? 


November 10, 2011 | reinier wrote:

the 7 point scale only lets me agree? ;) 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 3 + 5: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[3783 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Usertesting.com

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.