Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Are the Terms Formative and Summative Helpful or Harmful?

Jeff Sauro • February 16, 2010

Most usability testing involves finding and fixing problems as part of an iterative design process to make an interface more usable. It is typically called a Formative Usability Evaluation. In contrast, a Summative Usability Evaluation describes the current usability of an interface—as measured by things like task times, completion rates and satisfaction scores. Summative tells you how usable an interface is and formative tells you what isn't usable.

The terms come from Educational Theory where they are used in the same way except to describe student learning (formative—providing immediate feedback to improve leaning vs. summative—evaluating what was learned).

In general I think the terms are helpful and they are now part of the usability vernacular, however, I do have a major problem with an association. At some point practitioners began associating quantitative metrics with summative evaluations and use these terms interchangeably. Since most usability activities are formative this provides a good excuse for not using metrics or quantifying the usability.
At some point practitioners began using the terms quantitative and summative interchangeably--That's a bad thing.
That's a bad thing. Metrics are not exclusive to summative evaluations or benchmarking. Here are two examples
  1. Completion rates and task times can be valuable diagnostic measures during formative-type tests even when participants are thinking-aloud while they attempt tasks.
  2. You can estimate how much of the user-population is likely to be affected by problems you observed by using confidence intervals and the binomial probability formula.

I've heard practitioners say they don't have time for summative/quantitative evaluations since they're too busy testing designs. I've never done a summative evaluation and not collected UI problem information. If we found problems that needed to be fixed, we fed them back to the development team to address.  In fact, I'd argue there really shouldn't be a pure summative assessment. It's just bad practice to ignore problems users have even if your main goal is to gather a usability benchmark.
 
Photo by sheilaellen
A Formative Usability Test?


It would be one thing if a purely qualitative approach generated consistent and reliable results across practitioners. The CUE studies show that such methods from independent teams are highly variable. While I think almost any usability activity will generate a more usable interface (different ways up the same mountain), there are good and bad practices. The labels formative and qualitative should not mean disguising haphazard efforts, opinions and anecdotes for a legitimate method.

So I've come to think of formative and summative as a simplified way to characterize usability activities even though in practice there isn't as much of a bifurcation. I see them as harmful in that they perpetuate a misconception that quantitative data should only be collected with summative evaluations—which is to say rarely if ever. I advocate disassociating metrics from summative evaluations. Metrics should be collected during any usability evaluation.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar

Related Topics

Formative Testing, Summative
.

Posted Comments

There are 2 Comments

January 15, 2014 | Sarah Rosenbaum wrote:

Hi Jeff - I'm one of those people who've been mixing up formative with qualitative and summative with quantitative. Thanks for setting me straight :-)rnrnJust thought I would let you know that the link to the CUE studies on this page isn't working. I think CUE needs to be in capitol letters. http://www.dialogdesign.dk/CUE.html 


July 5, 2012 | Rosemary Young wrote:

Hi Jeff:

I am a year 3 university placement student with the University of Ulster (mature student 39!) in Biomedical Engineering here in Northern Ireland and I am being introduced to the world of Human Factors Engineering. It is daunting and overwhelming at first (as a newcommer to the subject) but after two weeks of reading, I am beginning to find a passion for the subject.

Thank you for your site, it is helping spell out some aspects, such as Formative and summative etc., in laymans terms and makes it easier to get a firm understanding of the different concepts involved.

Perhaps in the future our paths will meet and we can discuss the HFE when I become more adept!! 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 5 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[3793 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Usertesting.com

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.