Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

The Five Most Influential Papers in Usability

Jeff Sauro • July 7, 2010

I compiled a list of papers that have had a large and lasting influence on the field of Usability and User Experience. I then asked Jim Lewis and Joe Dumas, two pioneers in this field for their top five.  There was considerable overlap in both the papers and topics suggesting that while there may be some disagreement with the conclusions of the papers there is strong agreement on their impact.

1. Designing for usability: key principles and what designers think. Gould, J. D. and Lewis, C. (1985)

Go to any UX conference and you'll hear the same points as those raised in this pioneering paper: early and continual focus on users; empirical measurement, iterative design. View Paper [pdf]

Two honorable mentions for pioneering work include:

Al-Awar, J., Chapanis, A., and Ford, R.  (1981).  Tutorials for the first-time computer user.  IEEE Transactions on Professional Communication, 24, 30-37. This is one of the first descriptions of formative usability testing. Prior to this paper most user testing efforts were more summative (benchmark testing).

Shackel, B.  (1990).  Human factors and usability.  In J. Preece and L. Keller (Eds.), /Human-Computer Interaction, Selected Readings/ (pp. 27-41).  This paper defined usability as a function of efficiency, effectiveness & satisfaction (the ISO 9241 pt 11 standard). Despite many proposed extensions, we still think of usability in terms of these three aspects.

2. Heuristic evaluation of user interfaces, Nielsen, J., and Molich, R. (1990)

Despite much criticism, Heuristic Evaluation a "discount usability method" still plays a major role in interface evaluation. HE is best conducted with multiple trained professionals (ideally experts in both UX and the domain) and done prior to and in addition to a user-test. View Paper[pdf]

3. Damaged merchandise? A review of experiments that compare usability evaluation methods.  Gray, W. D., and Salzman, M. C.  (1998).

With the proliferation of discount methods such as Heuristic Evaluation came a number of papers determining which method was "better": User Testing or Heuristic Evaluation?  While there were a lot of good insights from these papers, Damaged Merchandise pointed out in detail that the methodologies used to make the comparisons were poorly designed and lead to fallacious conclusions. View Paper[pdf]

4. Refining the test phase of usability evaluation: how many subjects is enough?. Virzi, R. A. (1992)

Want to get a reaction? Just mention testing with five users to anyone in software development. Few topics elicit more emotion than small sample sizes in user testing. Robert Virzi's paper codified this number into UX history. It was one of a few papers which showed how to use the Binomial (or Poisson) distributions to arrive at the now famous formula. View Paper [Paid Link]

Other Honorable mentions in this category include:

Lewis, James (1982) "Testing Small System Customer Setup[pdf]" in Proceedings of the Human Factors Society 26th Annual Meeting p. 718-720 (1982).This is probably the first to use the binomial for usability problem discovery.

Nielsen, Jakob and Thomas K. Landauer (1993) "A mathematical model of the finding of usability problems [paid link]" Proceedings of the SIGCHI conference on Human factors in computing systems, p.206-213, April 24-29. This paper shows how the Poisson distribution can also be used for usability problem discovery. The Poisson and Binomial generate the same results.

5. Comparative Evaluation of Usability Tests Molich et al  (1998).

Do usability evaluators agree on what usability problems are?  There is considerable variability between usability evaluators. The Comparative Usability Evaluations (CUE) papers and panels have been helping us better understand the limits of human judgment as well as suggesting improvements for usability methods and reports. View Paper [pdf]

In the same year as the first CUE paper at UPA, Jacobsen, Hertzum and John published a paper using a more controlled setup but found similar findings. Since then there have been 8 CUE's and a number of papers written about the Evaluator Effect.

Jacobsen, N. E., Hertzum, M., & John, B. E. (1998) "The Evaluator Effect in Usability Tests[pdf]: Problem Detection & Severity Judgments" in Proceedings of the Human Factors Society 42nd  Annual Meeting p. 1336-1340

What research papers have influenced you ?

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...

Learn More

You Might Also Be Interested In:

Related Topics


Posted Comments

There are 5 Comments

September 23, 2013 | JEFFREY FORSYTH wrote:

Perhaps "seminal" would be more appropriate... 

November 7, 2012 | Martin wrote:

Jeff I am a Ux proffesional and your site is trully and inspiration! 

August 19, 2012 | Doyel Mirza wrote:

Nice post about <a href="">six sigma</a>.
Iím really impressed with your article, such great information you mentioned here..

August 25, 2010 | Jeff Sauro wrote:

Yeah, for a paper to influential it sort of needs to stand the test of time and so this list tends to be of older papers. This is not to say that there aren't many important and relevant findings of late, just that time will tell how influential they'll be.  

August 25, 2010 | mfeldman wrote:

The most recent of these papers is a dozen years old, and some over twice that. No major insights since then, eh? (I suppose "lasting" could mean "at least 12 years old") 

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 1 + 2: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4212 Subscribers]

Connect With Us


Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download