Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Should you use 5 or 7 point scales?

Jeff Sauro • August 25, 2010

If you've ever designed a survey or questionnaire you've probably wondered how many points the response options should have. Most questionnaires I've examined either use five point scales or seven-point scales.  Is one better?

7-point scales are slightly better

The short answer is that 7-point scales are a little better than 5-points—but not by much. The psychometric literature suggests that having more scale points is better but there is a diminishing return after around 11 points (Nunnally 1978).  Having seven points tends to be a good balance between having enough points of discrimination without having to maintain too many response options. So what are the consequences of this?

Users need choices

If there aren't enough response options users will be forced to choose the next best alternative and this introduces measurement error. For example, if users think a 5 is too high and a 4 is too low they are forced to settle on an option which is higher or lower than they wanted (assuming they can't pick a 4.5).  If this happens a lot then there aren't enough scale points.

System Usability Scale—Too Much Error?

The most popular usability questionnaire is the System Usability Scale (SUS) which uses ten five-point scales. Is there too much response error with SUS?

A recent article tested the SUS response error by counting the number of times users couldn't decide between two points. The users were allowed to "interpolate" or pick between points such as 3.5.  In 2.5% of the 858 responses users provided responses between two points (95% CI between 1.6% to 3.9%).
 
In contrast, when a 7-point version of the SUS was used there was no interpolating for any of the 840 ratings.
While this seems like compelling evidence to always use 7-point over 5 scales there are two tempering factors.  
  1. While there was error in the five point SUS it is unclear how much of an impact this actually has on the final SUS score since the study used different systems for each scale.

  2. Errors in statistics have a way of cancelling themselves out.  It is likely that many responses that are "forced" into higher numbers will be cancelled out by those forced into lower numbers.
There are many things that can go wrong with questions and responses: users misinterpret the question, users select the wrong box or administrators forget to invert the scales. The data suggests this type of error is small but can be addressed in the following four ways.

1.    For Single Item Questionnaires Use 7 or More Points

If you have only a single question you will get a lot more benefit by providing more response options: 7, 9 or 11 versus 5.  Joe Dumas and I found a single 7-point scale did well.

2.    For Multiple Item Questionnaires it matters Less

As you have more items in your questionnaire it matters less how many scales you have. The SUS has 10 questions meaning it has 50 points of discrimination. In fact the standardized SUMI questionnaire has only 3 point scales but it has 50 questions for a total of 150 points of discrimination.

3.    Don't Change Your Scales!

If you or your company has been using questionnaires with 5 points scales don't change them. Whatever small benefit you get from having more points will be more than offset by losing a valuable benchmark. This is especially the case for SUS which has publically available benchmarks.  

4.    Use 7 Points for New Scales

If you are designing a new scale and need to decide on the number of response options having 7 will likely give you a small benefit over five.  This benefit will likely only be realized if you have fewer response items (less than 10) and you use very large sample sizes.

Keep it in Perspective

In general the effects of usable or unusable applications tend to outweigh the much smaller effects of scale points, labels, scale directions and poorly written questions.  So if you find yourself debating whether there should be 5 or 7 points in your rating scales then you've probably already spent too much time worrying about it.  Focus more effort on finding good benchmarks to compare your results to and on what you'll do with those ratings when you get them back.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Rating Scale, Questionnaires, Survey
.

Posted Comments

There are 15 Comments

July 2, 2014 | Michelle Carter wrote:

I really don’t think 7-point scale would make much of a difference. As long as you provide with a neutral point in your likert scale respondents are cool with it.rnRegards,rn<a href=” http://www.sogosurvey.com/”> SoGoSurvey </a> 


June 17, 2014 | Katy McGregor wrote:

Hi Jeff, we've been using a 5 point scale survey assessing manager effectiveness for the last 3 years. What movement on the scale would you suggest signifies meaningful change? At an individual level (<20), team (<100), divisional (100+) level. Many thanks, Katy  


March 19, 2013 | Bob Thomas wrote:

Jeff, we've been using 5-point Likert scales for the last 5 years and would like to switch to 7-point Likert scales going forward. Any downside to this? 


January 14, 2012 | DMR Panda wrote:

I am thinking of converting data collected in a ten point scale to 5 point scale. Should i simple divide the whole data by 2 or any other method? Please advice. Thanks 


December 14, 2011 | Robert Ceurvorst wrote:

To Maria S: adding scale points does not increase your chances of finding significant differences. You are widening the scale, which may make differences between means larger, but it also increases the variance around the means, so the statistical significance isn't likely to change in any predictable way.  


September 12, 2011 | Maria Staffansson wrote:

I'm currently using a 5-point scale for surveys, but wondering if I should promote a 7-points scale instead. By using a 7-point scale I figure I will get more variable responses allowing more significant statistics on small samples of less than 50 participants...any thoughts on this? 


February 15, 2011 | Jeff Sauro wrote:

I think if you give a respondent the choice of a neutral value then it makes the most sense to compute your mean and standard deviations with all values.

Often researchers want to segment their data and for example, only look at respondents that provided a favorable or unfavorable opinion.

There's nothing wrong with that either. In general, neutral responses are legitimate for many attitudes and forcing responses often just increases the error. So given the choice between forcing and dropping the neutral responses, I'd choose the latter.  


February 10, 2011 | Ricky wrote:

When calculating the results of a survey, I've seen some people throw out the middle answer. This effectively changes the results. Do you think that overall evaluations of a survey should count the middle answer or not? 


September 8, 2010 | Jeff Sauro wrote:

David

Thanks for paper reference. It is indeed full of a lot of helpful information on the topic and emphasizes the subtle nuances and trade-offs. For example, labeling increases acquiescence bias but reduces response error and extreme responding. Negatively worded items increase response error (a highly replicated finding from other studies). Adding more scale points decreases extreme response bias.
But it’s important to keep the results in perspective, in general they appear to be small effects and when you’re using the same questionnaire to make comparisons, these sorts of errors tend matter less (fortunately) and especially when the subject is typically not controversial like attitudes toward usability.
 


September 4, 2010 | David Travis wrote:

You might be interested in a recent paper (Weijters et al, 2010 , Intern J. of Research in Marketing, 27, 236-247) that addresses this issue and shows that it\\\'s a bit more complex. People completing surveys with 7-point scales are more susceptible to extreme responding (i.e. picking one or other of the endpoints). They are also more likely to make mistakes when just the end-points are labelled (as with the SUS). These authors recommend using 5-point scales with each item on the scale fully labelled (e.g. strongly agree, agree, neutral, disagree, strongly disagree). 


August 25, 2010 | Jim Jarrett wrote:

Jeff,

Yes, I agree that the domains tend to have different levels of controversy/risk. I am personally often moving back and forth across the line when doing research... some of it is for marketing/persuasiveness/company reputation, some of it is for design feedback. Also, the persuasive elements of the design do impact usability and vice-versa. So, in a mixed question set, I do tend to keep the neutral/center value in the questions. I also do quite a few "rating" questions on a 10-point scale. Consistency is definitely an important consideration - especially in a single questionnaire.

Basically, when I get access to a real customer/buyer/user, I cast a broad net to get more information for the whole organization. (I'm at a startup now, so that means 60 people, about 20 of which will benefit directly; when I was at Rockwell, I was feeding this info back to more like 2000 people - of the 20,000 in the company - with only 20 paying attention .)

Also, my wife, Teresa, is Assistant Director for Program Evaluation and Development at University of Maryland Extension. So my thinking is influence by her research/evaluation work in extension programs across the state. Many of the results to questions asked in program evaluation are affected quite a bit based on the inclusion of a center/neutral point, and it's an active area of discussion in Extension Evaluation circles (http://www.eval.org/).

Cheers,
Jim 


August 25, 2010 | Jeff Sauro wrote:

Carolyn,
As it happens it is actually an 11-point scale for the ratings. It's a bit deceiving because it starts with 0 instead of 1. I do this so I can compare the results to Net Promoter scores. The Net Promoter scale is also 11 points (from 0 to 10).

That's interesting that you modified the SUS to make it have 7-points instead of 5. There's nothing wrong with doing that other than you won't be able to easily compare your scores with the few public ones that are floating around. This might not be that much of an issue as it sounds like you\'ve got some good historical data for comparisons.

On your final comment about reversing the scales it is actually something I'm collecting data on as we speak...more on that in the next few weeks. 


August 25, 2010 | Carolyn Snyder wrote:

I find it amusing that the scale for rating this article has 10 points :-).

I have always used a 7-point scale for the SUS. Anecdotally, out of a few hundred users I have seen a handful of interpolations. Perhaps I test more ambiguous things than you do?

Thanks for this article. I'd also be interested in seeing one about the pros and cons of reversing the scale for alternating questions.  


August 25, 2010 | Jeff Sauro wrote:

Jim,

So for things like opinion polling or where there is sensitive issues, removing negative options is a sort of trick for forcing people into a response. I don't see much of a need for it in usability data since most attitudes about usability are rather uncontroversial. To that end feeling indifferent or neutral toward an aspect of usability is a legitimate response. See Cox (1980) The optimal number of response alternatives for a scale: A review in the Journal of Marketing Research 17 pp 407-422 in which having a neutral option is preferred.

While I don't have a reference at the moment I can say that the N/A should generally be used sparingly but in attitudes about usability is often a legitimate response option. For example, in Jim Lewis's 3-question ASQ, one question asks "Overall, I am satisfied with the support information (on-line help, messages ,documentation) when completing the tasks ?" You can see where this just isn't applicable in all tasks and so is needed.

A lot of scale construction knowledge comes from marketing and survey research where the topics can be controversial. In many cases the same principles but often the effects are so small that they are overwhelmed by the un-usability of a system. 


August 25, 2010 | Jim Jarrett wrote:

I've also read arguments for dropping the middle point and using an even number of choices where you want to force the response to be either positive or negative, and not neutral, such as in opinion polling (Strongly disagree, agree, disagree, strongly disagree). The neutral position is a "cop out" or a dismissive answer. Any research on this?

Also, what about the use of an "N/A" or "opt out" checkbox for some questions? Any research where that was possible? This allows a person an escape route rather than the malicious compliance of giving an answer they can't or don't want to give. 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 5 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4095 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Usertesting.com

Loop11 Online Usabilty Testing

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.