Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Blogs & Articles

Missing data in a survey is a pain. What

Identifying the 3 Types of Missing Data

Jeff Sauro • April 15, 2014

Missing data in a survey is a pain. What's worse is if the missing values are systematically excluding a segment of your customers. This blog describes a technique to understand if your data is missing at random, missing completely at random or systematically missing.[Read More]

UX researchers have developed many techniques over the years for testing and validating their ideas.  Here are ten essential methods you can employ on your next project.

10 Essential User Experience Methods

Jeff Sauro • April 7, 2014

UX researchers have developed many techniques over the years for testing and validating their ideas. Here are ten essential methods you can employ on your next project.[Read More]

Measuring Usability has partnered with leading user experience firms to create Cindy (Consistent INterview Drone for usabilitY), the first autonomous usability test facilitator.

Automated Lab Based Usability Testing

Jeff Sauro • April 1, 2014

Measuring Usability has partnered with leading user experience firms to create Cindy (Consistent INterview Drone for usabilitY), the first autonomous usability test facilitator.[Read More]

Every time researchers conduct a usability test to uncover problems they

What the NCAA Tournament & Usability Testing Have in Common

Jeff Sauro • March 24, 2014

Every time researchers conduct a usability test to uncover problems they're also working with probabilities, even if they tell you they hate math! To understand the role of probabilities in usability testing it helps to see how they are used when picking winning teams for the NCAA tournament.[Read More]

Most people are comfortable with the concept of an average or percentage as a measure of quality. An equally important component of measuring the user experience is to understand variability. Here are 10 things to know about measuring variability in the user experience.

10 Things to Know About Variability in the User Experience

Jeff Sauro • March 18, 2014

Most people are comfortable with the concept of an average or percentage as a measure of quality. An equally important component of measuring the user experience is to understand variability. Here are 10 things to know about measuring variability in the user experience.[Read More]

When using statistics to make comparisons between designs, it

Understanding Effect Sizes in User Research

Jeff Sauro • March 11, 2014

When using statistics to make comparisons between designs, it's not enough to just say differences are statistically significant. Effect sizes provide a standardized way to understand the size of the difference across all types of metrics.[Read More]

To get a better understanding of problems and opportunities for your customers, you

4 Types of Customer Analytics Data to Collect

Jeff Sauro • March 4, 2014

To get a better understanding of problems and opportunities for your customers, you'll want to collect data from each of the following four customer analytics groups: descriptive, behavioral, interaction, and attitudinal.[Read More]

It

Is Observing One User Worse Than Observing None?

Jeff Sauro • February 25, 2014

It's possible for product stakeholders to be misled from watching a single session during a usability study. This leads teams to have a 2 or more rule when observing. However, watching just a single user still provides information on the impact of an interface problem. Over time, if stakeholders are consistently watching a single random user in each usability study, odds are they will be more likely to see common problems than freak occurrences.[Read More]

A lot of planning goes into a usability test.  Part of good planning means being prepared for the many things that can go wrong. Here are the ten most common problems we encounter in usability testing and some ideas for how to avoid or manage them when they inevitably occur.

10 Things That Can Go Wrong in a Usability Test

Jeff Sauro • February 18, 2014

A lot of planning goes into a usability test. Part of good planning means being prepared for the many things that can go wrong. Here are the ten most common problems we encounter in usability testing and some ideas for how to avoid or manage them when they inevitably occur.[Read More]

Inevitably one or a few of these survey response rate killers will creep into your next survey project, but sometimes knowing more about them will help reduce or prevent them to help keep your response rate high.

10 Ways to Get a Horrible Survey Response Rate

Jeff Sauro • February 11, 2014

Inevitably one or a few of these survey response rate killers will creep into your next survey project, but sometimes knowing more about them will help reduce or prevent them to help keep your response rate high.[Read More]

Here are four steps to take to get your questionnaire ready for a different language than it was written in: Translate, Back Translate, Pretest and Validate.

4 Steps to Translating a Questionnaire

Jeff Sauro • February 4, 2014

Here are four steps to take to get your questionnaire ready for a different language than it was written in: Translate, Back Translate, Pretest and Validate.[Read More]

While there are books written on measuring usability, it can be easy to get overwhelmed by the details and intimidated by the thought of having to deal with numbers. If I had to use five words to describe some best practices and some core principles of measuring usability, here they are: multi-method, triangulate, redundancy, actions, and attitudes.

Measuring Usability Best Practices in 5 Words

Jeff Sauro • January 28, 2014

While there are books written on measuring usability, it can be easy to get overwhelmed by the details and intimidated by the thought of having to deal with numbers. If I had to use five words to describe some best practices and some core principles of measuring usability, here they are: multi-method, triangulate, redundancy, actions, and attitudes.[Read More]

It

10 Ways to Conduct Usability Tests with Credit Cards & Personal Data

Jeff Sauro • January 21, 2014

It's difficult, and often prohibited, to ask users to use their own information during a usability test. Here are some options that help us get good usability data while working within the constraints of purchasing, logging in, or working with users' sensitive personal data.[Read More]

There are many legitimate criticisms from conclusions drawn using quantitative methods. But often disagreements are rooted more in politics than from p-values. Here are some thoughts on dealing with civil and uncivil criticism when presenting quantitative methods.

How to Deal With Quantitative Criticism

Jeff Sauro • January 14, 2014

There are many legitimate criticisms from conclusions drawn using quantitative methods. But often disagreements are rooted more in politics than from p-values. Here are some thoughts on dealing with civil and uncivil criticism when presenting quantitative methods.[Read More]

An analysis of 500 users

Usability & Net Promoter Benchmarks for Health Insurance Websites

Jeff Sauro • January 7, 2014

An analysis of 500 users' recent experiences across four major healthcare provider websites provides usability and Net Promoter Score benchmarks. We also collected data on what users were trying to accomplish and the major areas for improvement.[Read More]

In 2013 we featured both primary research and relevant research published in peer reviewed journals and websites. Our articles were served-up 2.2. million times to 895k visitors in 2013. Here are the ten articles that received most of those page views.

The Top 10 UX Articles of 2013

Jeff Sauro • December 30, 2013

In 2013 we featured both primary research and relevant research published in peer reviewed journals and websites. Our articles were served-up 2.2. million times to 895k visitors in 2013. Here are the ten articles that received most of those page views.[Read More]

There are a lot of misconceptions about when it is and when it is not appropriate to test with five users. Here are five examples of what you can and cannot learn from just a handful of users in a usability test.

5 Reasons You Should and Should Not Test With 5 Users

Jeff Sauro • December 17, 2013

There are a lot of misconceptions about when it is and when it is not appropriate to test with five users. Here are five examples of what you can and cannot learn from just a handful of users in a usability test.[Read More]

How important is the first choice users make when navigating a website? Turns out it

The Importance of the First Choice in Website Navigation

Jeff Sauro • December 10, 2013

How important is the first choice users make when navigating a website? Turns out it's even more important than we thought. Across 8 tree-test studies we found users were more than six times as likely to succeed in finding an item in a navigation structure if their first choice was down the optimal path.[Read More]

A one year longitudinal study of iPhone usage found interesting patterns in behavior and differences by socioeconomic status.

How Smartphone Usage Changes Over Time and By Income Group

Jeff Sauro • December 3, 2013

A one year longitudinal study of iPhone usage found interesting patterns in behavior and differences by socioeconomic status.

[Read More]

A detailed infographic covering many of the key events, publications and people that have shaped the profession of usability for the last 100 years.

Timeline of Usability Infographic

Jeff Sauro • November 20, 2013

A detailed infographic covering many of the key events, publications and people that have shaped the profession of usability for the last 100 years.[Read More]

See how well you can answer some questions on the core concepts of usability, with an emphasis on usability evaluation.

Usability 101 Quiz

Jeff Sauro • November 12, 2013

See how well you can answer some questions on the core concepts of usability, with an emphasis on usability evaluation.

[Read More]

Focus on decisions and not deliverables. It

5 Tips for Applying Lean UX to User Research

Jeff Sauro • November 5, 2013

Focus on decisions and not deliverables. It's one of the main concepts around Lean thinking in general, and Lean UX in particular. Bloated test plans, power point presentations, pixel-perfect concepts and all the clutter that comes from assembling a team to develop an application or website. Here are five tips for helping make your user research lean.[Read More]

An analysis of problems from nine usability test across different devices revealed no correlation between problem frequency and severity. This means that the first few users in a usability test are just as likely to encounter minor as much as critical problems.

Are Severe Problems Uncovered Earlier in Usability Tests?

Jeff Sauro • October 30, 2013

An analysis of problems from nine usability test across different devices revealed no correlation between problem frequency and severity. This means that the first few users in a usability test are just as likely to encounter minor as much as critical problems.[Read More]

When designing an application, website or product, three things help generate a more usable experience: an early focus on the users and tasks, empirical measurement, and iterative design. These principles were articulated almost 30 years ago in one of the most influential papers in usability. They are just as relevant today.

Designing for Usability: 3 Key Principles

Jeff Sauro • October 22, 2013

When designing an application, website or product, three things help generate a more usable experience: an early focus on the users and tasks, empirical measurement, and iterative design. These principles were articulated almost 30 years ago in one of the most influential papers in usability. They are just as relevant today.[Read More]

There are four major design types with relevance to user research : experimental, quasi-experimental, correlational and single subject. These research designs proceed from a level of high validity and generalizability to ones with lower validity and generalizability.

4 Experiment Types for User Research

Jeff Sauro • October 15, 2013

There are four major design types with relevance to user research : experimental, quasi-experimental, correlational and single subject. These research designs proceed from a level of high validity and generalizability to ones with lower validity and generalizability.[Read More]

There are a lot of methods to use to improve the user experience. We detailed many of them in an earlier blog. We know many of you are visual learners so we created an infographic of when to use which UX method.

UX Methods Infographic

Jeff Sauro • October 9, 2013

There are a lot of methods to use to improve the user experience. We detailed many of them in an earlier blog. We know many of you are visual learners so we created an infographic of when to use which UX method.[Read More]

A usability test plan should not only lay out the framework of the study, but also help identify problems with the methodology, metrics or tasks while something can still be done to fix things. This blog describes the common sections we use.

What Goes into a Usability Test Plan?

Jeff Sauro • October 1, 2013

A usability test plan should not only lay out the framework of the study, but also help identify problems with the methodology, metrics or tasks while something can still be done to fix things. This blog describes the common sections we use. [Read More]

It

6 Steps to Identifying Usability Problems

Jeff Sauro • September 17, 2013

It's relatively easy to spot when users have problems in an interface. It's a lot harder to know what, if anything, to do about them. To identify usability problems more efficiently, start by recording the unintended user actions, identify what in the interface is causing them, assign a severity rating and report the frequency in a problem list with screen shots and context to generate design solutions.[Read More]

Who are the users and what are they trying to do? Answering those two questions are essential first steps to measuring and improving the right things on an interface. A top-tasks analysis is a quick and effective method for obtaining which critical few tasks, among dozens, are the ones a website or software product needs to do well.

How to Conduct a Top Task Analysis

Jeff Sauro • September 10, 2013

Who are the users and what are they trying to do? Answering those two questions are essential first steps to measuring and improving the right things on an interface. A top-tasks analysis is a quick and effective method for obtaining which critical few tasks, among dozens, are the ones a website or software product needs to do well.[Read More]

When the purpose of your website is not to increase traffic or convert browsers to buyers, measuring the return on investment needs different metrics. Associating usability changes to things like reduced support calls, less postage and handling can provide a good understanding of how research projects pay off for users and tax payers.

Measuring Usability ROI for Government Websites

Jeff Sauro • September 4, 2013

When the purpose of your website is not to increase traffic or convert browsers to buyers, measuring the return on investment needs different metrics. Associating usability changes to things like reduced support calls, less postage and handling can provide a good understanding of how research projects pay off for users and tax payers.[Read More]

Naming a product is like naming a baby.  Everyone has opinions and you are stuck with it for a long time!  Don

Measuring the Effectiveness of a Product Name

Jeff Sauro • August 27, 2013

Naming a product is like naming a baby. Everyone has opinions and you are stuck with it for a long time! Don't just rely on the creative brain trust, legal team or CEO's wife to decide which of the ideas will be the new moniker. Test the names with users using some of these methods.[Read More]

Despite the vast power of the internet and search engines, it

17 Periodicals for Usability Research

Jeff Sauro • August 20, 2013

Despite the vast power of the internet and search engines, it's still surprisingly difficult to know where to look to find relevant, peer-reviewed research when you need it. For reading the latest research in usability and human-computer interaction, here are several journals and periodicals that generate quality findings.[Read More]

It

Best Practices for Using Statistics on Small Sample Sizes

Jeff Sauro • August 13, 2013

It's a common misconception that you can't use statistics with small sample sizes (less than 30 or so). Statistical analysis with small samples is like making astronomical observations with binoculars--you are limited to seeing big differences but you can still use the correct procedure to make the most of your data. This blog discusses the latest research on which procedures work for small sample sizes in user research.[Read More]

Qualitative research is often used as a catch-all phrase to mean not to expect any "hard numbers" from research findings. While qualitative research is the collection and analysis of primarily non-numerical activities (words, pictures and actions), it doesn

7 Steps to Conducting Better Qualitative Research

Jeff Sauro • August 6, 2013

Qualitative research is often used as a catch-all phrase to mean not to expect any "hard numbers" from research findings. While qualitative research is the collection and analysis of primarily non-numerical activities (words, pictures and actions), it doesn't mean you can't apply a structured approach to your research efforts.[Read More]

Accounting for problem frequency and severity are two critical ingredients when communicating the importance of usability problems. There have been many different scales proposed for problem severity, in general each method has a similar structure: a set of ordered categories reflecting the impact the problem has on the user, from minor to major.

Rating the Severity of Usability Problems

Jeff Sauro • July 30, 2013

Accounting for problem frequency and severity are two critical ingredients when communicating the importance of usability problems. There have been many different scales proposed for problem severity, in general each method has a similar structure: a set of ordered categories reflecting the impact the problem has on the user, from minor to major.[Read More]

One of the most cost effective and easiest ways to understand who your users are and what they are trying to do is to conduct a true intent study. Using a simple popup on your website allows you to ask core demographic questions, goal-oriented questions and even recruit participants for usability studies.

Measuring Website Visitors' True Intent

Jeff Sauro • July 23, 2013

One of the most cost effective and easiest ways to understand who your users are and what they are trying to do is to conduct a true intent study. Using a simple popup on your website allows you to ask core demographic questions, goal-oriented questions and even recruit participants for usability studies.[Read More]

Navigation is at the heart of the user experience for websites, software and mobile apps. Despite improvements in search, most users still rely on browsing as a strategy to find information or accomplish tasks.  Here are 20 resources on designing and evaluating navigation structures.

20 Resources for Evaluating Website Navigation

Jeff Sauro • July 16, 2013

Navigation is at the heart of the user experience for websites, software and mobile apps. Despite improvements in search, most users still rely on browsing as a strategy to find information or accomplish tasks. Here are 20 resources on designing and evaluating navigation structures.[Read More]

While most usability activities involve finding and fixing problems on websites, it

How to Benchmark Website Usability

Jeff Sauro • July 9, 2013

While most usability activities involve finding and fixing problems on websites, it's good practice to know how usable a website is through a regularly scheduled benchmark. Here are the steps we follow in designing, conducting and analyzing benchmark usability tests.[Read More]

Gathering meaningful insights starts with summarizing raw responses. How to summarize and interpret those responses isn

How to Summarize & Display Survey Data

Jeff Sauro • July 2, 2013

Gathering meaningful insights starts with summarizing raw responses. How to summarize and interpret those responses isn't always immediately obvious. Here are some of the most common survey questions and response options and some ways we've summarized them, including binary responses, single-select, multiple-select, forced-rank, rating scales, Net Promoter, and open comments.[Read More]

Measuring users

Measuring User Confidence in Usability Tests

Jeff Sauro • June 25, 2013

Measuring users' self reported confidence is a valuable metric for diagnosing interaction problems, both by itself and when combined with task completion to generate disaster rates. Where possible, it's best to have an objective measure of task completion along with a self-reported measure like confidence as users generally are overconfident in their ability to complete tasks successfully.[Read More]

The system usability scale is a ten-item questionnaire administered to users for measuring the perceived ease of use of software, hardware, cell phones and websites.  Here are ten things to know about it, including: industry benchmarks, sample sizes, predicting customer loyalty, effects of prior experience and normality.

10 Things to Know About the System Usability Scale (SUS)

Jeff Sauro • June 18, 2013

The system usability scale is a ten-item questionnaire administered to users for measuring the perceived ease of use of software, hardware, cell phones and websites. Here are ten things to know about it, including: industry benchmarks, sample sizes, predicting customer loyalty, effects of prior experience and normality.[Read More]

Is a beautiful website more usable? There have been consistent findings that show perceptions of usability and appearance are correlated. Some recent research suggests that a more usable website improves perceptions of beauty, rather than beauty improving perceptions of usability. Appearance isn

Measuring The Visual Appeal of Websites

Jeff Sauro • June 11, 2013

Is a beautiful website more usable? There have been consistent findings that show perceptions of usability and appearance are correlated. Some recent research suggests that a more usable website improves perceptions of beauty, rather than beauty improving perceptions of usability. Appearance isn't just a beauty contest; there is strong evidence that more attractive websites generate higher opinions of trust which impacts loyalty and sales.[Read More]

The Net Promoter Score is intended to be a proxy for revenue and future growth and is easier to make associations with using user-experience metrics.  You can then indirectly tie changes made through early development research efforts to later measurements of customers

Associating UX Changes to the Net Promoter Score

Jeff Sauro • June 4, 2013

The Net Promoter Score is intended to be a proxy for revenue and future growth and is easier to make associations with using user-experience metrics. You can then indirectly tie changes made through early development research efforts to later measurements of customers' likelihood to recommend through such metrics.[Read More]

It can be a small scale do-it-yourself test or a large sample corporate test, but finding available users can be a burden. The process by which you find your users will vary depending on what you are testing, the types of users you need, and the stage of testing (early versus late). But to help lessen the burden (and remove an excuse not to conduct usability testing) here are seven sources to help find users.

7 Ways to Find Users for Usability Testing

Jeff Sauro • May 28, 2013

It can be a small scale do-it-yourself test or a large sample corporate test, but finding available users can be a burden. The process by which you find your users will vary depending on what you are testing, the types of users you need, and the stage of testing (early versus late). But to help lessen the burden (and remove an excuse not to conduct usability testing) here are seven sources to help find users.[Read More]

While the fundamental metric of findability is whether users find an item or not, often other metrics provide clues to problems in the hierarchy even if users manage to find an item.  Here are 10 metrics to collect and dig into when you

10 Metrics for Testing Website Navigation

Jeff Sauro • May 21, 2013

While the fundamental metric of findability is whether users find an item or not, often other metrics provide clues to problems in the hierarchy even if users manage to find an item. Here are 10 metrics to collect and dig into when you're looking to improve website navigation.[Read More]

You don

Five Techniques for Moderating Usability Tests

Jeff Sauro • May 14, 2013

You don't need to be a trained psychotherapist to conduct effective usability sessions, but it always helps to refine the art of understanding human behavior and intentions when looking to improve the customer experience. Here are five techniques to work on: Not asking why, not planting ideas, minimizing yes/no questions, reducing the "Would You?" questions and turning questions back to the user. [Read More]

For as long as user interfaces have had icons, there have been strong opinions about what makes a good icon. Instead of making decisions based on the pay-grade of the people in a meeting, consider using these seven data driven approaches to help make icons more effective.

Seven Ways to Test the Effectiveness of Icons

Jeff Sauro • May 7, 2013

For as long as user interfaces have had icons, there have been strong opinions about what makes a good icon. Instead of making decisions based on the pay-grade of the people in a meeting, consider using these seven data driven approaches to help make icons more effective.[Read More]

Almost all research contains mistakes in methodology, measurement or interpretation. Rarely do the mistakes render the research completely useless. This blog discussed five of the more common ones. But don

Five User Research Mistakes to Avoid

Jeff Sauro • May 1, 2013

Almost all research contains mistakes in methodology, measurement or interpretation. Rarely do the mistakes render the research completely useless. This blog discussed five of the more common ones. But don't let the fear of mistakes or shortcomings prevent you from conducting new user research.[Read More]

To measure whether users understand a price, concept or design you can

The 3 Rís of Measuring Design Comprehension

Jeff Sauro • April 23, 2013

To measure whether users understand a price, concept or design you can't just ask them. To measure comprehension use the three R's: recognition (multiple choice), recall (open-response) and recounting (explaining to a friend).[Read More]

Crafting task scenarios is a balance between providing enough information and not leading the user. Here are seven tips for crafting a better scenario for usability testing including providing enough context, being specific, using the users

Seven Tips for Writing Usability Task Scenarios

Jeff Sauro • April 16, 2013

Crafting task scenarios is a balance between providing enough information and not leading the user. Here are seven tips for crafting a better scenario for usability testing including providing enough context, being specific, using the users' language and having correct solutions.[Read More]

There are two common uses of the term learnability, one is the ability of an interface to allow users to accomplish tasks on the first attempt.  The second is usability over time. A more learnable system is one that allows users to complete tasks more quickly with less practice with a system.

How to Measure Learnability

Jeff Sauro • April 9, 2013

There are two common uses of the term learnability, one is the ability of an interface to allow users to accomplish tasks on the first attempt. The second is usability over time. A more learnable system is one that allows users to complete tasks more quickly with less practice with a system.[Read More]

A tree test is like a usability test on the skeleton of your navigation with the design "skin" removed. It allows you to isolate problems in findability in your taxonomy, groups or labels that are not attributable to the designs or search engine. Here are some common questions and answers about the method including when to use it, how to interpret the output, the number of items to test and how long it takes.

Using Tree-Testing To Test Information Architecture

Jeff Sauro • March 26, 2013

A tree test is like a usability test on the skeleton of your navigation with the design "skin" removed. It allows you to isolate problems in findability in your taxonomy, groups or labels that are not attributable to the designs or search engine. Here are some common questions and answers about the method including when to use it, how to interpret the output, the number of items to test and how long it takes.[Read More]

Card sorting is a popular method for improving the organization of websites and software. Here are some common questions and answers about the method including when to use it, how to interpret the output, the number of items to test and how long it takes.

Using Card Sorting To Test Information Architecture

Jeff Sauro • March 19, 2013

Card sorting is a popular method for improving the organization of websites and software. Here are some common questions and answers about the method including when to use it, how to interpret the output, the number of items to test and how long it takes.[Read More]

There are a number of popular methods used in improving the user experience at all phases of research and design. Here are over twenty of the more popular methods we use and when we use them in the product development cycle.

What UX Methods to Use and When to Use Them

Jeff Sauro • March 12, 2013

There are a number of popular methods used in improving the user experience at all phases of research and design. Here are over twenty of the more popular methods we use and when we use them in the product development cycle.[Read More]

Usability is hardly physics or chemistry. But there are some important principles from decades of research in Human Computer Interaction (HCI) that apply to design and user research. Here are five famous laws that can be applied to improving the user experience of applications and websites: Miller

Five HCI Laws for User Experience Design

Jeff Sauro • March 5, 2013

Usability is hardly physics or chemistry. But there are some important principles from decades of research in Human Computer Interaction (HCI) that apply to design and user research. Here are five famous laws that can be applied to improving the user experience of applications and websites: Miller's Law, Fitts' Law, Hick-Hyman Law, Power Law of Practice and the Pareto Law.[Read More]

Face-to-face usability sessions provide rich feedback without the biases introduced by multi-person sessions, but at a cost. One-on-one in person sessions can be time consuming, hard to recruit for, biased, expensive and a pain to schedule. For the same cost we recommend a mix of surveys, moderated (remote and in-person) and unmoderated testing to maximize usability testing budgets.

5 Ways to Maximize your Usability Testing Efforts

Jeff Sauro • February 26, 2013

Face-to-face usability sessions provide rich feedback without the biases introduced by multi-person sessions, but at a cost. One-on-one in person sessions can be time consuming, hard to recruit for, biased, expensive and a pain to schedule. For the same cost we recommend a mix of surveys, moderated (remote and in-person) and unmoderated testing to maximize usability testing budgets. [Read More]

Six Sigma is a methodology for improving the quality of everything from the manufacturing of minute electronic parts to the development of complex software. While there are many tools in the six sigma toolbox, here are six that we apply when improving the user experience of websites, software and hardware.

6 Tools for Applying Six Sigma to the User Experience

Jeff Sauro • February 19, 2013

Six Sigma is a methodology for improving the quality of everything from the manufacturing of minute electronic parts to the development of complex software. While there are many tools in the six sigma toolbox, here are six that we apply when improving the user experience of websites, software and hardware.[Read More]

The profession of usability as we know it largely started in the 1980s. Many methods have their roots in the earlier fields of Ergonomics and Human Factors which began near the beginning of the 20th century and had a strong influence through the World War II. While not exhaustive, the following is a timeline of several key events, people and publications that have shaped the history and future of usability.

A Brief History of Usability

Jeff Sauro • February 11, 2013

The profession of usability as we know it largely started in the 1980s. Many methods have their roots in the earlier fields of Ergonomics and Human Factors which began near the beginning of the 20th century and had a strong influence through the World War II. While not exhaustive, the following is a timeline of several key events, people and publications that have shaped the history and future of usability.[Read More]

Hypothesis testing is at the heart of modern statistical thinking and a core part of the Lean methodology. Instead of approaching design decisions with pure instinct and arguments in conference rooms, form a testable statement, invite users, define metrics, collect data and draw a conclusion.

Hypothesis Testing in the User Experience

Jeff Sauro • February 5, 2013

Hypothesis testing is at the heart of modern statistical thinking and a core part of the Lean methodology. Instead of approaching design decisions with pure instinct and arguments in conference rooms, form a testable statement, invite users, define metrics, collect data and draw a conclusion.[Read More]

In building a better experience, there are many questions about mobile device usage and how designers can best meet users

15 Mobile UX Facts and Insights

Jeff Sauro • January 29, 2013

In building a better experience, there are many questions about mobile device usage and how designers can best meet users' needs with apps and responsive designs. From usage and ownership to purchasing behavior, here are 15 data points to help in the decision making.[Read More]

The process of thinking through potential problems and knowing how to detect and prevent problems before they create a UX catastrophe is what the Failure Modes Effect Analysis (FMEA) is all about. It takes into account problem frequency, severity and ability to detect into a single priority number.

Prioritizing Problems in the User Experience: The FMEA

Jeff Sauro • January 22, 2013

The process of thinking through potential problems and knowing how to detect and prevent problems before they create a UX catastrophe is what the Failure Modes Effect Analysis (FMEA) is all about. It takes into account problem frequency, severity and ability to detect into a single priority number.[Read More]

When we have a frustrating or poor experience, we can hate the product, tell our friends about our bad experience, post it to Twitter, and we probably will not use the service or product again. Using some familiar usability metrics and re-framing them, we can understand and eliminate poor experiences.

The Value of Measuring Poor User Experiences

Jeff Sauro • January 15, 2013

When we have a frustrating or poor experience, we can hate the product, tell our friends about our bad experience, post it to Twitter, and we probably will not use the service or product again. Using some familiar usability metrics and re-framing them, we can understand and eliminate poor experiences.[Read More]

Too much to do and too little timeóthat

Prioritizing UI Improvements: The QFD

Jeff Sauro • January 8, 2013

Too much to do and too little timeóthat's product design. If you tried to fix or improve everything, you'd never get any new products released. You have to prioritize. Prioritize is exactly what the Quality Function Deployment (QFD) method does. It provides a structured way to prioritize features, functions or even website content by taking into account both business priorities (the voice of the company) and the customer/user priorities (the voice of the customer).[Read More]

On MeasuringUsability.com our servers saw 596k visitors and 1.6 Million page views in 2012. We wrote 49 articles and created two calculators.  The 10 most popular articles each received at least 10,000 page views with the top three seeing more than 20k.

The 10 Most Popular UX Articles from 2012

Jeff Sauro • January 2, 2013

On MeasuringUsability.com our servers saw 596k visitors and 1.6 Million page views in 2012. We wrote 49 articles and created two calculators. The 10 most popular articles each received at least 10,000 page views with the top three seeing more than 20k.[Read More]

Surveys are ubiquitous.  We answer them and we develop them because they are an efficient means for collecting feedback and insight from current and prospective customers. Here are a mix of ten tips, insights and philosophies to consider for your next survey.

10 Tips For Your Next Survey

Jeff Sauro • December 18, 2012

Surveys are ubiquitous. We answer them and we develop them because they are an efficient means for collecting feedback and insight from current and prospective customers. Here are a mix of ten tips, insights and philosophies to consider for your next survey.[Read More]

You donít need to be a mathematician to quantify the problems and improvements in user interfaces. Often the most compelling metrics are simple to compute and require no more than arithmetic and basic algebra. While most of us were exposed to these concepts in 8th and 9th grade, they are easy to forget and probably didnít seem applicable when not learned in context. Here are a few fundamental concepts to help with quantifying the user experience.

8 Core Concepts for Quantifying the User Experience

Jeff Sauro • December 11, 2012

You donít need to be a mathematician to quantify the problems and improvements in user interfaces. Often the most compelling metrics are simple to compute and require no more than arithmetic and basic algebra. While most of us were exposed to these concepts in 8th and 9th grade, they are easy to forget and probably didnít seem applicable when not learned in context. Here are a few fundamental concepts to help with quantifying the user experience.[Read More]

Few things are more revealing than watching just a few users attempt tasks on a website or software.While watching users you should ask them some key questions to help put the observations into perspective. Here are eight recommendations for helping quantify both attitudes and put the insightful observations into context.

Asking the Right User Experience Questions

Jeff Sauro • December 5, 2012

Few things are more revealing than watching just a few users attempt tasks on a website or software.While watching users you should ask them some key questions to help put the observations into perspective. Here are eight recommendations for helping quantify both attitudes and put the insightful observations into context. [Read More]

Whether you

10 Things to Know about A/B Testing

Jeff Sauro • November 27, 2012

Whether you're new to A/B testing or a seasoned practitioner, here are 10 things you should know about this essential method for quantifying the user experience: sample size, statistical significance, stopping early and practical significance from confidence intervals around the difference.[Read More]

Unmoderated usability testing is one of the fastest growing usability methods. Here are 10 things to know about this essential approach.

10 Things to Know About Unmoderated Usability Testing

Jeff Sauro • November 13, 2012

Unmoderated usability testing is one of the fastest growing usability methods. Here are 10 things to know about this essential approach.[Read More]

In a survey of over 2000 users from a selection of 30 high traffic websites, we asked what one thing users would improve. We found some usual suspects: search, navigation, appearance along with security and information content.

One thing you should fix on your website

Jeff Sauro • November 7, 2012

In a survey of over 2000 users from a selection of 30 high traffic websites, we asked what one thing users would improve. We found some usual suspects: search, navigation, appearance along with security and information content.[Read More]

After users attempt a task, record how difficult they found it using the Single Ease Question (SEQ). We use the SEQ on websites, mobile devices, software and paper-prototypes to generate a reliable measure of perceived task difficulty over time and against comparable interfaces.

10 Things To Know About The Single Ease Question (SEQ)

Jeff Sauro • October 30, 2012

After users attempt a task, record how difficult they found it using the Single Ease Question (SEQ). We use the SEQ on websites, mobile devices, software and paper-prototypes to generate a reliable measure of perceived task difficulty over time and against comparable interfaces.[Read More]

In order to improve findability, use the same core usability metrics to assess how well users can find items. Define what users are trying to find, measure how well they can currently find it, make improvements to the navigation and include cross-linking, then measure the same items and compare statistically.

How to Measure Findability

Jeff Sauro • October 23, 2012

In order to improve findability, use the same core usability metrics to assess how well users can find items. Define what users are trying to find, measure how well they can currently find it, make improvements to the navigation and include cross-linking, then measure the same items and compare statistically.[Read More]

Quantifying the user experience is the first step to making measured improvements. One of the first questions with any metrics is "what

10 Benchmarks for User Experience Metrics

Jeff Sauro • October 16, 2012

Quantifying the user experience is the first step to making measured improvements. One of the first questions with any metrics is "what's a good score?". Like in sports, a good score depends on the metric and context. Here are 10 benchmarks with some context to help make your metrics more manageable.[Read More]

An analysis of 122 usability studies with data from 833 users found that within a test, task performance and System Usability Scale scores have a low correlation. When aggregated at a higher level, SUS scores tracked completion rates better, suggesting SUS scores alone can estimate, at least crudely, task performance.

Predicting Task Completion with the System Usability Scale

Jeff Sauro • October 9, 2012

An analysis of 122 usability studies with data from 833 users found that within a test, task performance and System Usability Scale scores have a low correlation. When aggregated at a higher level, SUS scores tracked completion rates better, suggesting SUS scores alone can estimate, at least crudely, task performance.[Read More]

By estimating the percent of customers that recommend a website or product, those who were referred by someone, plus the "likelihood to recommend" question, you can estimate how many promoters it takes to generate enough positive word-of-mouth to net a new customer.

Quantifying the Value of a Promoter

Jeff Sauro • October 2, 2012

By estimating the percent of customers that recommend a website or product, those who were referred by someone, plus the "likelihood to recommend" question, you can estimate how many promoters it takes to generate enough positive word-of-mouth to net a new customer.[Read More]

As UX continues to mature it

Five Critical Quantitative UX Concepts

Jeff Sauro • September 25, 2012

As UX continues to mature it's becoming harder to avoid using statistics to quantify design improvements. Here are five of the more critical but challenging concepts that take practice and patience but are worth the effort to understand.[Read More]

The Pareto Principle, also called the 80/20 rule shows that a small percentage of things explain a large percentage of outcomes. In user research, an effective strategy is to separate the vital few from the trivial many to improve the user experienceówhether it be usability problems, feature requests, calls into support, software bugs or revenue.

Applying the Pareto Principle to the User Experience

Jeff Sauro • September 12, 2012

The Pareto Principle, also called the 80/20 rule shows that a small percentage of things explain a large percentage of outcomes. In user research, an effective strategy is to separate the vital few from the trivial many to improve the user experienceówhether it be usability problems, feature requests, calls into support, software bugs or revenue.[Read More]

A heuristic evaluation of two websites from four evaluators found 32% of the issues 50 users had during a usability test. Heuristic evaluations typically find between 30% and 50% of issues found in usability tests. In general, HE is best used in addition to usability testing to generate the most complete picture of potential and actual interface problems.

How Effective are Heuristic Evaluations?

Jeff Sauro • September 6, 2012

A heuristic evaluation of two websites from four evaluators found 32% of the issues 50 users had during a usability test. Heuristic evaluations typically find between 30% and 50% of issues found in usability tests. In general, HE is best used in addition to usability testing to generate the most complete picture of potential and actual interface problems.[Read More]

In user research, it

Triangulate for Better User Research

Jeff Sauro • August 28, 2012

In user research, it's hard to know which method or which metric will generate the right insights a head of time. By using multiple methods and measures and in effect "averaging" the results you provide the most reliable picture of the user experience.[Read More]

Usability testing, like most studies of human behavior has biases. Many of these biases stem from the observer effect--participants know they

9 Biases in Usability Testing

Jeff Sauro • August 21, 2012

Usability testing, like most studies of human behavior has biases. Many of these biases stem from the observer effect--participants know they're being watched and act differently. Biases can never be entirely eliminated. The best remedy is often being aware of them and communicating their potential impact on decisions.[Read More]

We review an experiment where four independent evaluators inspected two websites (Budget.com and Enterprise.com). The results are consistent with earlier research: different evaluators find different issues, experts find more but not all issues compared to novices, and concerns about false alarms and misses persist with inspection methods compared to usability testing.

The Value of Multiple Evaluators in Heuristic Evaluations

Jeff Sauro • August 14, 2012

We review an experiment where four independent evaluators inspected two websites (Budget.com and Enterprise.com). The results are consistent with earlier research: different evaluators find different issues, experts find more but not all issues compared to novices, and concerns about false alarms and misses persist with inspection methods compared to usability testing.[Read More]

The focus group has its place in UX research but is usually the wrong tool for the job. Combining some aspects of focus groups with usability testing in a 1:1 session allows the researcher to gather both motivations as well as task-based outcomes with an interface.

The F-Word in User Experience

Jeff Sauro • August 7, 2012

The focus group has its place in UX research but is usually the wrong tool for the job. Combining some aspects of focus groups with usability testing in a 1:1 session allows the researcher to gather both motivations as well as task-based outcomes with an interface.[Read More]

Personas are popular in UX research with 65% of practitioners using them. A persona is more than just a user or something the marketing department made up. Here are seven core ideas everyone should know about personas in UX.

7 Core Ideas about Personas and The User Experience

Jeff Sauro • July 31, 2012

Personas are popular in UX research with 65% of practitioners using them. A persona is more than just a user or something the marketing department made up. Here are seven core ideas everyone should know about personas in UX.[Read More]

Lean is not a new diet plan. It

Lean Doesn't Mean Less: Understanding Lean UX

Jeff Sauro • July 24, 2012

Lean is not a new diet plan. It's keeping the quality while reducing the waste in the processes. It's not doing fewer methods to improve the user experience--it's refining the methods so we spend less time processing, producing and waiting.[Read More]

Card sorting is a popular method for understanding users

Card Sorting + Tree Testing : The Science of Great Site Navigation

Jeff Sauro • July 17, 2012

Card sorting is a popular method for understanding users' mental models for improving website navigation. The method is also best used in conjunction with tree-testing (reverse card sorting). This blog reviews the combined approach we used in the Target.com navigation case study of an open card sort and tree-test. It also includes answers to the questions asked during the webinar.[Read More]

Have you wanted to purchase something online, but couldn

9 Insights from ecommerce Usability Studies

Jeff Sauro • July 10, 2012

Have you wanted to purchase something online, but couldn't? Here are nine things we see rather often in our usability tests of eCommerce websites: not showing shipping +tax prior to checkout, registration, saving form information and the quality of user reviews.[Read More]

While the look of our interfaces has changed, the constant demand to respond to alerts hasn

Over Messaging the User Experience

Jeff Sauro • July 4, 2012

While the look of our interfaces has changed, the constant demand to respond to alerts hasn't. There are friendlier ways of alerting users, however, it's less about how the message is displayed but the fact that it is displayed. In the end it's finding ways of having fewer messages, with less content, less often.[Read More]

The one question everyone using statistics asks is : What statistical procedure do I use? In this blog I

What Statistical Test do I Use?

Jeff Sauro • June 26, 2012

The one question everyone using statistics asks is : What statistical procedure do I use? In this blog I've created a clickable decision map which links to the correct procedure and online calculator after answering a couple questions.[Read More]

If they can find it they will buy it. But how will they find it?  An evaluation of 1500 users across 9 websites and 25 tasks found that about 14% of users start with search. While the percent of users browsing versus searching varies based on the task, website and prominence of the search box, in general most users start browsing.

Search vs. Browse on Websites

Jeff Sauro • June 19, 2012

If they can find it they will buy it. But how will they find it? An evaluation of 1500 users across 9 websites and 25 tasks found that about 14% of users start with search. While the percent of users browsing versus searching varies based on the task, website and prominence of the search box, in general most users start browsing.[Read More]

In a study of 881 users across 5 usability studies and 38 tasks, we found men were about 6% more confident than women when they failed a task.  As data from other studies suggest while both genders tend to be overconfident in their abilities, in the absence of a clear indication of failure, men tend to assume they were successful.

Are men overconfident users?

Jeff Sauro • June 12, 2012

In a study of 881 users across 5 usability studies and 38 tasks, we found men were about 6% more confident than women when they failed a task. As data from other studies suggest while both genders tend to be overconfident in their abilities, in the absence of a clear indication of failure, men tend to assume they were successful.[Read More]

Organizing usability problems into a user by problem matrix helps estimate the prevalence of problems, the percent of problems uncovered and increases the clarity and credibility of usability testing methods.

Report Usability Issues in a User by Problem Matrix

Jeff Sauro • June 6, 2012

Organizing usability problems into a user by problem matrix helps estimate the prevalence of problems, the percent of problems uncovered and increases the clarity and credibility of usability testing methods.[Read More]

There is no usability thermometer to tell you how easy to use a website or software application is. By averaging together the common metrics of effectiveness (completion rates, errors), efficiency (time) and satisfaction (task-level satisfaction) we generate an accurate and actionable picture of usability.

10 Things to Know about the Single Usability Metric (SUM)

Jeff Sauro • May 30, 2012

There is no usability thermometer to tell you how easy to use a website or software application is. By averaging together the common metrics of effectiveness (completion rates, errors), efficiency (time) and satisfaction (task-level satisfaction) we generate an accurate and actionable picture of usability.[Read More]

My thoughts on measuring the customer experience, including challenges to getting a company wide measure and some best practices for measuring and improving.

Measuring the Customer Experience: Questions and Answers

Jeff Sauro • May 22, 2012

My thoughts on measuring the customer experience, including challenges to getting a company wide measure and some best practices for measuring and improving.[Read More]

Errors can be categorized as slips (like a typo) or a mistake (incorrect goal) and are common occurrences in usability tests. Errors are often caused by problems in an interface and lead to longer task times, higher task failure and lower satisfaction ratings. While errors can

Measuring Errors in the User Experience

Jeff Sauro • May 15, 2012

Errors can be categorized as slips (like a typo) or a mistake (incorrect goal) and are common occurrences in usability tests. Errors are often caused by problems in an interface and lead to longer task times, higher task failure and lower satisfaction ratings. While errors can't be entirely eliminated, they can often be reduced substantially by reducing the opportunity for an error.[Read More]

Regardless of whether you

5 Valuable Skills for UX Professionals

Jeff Sauro • May 9, 2012

Regardless of whether you're more on the research side or more on the design side of the User Experience, here are five skills that will make you more valuable and effective in your job.[Read More]

When it comes to testing websites there are many unmoderated and moderated solutions. But if you

How to Conduct a Usability test on a Mobile Device

Jeff Sauro • May 1, 2012

When it comes to testing websites there are many unmoderated and moderated solutions. But if you've ever tried to evaluate an app or website on a mobile phone or tablet there are fewer options. Here's our solution.[Read More]

Increasingly companies are adopting the Net Promoter Score as the corporate metric. All metrics, including user experience metrics should roll up to the Net Promoter Score. Here are 10 things to know about the Net Promoter Score if you

10 Things to Know about Net Promoter Scores and the User Experience

Jeff Sauro • April 24, 2012

Increasingly companies are adopting the Net Promoter Score as the corporate metric. All metrics, including user experience metrics should roll up to the Net Promoter Score. Here are 10 things to know about the Net Promoter Score if you're concerned about improving the user experience.[Read More]

One of the simplest ways to measure any event is a binary metric coded as a 1 or 0. It

6 Proportions to Compare when Improving the User Experience

Jeff Sauro • April 17, 2012

One of the simplest ways to measure any event is a binary metric coded as a 1 or 0. It's at the heart of computing and plays a critical role in user research.[Read More]

Compare two independent proportions for A/B testing or comparing completion rates or conversion rates for small and large sample sizes. Uses the N-1 Two proportion test and Fisher Exact test to generate p-values.

A/B Test Calculator

Jeff Sauro • April 17, 2012

Compare two independent proportions for A/B testing or comparing completion rates or conversion rates for small and large sample sizes. Uses the N-1 Two proportion test and Fisher Exact test to generate p-values.[Read More]

Despite the rise in unmoderated usability testing, the bulk of evaluations are still done with a facilitator.
Whether you are sitting next to the user in a lab or sharing screens with someone thousands of miles away, here are 20 practical tips for your next moderated usability test.

20 Tips for your Next Moderated Usability Test

Jeff Sauro • April 10, 2012

Despite the rise in unmoderated usability testing, the bulk of evaluations are still done with a facilitator. Whether you are sitting next to the user in a lab or sharing screens with someone thousands of miles away, here are 20 practical tips for your next moderated usability test.[Read More]

When you need a user-centered view of categories, labels or groups to improve findability, card sorting is the way to go. Here are at least 10 things you should know about this popular user research method.

10 Things to Know about Card Sorting

Jeff Sauro • April 3, 2012

When you need a user-centered view of categories, labels or groups to improve findability, card sorting is the way to go. Here are at least 10 things you should know about this popular user research method.[Read More]

Standardized Usability Questionnaires offer the advantage of higher reliability, validity and sensitivity and some offer the advantage of a normalized database which allows you to compare a score to hundreds of others.

8 Advantages of Standardized Usability Questionnaires

Jeff Sauro • March 27, 2012

Standardized Usability Questionnaires offer the advantage of higher reliability, validity and sensitivity and some offer the advantage of a normalized database which allows you to compare a score to hundreds of others.[Read More]

Compute confidence intervals around continuous data using either raw or summary data.

Confidence Interval Calculator

Jeff Sauro • March 21, 2012

Compute confidence intervals around continuous data using either raw or summary data.

[Read More]

You don

10 Things to know about Confidence Intervals

Jeff Sauro • March 21, 2012

You don't need a PhD in statistics to understand and use confidence intervals. Here are 10 things to know to get you started using and interpreting confidence intervals for your next user research project.[Read More]

Rarely can we talk to all users in the population we

7 S's of User Research Sampling

Jeff Sauro • March 13, 2012

Rarely can we talk to all users in the population we're studying, instead we sample. Here are 7 S's to help in your sampling: Simple Random, Starbucks, Stratified, Snowball, Spot, Sequential and Serial sampling.[Read More]

Many of the reasons people don

Nine misconceptions about statistics and usability

Jeff Sauro • March 7, 2012

Many of the reasons people don't use statistics with usability data are based on misconceptions about what you can and can't do with statistics and the advantage they provide in reducing uncertainly and clarifying recommendations. Here are nine of the more common misconceptions I've heard.[Read More]

After the successful webinar on Best Practices for Remote Usability Testing, we received many questions about how I performed the analysis: sample size questions, time on task and other logistic issues are covered.

20 Questions Answered about Unmoderated Usability Testing

Jeff Sauro • February 29, 2012

After the successful webinar on Best Practices for Remote Usability Testing, we received many questions about how I performed the analysis: sample size questions, time on task and other logistic issues are covered.[Read More]

The right measure will: identify problem areas, track improvements over time, be meaningful to the customer. The wrong measure can: identify wrong areas of focus, miss problems all together, lead to unintended consequences and alienate customers. Finding the right measure means taking multiple measures and seeing which one best tracks other customer sentiments and revenue.

Managing the Right Customer Experience Measure

Jeff Sauro • February 21, 2012

The right measure will: identify problem areas, track improvements over time, be meaningful to the customer. The wrong measure can: identify wrong areas of focus, miss problems all together, lead to unintended consequences and alienate customers. Finding the right measure means taking multiple measures and seeing which one best tracks other customer sentiments and revenue.[Read More]

There are more than five challenges facing UX professionals today, but here are five that tend to cross projects and products: Time, Costs, Tools & Techniques, Finding Representative Users and Deliverables.

Five challenges and pain points facing UX professionals today

Jeff Sauro • February 14, 2012

There are more than five challenges facing UX professionals today, but here are five that tend to cross projects and products: Time, Costs, Tools & Techniques, Finding Representative Users and Deliverables. [Read More]

An analysis of 4000 users across 59 websites found that when users fail to accomplish their goals, they are 5 times less likely to return to the website and 3 times more likely to tell their friends not to visit the website.

The high cost of task failure on websites

Jeff Sauro • February 7, 2012

An analysis of 4000 users across 59 websites found that when users fail to accomplish their goals, they are 5 times less likely to return to the website and 3 times more likely to tell their friends not to visit the website.[Read More]

An analysis of SUPR-Q data from users of popular social networking sites reveals users generally don

Distrust in Social Networks: Google+, Twitter, Facebook

Jeff Sauro • January 31, 2012

An analysis of SUPR-Q data from users of popular social networking sites reveals users generally don't trust social networks. Facebook, Twitter and Google+ all fall within the bottom quintile of website trust. Facebook leads the pack with the highest Net Promoter Score and usability.[Read More]

The distinction between a qualitative study and quantitative study is a false dichotomy. It doesn

5 Examples of Quantifying Qualitative Data

Jeff Sauro • January 24, 2012

The distinction between a qualitative study and quantitative study is a false dichotomy. It doesn't cost more money to quantify or use statistics. It just takes some training and confidence. Not only can qualitative data be categorized into quantities, but it can prompt further questions and discovery for usability improvement.[Read More]

There are advantages and disadvantages to the different usability testing methods: lab-based, remote moderated and remote unmoderated. A combination of methods provides a more comprehensive picture of the user experience but is not always possible. Consider these nine factors when deciding on a method.

Comparison of Usability Testing Methods

Jeff Sauro • January 17, 2012

There are advantages and disadvantages to the different usability testing methods: lab-based, remote moderated and remote unmoderated. A combination of methods provides a more comprehensive picture of the user experience but is not always possible. Consider these nine factors when deciding on a method.[Read More]

Despite UX being a field with wide ranging skills, the average salaries for individual contributors are about the same across job functions at about $85k and haven

Salaries for UX Researchers, Designers, Managers and IAs

Jeff Sauro • January 10, 2012

Despite UX being a field with wide ranging skills, the average salaries for individual contributors are about the same across job functions at about $85k and haven't changed much over two years.[Read More]

System Usability Scale (SUS) scores are often collected along with Net Promoter Scores in evaluations of software and website usability. An examination of 81 datasets from 2200 users shows that dividing SUS scores by 10 does a decent job of predicting the Net Promoter Score.

Predicting Net Promoter Scores from System Usability Scale Scores

Guest Post By Jim Lewis • January 3, 2012

System Usability Scale (SUS) scores are often collected along with Net Promoter Scores in evaluations of software and website usability. An examination of 81 datasets from 2200 users shows that dividing SUS scores by 10 does a decent job of predicting the Net Promoter Score.[Read More]

Thank you to the 585k visitors and 1.3 Million page views on MeasuringUsability.com in 2011. Of the 52 articles written in 2011, in ascending order, here are the 15 most popular.

The Most Popular UX Articles of 2011

Jeff Sauro • December 27, 2011

Thank you to the 585k visitors and 1.3 Million page views on MeasuringUsability.com in 2011. Of the 52 articles written in 2011, in ascending order, here are the 15 most popular.[Read More]

Improving the user experience means starting with the right measure or measures to manage. Here are 10 of the more common ones I

10 Ways to Measure & Manage the User Experience

Jeff Sauro • December 20, 2011

Improving the user experience means starting with the right measure or measures to manage. Here are 10 of the more common ones I've written about in 2011.[Read More]

The only thing worse than users failing a task is users failing a task and thinking they

Measuring User Interface Disasters

Jeff Sauro • December 14, 2011

The only thing worse than users failing a task is users failing a task and thinking they've completed it successfully. This is a disaster. Disasters can be tracked by measuring task completion rates and task-level confidence. Data from 174 tasks show the likely prevalence of disasters in consumer software and websites.[Read More]

What sample size do i need? It

How to find the right sample size for a Usability Test

Jeff Sauro • December 7, 2011

What sample size do i need? It's usually the first and most difficult question to answer when planning a usability evaluation. There are actually good ways for estimating the sample size that don't rely on intuition, dogma or conventions.[Read More]

There isn

10 Essential Usability Metrics

Jeff Sauro • November 30, 2011

There isn't a usability thermometer to tell you how usable your software or website is. Instead we rely on the impact of good and bad usability to assess the quality of the user experience. Assessing that impact starts by knowing and collecting these 10 metrics.[Read More]

The reported number of UX professionals in both small and large organizations has increased between 20% and 30% over the last two years. This can be attributed to both an increase in actual UX professionals and a broadening of the jobs that fall under the UX umbrella.

The Growth of UX Organizations

Jeff Sauro • November 22, 2011

The reported number of UX professionals in both small and large organizations has increased between 20% and 30% over the last two years. This can be attributed to both an increase in actual UX professionals and a broadening of the jobs that fall under the UX umbrella.[Read More]

For as long as there have been websites it seems that there

Click versus Clock: Measuring Website Efficiency

Jeff Sauro • November 15, 2011

For as long as there have been websites it seems that there's been a call to reduce the number of clicks to improve the user experience. An analysis of click counts and task times across 3 eCommerce website usability tests and 1200 users found that clicks and time have a correlation of .5. Clicks predict around 25% of task time, meaning it's better to directly measure or estimate task times than click counts when improving website efficiency.[Read More]

There are many great methods for gathering insights from users and many more software tools. Here are the tools and services I use when conducting user research.

The Right Tool for the User Research Method

Jeff Sauro • November 8, 2011

There are many great methods for gathering insights from users and many more software tools. Here are the tools and services I use when conducting user research.[Read More]

A product must be useful and usable to be adopted and both have been shown to predict reported usage. However, perceived usefulness is 1.5 times more important than usability when predicting technology acceptance.

Measuring Usefulness

Jeff Sauro • November 1, 2011

A product must be useful and usable to be adopted and both have been shown to predict reported usage. However, perceived usefulness is 1.5 times more important than usability when predicting technology acceptance.[Read More]

Build it and they might come. Build trust and they will stay. Make it usable and credible and they may tell their friends. Measuring credibility and trust along with usability should be part of benchmarking efforts. Is your website trusted as little as Facebook or as much as PayPal and Apple?

When credibility and trust matter more than usability

Jeff Sauro • October 26, 2011

Build it and they might come. Build trust and they will stay. Make it usable and credible and they may tell their friends. Measuring credibility and trust along with usability should be part of benchmarking efforts. Is your website trusted as little as Facebook or as much as PayPal and Apple?[Read More]

Good website navigation is essential for task success. When users

Getting the first click right

Jeff Sauro • October 19, 2011

Good website navigation is essential for task success. When users' first click is down an incorrect path, only 46% eventually succeed. Here are some ideas on testing first click success.[Read More]

Recent changes in Netflix pricing, services and poor communication substantially affected key customer metrics. A longitudinal analysis of customers shows a drop of 80 percentage points in the Net Promoter Score and a drop in credibility rankings from the 99th percentile to the 62nd percentile. What

Netflix by the Numbers: Net Promoter and Credibility Scores Decline

Jeff Sauro • October 11, 2011

Recent changes in Netflix pricing, services and poor communication substantially affected key customer metrics. A longitudinal analysis of customers shows a drop of 80 percentage points in the Net Promoter Score and a drop in credibility rankings from the 99th percentile to the 62nd percentile. What's bad for Netflix is a reminder to measure early and often.[Read More]

A successful website needs to be usable, credible and visually appealing. This will generate positive word of mouth, repeat visitors and ultimately a more successful website. The SUPR-Q (Standardized Universal Percentile Rank) measures these concepts in 13 items along with a robust database of 200 websites to know where your scores ranks.

The Essential Elements of a Successful Website

Jeff Sauro • October 4, 2011

A successful website needs to be usable, credible and visually appealing. This will generate positive word of mouth, repeat visitors and ultimately a more successful website. The SUPR-Q (Standardized Universal Percentile Rank) measures these concepts in 13 items along with a robust database of 200 websites to know where your scores ranks.[Read More]

These are three of the most important words for anyone trying to make better decisions with data. For most measures of customer experience there are at least three good places to start to look for meaningful comparables: a prior version,  an industry average or a leading competitor.

Compared to What? Making Sense of Customer Experience Metrics

Jeff Sauro • September 28, 2011

These are three of the most important words for anyone trying to make better decisions with data. For most measures of customer experience there are at least three good places to start to look for meaningful comparables: a prior version, an industry average or a leading competitor.[Read More]

It

5 Benefits of Remote Usability Testing

Jeff Sauro • September 20, 2011

It's not easy finding and scheduling users to participate in a usability test. It gets a lot easier to observe even difficult to find users when the commitment is just a few minutes in front of their computer.[Read More]

While overall men tend to make around 4% more than women, the difference is largely explained by age and years of experience. Women make the same or more than men below age 35 but tend to make less for the higher age cohorts.

Are women paid less than men in UX?

Jeff Sauro • September 13, 2011

While overall men tend to make around 4% more than women, the difference is largely explained by age and years of experience. Women make the same or more than men below age 35 but tend to make less for the higher age cohorts.[Read More]

Completion rates are the fundamental usability metric. They are easy to collect and understand and should be reported with confidence intervals at every stage of development.

10 Things to Know about Completion Rates

Jeff Sauro • September 6, 2011

Completion rates are the fundamental usability metric. They are easy to collect and understand and should be reported with confidence intervals at every stage of development.[Read More]

The results of the recent UPA Salary survey show 82% of UX professionals use some form of usability testing, 75% perform expert reviews and only 16% have the budget to perform eye-tracking studies. Almost 40% use focus groups and more than two-thirds use prototyping.

The Methods UX Professionals Use

Jeff Sauro • August 30, 2011

The results of the recent UPA Salary survey show 82% of UX professionals use some form of usability testing, 75% perform expert reviews and only 16% have the budget to perform eye-tracking studies. Almost 40% use focus groups and more than two-thirds use prototyping.[Read More]

The median UX professional salary for 2011 is $90k, up around $5,500 (7%) from 2009. The key factors that drive 40% of salary variation are: years of experience, having a PhD, being a manager and whether you live in the Western or Northeastern US.

How much are you worth? 2011 Salary Data for UX Professionals

Jeff Sauro • August 23, 2011

The median UX professional salary for 2011 is $90k, up around $5,500 (7%) from 2009. The key factors that drive 40% of salary variation are: years of experience, having a PhD, being a manager and whether you live in the Western or Northeastern US.[Read More]

There isn

The four corners of usability measurement

Jeff Sauro • August 16, 2011

There isn't a single silver bullet technique or tool which will uncover all usability problems. Instead, practitioners should quadrangulate using the four corners of usability measurement: user testing, inspection methods, cognitive modeling and standardized questionnaires.[Read More]

Perhaps it

10 Things to Know about Task Times

Jeff Sauro • August 9, 2011

Perhaps it's something about the precision of minutes and seconds that demands greater scrutiny. There's a lot to consider when measuring and analyzing task times in usability tests: among them are how to report the average time, what to do with failed task-attempts, thinking aloud and figuring out how long the task should take.[Read More]

Heuristic Evaluations and Cognitive Walkthroughs are both usability Inspection Methods that can be used early and often in design to detect many of the problems seen in usability testing. Cognitive Walkthoughs have more of an emphasis on the learnability of a task. Both methods are best performed with multiple evaluators.

Whatís the difference between a Heuristic Evaluation and a Cognitive Walkthrough?

Jeff Sauro • August 2, 2011

Heuristic Evaluations and Cognitive Walkthroughs are both usability Inspection Methods that can be used early and often in design to detect many of the problems seen in usability testing. Cognitive Walkthoughs have more of an emphasis on the learnability of a task. Both methods are best performed with multiple evaluators.[Read More]

If you collect nothing else in a usability test it should be a list of problems encountered by users. It seems so simple yet there is a rich history of how many users you need to test, what constitutes a problem and which method to use.

10 Things to Know about Usability Problems

Jeff Sauro • July 26, 2011

If you collect nothing else in a usability test it should be a list of problems encountered by users. It seems so simple yet there is a rich history of how many users you need to test, what constitutes a problem and which method to use.[Read More]

A usability problem to one usability expert is a feature to another. There is a balance between business interests and user interests but the two aren

Is that a Usability Problem or a Feature?

Jeff Sauro • July 19, 2011

A usability problem to one usability expert is a feature to another. There is a balance between business interests and user interests but the two aren't mutually exclusive. The discussion should be about short-term revenue at the expense of long term profits--not usability versus revenue.[Read More]

The mean response to the likelihood to recommend question predicts the Net Promoter Score very well. Net Promoter Scoring loses about 4% of the response information. It may be more beneficial to report the Net Promoter Score to executives but use the mean for statistical comparisons.

Net Promoter Scoring: The Mean works as well as Promoters minus Detractors

Jeff Sauro • July 12, 2011

The mean response to the likelihood to recommend question predicts the Net Promoter Score very well. Net Promoter Scoring loses about 4% of the response information. It may be more beneficial to report the Net Promoter Score to executives but use the mean for statistical comparisons.[Read More]

I

14 burdens placed on the user

Jeff Sauro • July 5, 2011

I've listed 14 of the more frequent/painful burdens I experience in the hope we can shift the burden more from the human back to the computer: passwords, account numbers and confirmation dialogues get special attention.[Read More]

Measure the percent of users that find the content, the percent who suggest a category label and the percent who place a card in a group along with confidence intervals to improve website navigation.

Three easy metrics for improving website navigation

Jeff Sauro • June 28, 2011

Measure the percent of users that find the content, the percent who suggest a category label and the percent who place a card in a group along with confidence intervals to improve website navigation.[Read More]

It

Five important things to do before any redesign

Jeff Sauro • June 21, 2011

It's easy to get overwhelmed by methods, tools and technology when making design improvements. Be sure to get the critical few tasks right by defining the users, what they're trying to do and how they do it on your application.[Read More]

Using the lower-boundary of a confidence interval on a pre-test response rate will provide an accurate estimate of your full-survey

How to estimate a survey response rate

Jeff Sauro • June 15, 2011

Using the lower-boundary of a confidence interval on a pre-test response rate will provide an accurate estimate of your full-survey's response rate. You can use this to estimate how many total invites you need to maintain a margin of error in your responses.[Read More]

Many software companies track and use the Net Promoter Score as a gauge of customer loyalty. In the largest study of consumer and productivity software, over 1700 current customers provide Net Promoter Scores and Usability scores for 17 products.

Usability and Net Promoter Benchmarks for Consumer Software

Jeff Sauro • June 8, 2011

Many software companies track and use the Net Promoter Score as a gauge of customer loyalty. In the largest study of consumer and productivity software, over 1700 current customers provide Net Promoter Scores and Usability scores for 17 products.[Read More]

It

Should you care if your rating scale data is interval or ordinal?

Jeff Sauro • June 1, 2011

It's fine to compute means and statistically analyze ordinal data from rating scales--the numbers don't know where they came from. But just because one rating is twice as high as another does not mean users are really twice as satisfied.[Read More]

When we look to improve the user experience of software or websites, sometimes the best improvements aren

Four Terrific and Four Terrible User Experiences

Jeff Sauro • May 24, 2011

When we look to improve the user experience of software or websites, sometimes the best improvements aren't slight tweaks to the interface but involve eliminating steps altogether. Here are four examples of terrific and terrible experiences from the physical world to inspire the digital one.[Read More]

In measuring perceived usability of five popular websites, a single difficult task will lower post-test usability scores by 8%. This is largely driven by the least experienced users whose scores dropped by almost 20%.  A difficult task doesn

How much does the usability test affect perceptions of usability?

Jeff Sauro • May 17, 2011

In measuring perceived usability of five popular websites, a single difficult task will lower post-test usability scores by 8%. This is largely driven by the least experienced users whose scores dropped by almost 20%. A difficult task doesn't appear to affect the most experienced users' attitudes.[Read More]

Closed ended rating scale data is easy to summarize and hard to interpret. These five approaches provide meaning to raw responses and often generate similar results.

How to interpret survey responses: 5 techniques

Jeff Sauro • May 10, 2011

Closed ended rating scale data is easy to summarize and hard to interpret. These five approaches provide meaning to raw responses and often generate similar results.[Read More]

It

8 Research Based Insights for User Experience Surveys

Jeff Sauro • May 4, 2011

It's easy to get derailed when writing a survey or questionnaire when measuring the user experience. You need to worry about what to ask, who to ask and how what you're asking affects the responses. These eight insights will help make things a bit smoother.[Read More]

There is a long tradition of including items in questionnaires that are phrased both positively and negatively to minimize extreme response and acquiescent biases. An analysis of an all positively worded version of the SUS found little evidence for these biases. This suggests response bias effects are small and outweighed by the real effects of miscoding by researchers and misinterpreting by users.

Are both positive and negative items necessary in questionnaires?

Jeff Sauro • April 26, 2011

There is a long tradition of including items in questionnaires that are phrased both positively and negatively to minimize extreme response and acquiescent biases. An analysis of an all positively worded version of the SUS found little evidence for these biases. This suggests response bias effects are small and outweighed by the real effects of miscoding by researchers and misinterpreting by users.[Read More]

Research generally shows that including a neutral response will affect the distribution of responses and sometimes lead to different conclusions. However, this is less important when assessing usability as you

Survey items should include a neutral response: Agree, Disagree, Undecided?

Jeff Sauro • April 19, 2011

Research generally shows that including a neutral response will affect the distribution of responses and sometimes lead to different conclusions. However, this is less important when assessing usability as you're usually more concerned about comparisons over time or against a benchmark than the percent of users who agree to statements[Read More]

Task time is the best way to measure the efficiency of a task and it is a metric that everyone understands. Task-times will differ (often substantially) between lab tests, remote unmoderated tests and estimates from Keystroke Level Modeling. The trick is to be consistent. Use the same method in both comparisons and focus on the improvement instead of the absolute time regardless of the method.

Will the real task-time please stand up?

Jeff Sauro • April 12, 2011

Task time is the best way to measure the efficiency of a task and it is a metric that everyone understands. Task-times will differ (often substantially) between lab tests, remote unmoderated tests and estimates from Keystroke Level Modeling. The trick is to be consistent. Use the same method in both comparisons and focus on the improvement instead of the absolute time regardless of the method.[Read More]

You can use the cause and effect diagram at all stages of development to help brainstorm root causes for interaction problems and other unwanted outcomes. It becomes an excellent input for finding design solutions and turns negative thinking into positive user experiences.

Diagnosing Interaction Problems with Cause and Effect Diagrams

Jeff Sauro • April 5, 2011

You can use the cause and effect diagram at all stages of development to help brainstorm root causes for interaction problems and other unwanted outcomes. It becomes an excellent input for finding design solutions and turns negative thinking into positive user experiences.[Read More]

Every field has its set of hot-button issues and usability is no exception. Here are six topics that tend to generate passionate discussions: Quantifying Usability, Reliability of Evaluators, Certification, Unmoderated Testing Quality, Heuristic Evaluations and Sample Size.

Six controversial topics in usability

Jeff Sauro • March 28, 2011

Every field has its set of hot-button issues and usability is no exception. Here are six topics that tend to generate passionate discussions: Quantifying Usability, Reliability of Evaluators, Certification, Unmoderated Testing Quality, Heuristic Evaluations and Sample Size.[Read More]

Context matters in deciding what a good completion rate is for a task, however, knowing what other task completion rates are can be a good starting point given no other data. An analysis of almost 1200 usability tasks shows that the average task-completion rate is a 78%. Anything above a 78% is above average and can be a good starting point for goal setting. I

What is a good task-completion rate?

Jeff Sauro • March 21, 2011

Context matters in deciding what a good completion rate is for a task, however, knowing what other task completion rates are can be a good starting point given no other data. An analysis of almost 1200 usability tasks shows that the average task-completion rate is a 78%. Anything above a 78% is above average and can be a good starting point for goal setting. I'm not suggesting you use this information blindly when establishing a goal for a task-completion rate.[Read More]

Asking users to rate the difficulty/ease of a task-scenario (without even attempting the task) can predict around half the variation in ratings from users that actually attempted the task. Users tend to overestimate how difficult a task is but on average were able to predict task difficulty ratings to within 17%. The gap between expectation and experience can be a powerful predictor of an interaction problem--something recently seen on eBay.

How well can users predict task-level usability?

Jeff Sauro • March 15, 2011

Asking users to rate the difficulty/ease of a task-scenario (without even attempting the task) can predict around half the variation in ratings from users that actually attempted the task. Users tend to overestimate how difficult a task is but on average were able to predict task difficulty ratings to within 17%. The gap between expectation and experience can be a powerful predictor of an interaction problem--something recently seen on eBay.[Read More]

By quantifying design efforts and outcomes all organizations can benefit from understanding how improving the user experience can improve the bottom line. You need to measure tactical design changes, customer loyalty and some indicator of product revenue to identify key drivers of growth.

Usability as a Key Driver of Revenue

Jeff Sauro • March 8, 2011

By quantifying design efforts and outcomes all organizations can benefit from understanding how improving the user experience can improve the bottom line. You need to measure tactical design changes, customer loyalty and some indicator of product revenue to identify key drivers of growth.[Read More]

A lot of effort goes into simplifying interactions, reducing bugs and enhancing features. By providing simple quantitative measures of improvements in the user experience you have the data to both justify design efforts and get a better idea of what methods worked.

8 Ways to Show Design Changes Improved the User Experience

Jeff Sauro • March 1, 2011

A lot of effort goes into simplifying interactions, reducing bugs and enhancing features. By providing simple quantitative measures of improvements in the user experience you have the data to both justify design efforts and get a better idea of what methods worked.[Read More]

Quantifying the frequency of comments with a binomial confidence interval helps you estimate a sentiment in the total user population and prioritize findings.

How to quantify comments

Jeff Sauro • February 21, 2011

Quantifying the frequency of comments with a binomial confidence interval helps you estimate a sentiment in the total user population and prioritize findings.[Read More]

To know if design changes improved the usability of an application, you first need a baseline measure of usability from a benchmark test.  Benchmark usability tests don

10 Tips for Benchmark Usability Tests

Jeff Sauro • February 14, 2011

To know if design changes improved the usability of an application, you first need a baseline measure of usability from a benchmark test. Benchmark usability tests don't happen a lot, so to help you in your next test I've assembled a list of ten tips to help you get the most out of your effort.[Read More]

Observing customer behavior is an excellent way for discovering opportunities for product innovation. The number of customers you need to observe can be determined using the binomial probability formula and will vary depending on how common customer behaviors are and how certain you need to be.

How many customers should you observe?

Jeff Sauro • February 8, 2011

Observing customer behavior is an excellent way for discovering opportunities for product innovation. The number of customers you need to observe can be determined using the binomial probability formula and will vary depending on how common customer behaviors are and how certain you need to be.[Read More]

At only 10 items, SUS may be quick to administer and score, but data from over 5000 users and almost 500 different studies suggests that SUS is far from dirty. Its reliability and validity are as high as or higher than commercial questionnaires. Its versatility, brevity and wide-usage means that despite inevitable changes in technology, we can probably count on SUS being around for at least another 25 years.

Measuring Usability with the System Usability Scale (SUS)

Jeff Sauro • February 2, 2011

At only 10 items, SUS may be quick to administer and score, but data from over 5000 users and almost 500 different studies suggests that SUS is far from dirty. Its reliability and validity are as high as or higher than commercial questionnaires. Its versatility, brevity and wide-usage means that despite inevitable changes in technology, we can probably count on SUS being around for at least another 25 years.[Read More]

Responses to rating scale data typically donít follow a normal distribution. However, this is unlikely to affect the accuracy of statistical calculations because the distribution of error in the measurement is normally distributed.

Are Net Promoter Scores Normally Distributed?

Jeff Sauro • January 26, 2011

Responses to rating scale data typically donít follow a normal distribution. However, this is unlikely to affect the accuracy of statistical calculations because the distribution of error in the measurement is normally distributed.[Read More]

SUS scores from 1100 users from 62 websites showed that repeat users rate the website as 11% more usable than first time users. SUS scores from 800 users of 16 consumer software products showed that the most experienced users (5+ years of experience) also rate the software as 11% more usable than the least experienced users (less than 3 years of experience).

Does prior experience affect perceptions of usability?

Jeff Sauro • January 19, 2011

SUS scores from 1100 users from 62 websites showed that repeat users rate the website as 11% more usable than first time users. SUS scores from 800 users of 16 consumer software products showed that the most experienced users (5+ years of experience) also rate the software as 11% more usable than the least experienced users (less than 3 years of experience).[Read More]

At most 8% of users read license agreements on software programs. Users typically spend no more than a few seconds on the license page. Users breeze by the page probably because they really have no choice if they want to use the program and cannot negotiate terms.

Do users read license agreements?

Jeff Sauro • January 11, 2011

At most 8% of users read license agreements on software programs. Users typically spend no more than a few seconds on the license page. Users breeze by the page probably because they really have no choice if they want to use the program and cannot negotiate terms.[Read More]

Keystroke Level Modeling can predict error-free experienced user times to an accuracy of between 10-20%. KLM is ideally suited for comparing the efficiency of alternative designs early in the design stage when it

Measuring Task Times without Users

Jeff Sauro • January 4, 2011

Keystroke Level Modeling can predict error-free experienced user times to an accuracy of between 10-20%. KLM is ideally suited for comparing the efficiency of alternative designs early in the design stage when it's difficult to measure task time from user-testing.[Read More]

There were many interesting findings reported on MeasuringUsability.com in 2010. Here is my top-10 list to get you thinking for 2011.

Top 10 Research-Based Usability Findings of 2010

Jeff Sauro • December 29, 2010

There were many interesting findings reported on MeasuringUsability.com in 2010. Here is my top-10 list to get you thinking for 2011.[Read More]

Links to 25 books, papers, chapters and questionnaires on measuring usability.

25 Resources for Measuring Usability

Jeff Sauro • December 21, 2010

Links to 25 books, papers, chapters and questionnaires on measuring usability.

[Read More]

This is the working outline of our book with Morgan Kaufmann. It will bring together almost a decade of research on finding the best statistical approaches to solving the most common issues in user research. Anticipated publication date Spring 2012.

Quantifying the User Experience: Practical Statistics for User Research the Book (TOC)

Jeff Sauro • December 21, 2010

This is the working outline of our book with Morgan Kaufmann. It will bring together almost a decade of research on finding the best statistical approaches to solving the most common issues in user research. Anticipated publication date Spring 2012.[Read More]

Top-box and top-two-box scoring is appealing for summarizing responses in the absence of benchmarks or comparisons. Top-box scoring has the disadvantage of losing information about precision and variability. Reducing 5, 7 or 11 response options to two or three options can mask real changes in attitudes.

Top-Box Scoring of Rating Scale Data

Jeff Sauro • December 14, 2010

Top-box and top-two-box scoring is appealing for summarizing responses in the absence of benchmarks or comparisons. Top-box scoring has the disadvantage of losing information about precision and variability. Reducing 5, 7 or 11 response options to two or three options can mask real changes in attitudes.[Read More]

Around 10% of paid respondents to online surveys are rushing through surveys to get the honorarium. With longer surveys and more complicated questions, closer to 20% of responses will be unusable.

How many people cheat in online surveys?

Jeff Sauro • December 7, 2010

Around 10% of paid respondents to online surveys are rushing through surveys to get the honorarium. With longer surveys and more complicated questions, closer to 20% of responses will be unusable.[Read More]

Users rate tasks as more difficult when given 5 or 60 seconds compared to no time limit. These lower task-ratings don

What happens to task-ratings when you interrupt users?

Jeff Sauro • November 30, 2010

Users rate tasks as more difficult when given 5 or 60 seconds compared to no time limit. These lower task-ratings don't appear to translate to overall website ratings. Concurrent ratings may pinpoint interaction problems more precisely than post-task ratings but add time to test-sessions and task-times.[Read More]

Survey results show completion rates and UI problems dominate formative usability tests. Task time data is more common in summative usability tests, yet still prevalent in formative tests. At least half of respondents use some form of usability questionnaire in both types of tests.

What metrics are collected in usability tests?

Jeff Sauro • November 23, 2010

Survey results show completion rates and UI problems dominate formative usability tests. Task time data is more common in summative usability tests, yet still prevalent in formative tests. At least half of respondents use some form of usability questionnaire in both types of tests.[Read More]

In his book Outliers, Malcolm Gladwell identifies the years around 1955 as the best time to become a software millionaire.  Bill Gates, Steve Jobs and Eric Schmidt all have birthdays in 1955. However, an examination of a larger dataset of 41 software millionaires reveals twice as many were born outside the 1950s. CEOs from all industries happen to be born in the 1950s--a likely consequence of the baby-boom and not particular birth years.

Were most Software Millionaires born around 1955?

Jeff Sauro • November 17, 2010

In his book Outliers, Malcolm Gladwell identifies the years around 1955 as the best time to become a software millionaire. Bill Gates, Steve Jobs and Eric Schmidt all have birthdays in 1955. However, an examination of a larger dataset of 41 software millionaires reveals twice as many were born outside the 1950s. CEOs from all industries happen to be born in the 1950s--a likely consequence of the baby-boom and not particular birth years.[Read More]

System Usability Scale (SUS) Scores from users who had only 5 seconds to assess the usability of a website were statistically indistinguishable from users who had no time limit. Users who had only 60 seconds on a website tend to rate websites as more usable than those who had only 5 seconds or no time limit.

5 Second Usability Tests

Jeff Sauro • November 9, 2010

System Usability Scale (SUS) Scores from users who had only 5 seconds to assess the usability of a website were statistically indistinguishable from users who had no time limit. Users who had only 60 seconds on a website tend to rate websites as more usable than those who had only 5 seconds or no time limit.[Read More]

The results of an email survey found 80% of Formative usability tests have less than 15 users. Summative usability test sample sizes are around 3 times larger for respondents who conducted both types of tests.

How many users do people actually test?

Jeff Sauro • November 2, 2010

The results of an email survey found 80% of Formative usability tests have less than 15 users. Summative usability test sample sizes are around 3 times larger for respondents who conducted both types of tests.[Read More]

In a usability test we

What really happens in the usability lab?

Jeff Sauro • October 27, 2010

In a usability test we're taught to be neutral observers of user behavior. But what actually happens in practice? A review of 14 usability testing sessions from seven companies suggests that what we preach is a bit different than what we practice.[Read More]

UX and marketing can compete for the same limited resources in an organization and at times can be at odds on product direction. In my experience they are an unlikely couple and have a lot in common.

9 things UX and Marketing have in common

Jeff Sauro • October 19, 2010

UX and marketing can compete for the same limited resources in an organization and at times can be at odds on product direction. In my experience they are an unlikely couple and have a lot in common.[Read More]

Three books on usability testing from the early 90

5 Classic Usability Books

Jeff Sauro • October 12, 2010

Three books on usability testing from the early 90's, one on usability ROI and another on estimating experienced user task times without testing any users.[Read More]

When users document the usability problems they encounter during a usability test, they find around half as many problems as trained professionals watching users in a lab. This low-cost method is a good supplement to heuristic evaluations and user testing.

Can users self-report usability problems?

Jeff Sauro • October 6, 2010

When users document the usability problems they encounter during a usability test, they find around half as many problems as trained professionals watching users in a lab. This low-cost method is a good supplement to heuristic evaluations and user testing.[Read More]

Usability problem frequencies from 24 usability tests show that users are almost ten-times more likely to encounter a usability problem in a business application than a website. Users are about half as likely to encounter a problem in consumer software than a business application.

How common are usability problems?

Jeff Sauro • September 29, 2010

Usability problem frequencies from 24 usability tests show that users are almost ten-times more likely to encounter a usability problem in a business application than a website. Users are about half as likely to encounter a problem in consumer software than a business application.[Read More]

Items in the System Usability Scale (SUS) were rephrased to either all extreme positive or all extreme negative wording and average scores were compared. Users disagree more with items that are worded in the extreme and resulted in significantly different SUS Scores.

That's the worst website ever!: Effects of extreme survey items

Jeff Sauro • September 21, 2010

Items in the System Usability Scale (SUS) were rephrased to either all extreme positive or all extreme negative wording and average scores were compared. Users disagree more with items that are worded in the extreme and resulted in significantly different SUS Scores.[Read More]

There is a modest bias towards the left side of a rating scale. This matters most for stand-alone surveys or questionnaires when no comparisons are being made. In such cases, the number of "agree" or top-box statements will be higher if placed on the left-side of a scale.

Survey Respondents Prefer the Left Side of a Rating Scale

Jeff Sauro • September 14, 2010

There is a modest bias towards the left side of a rating scale. This matters most for stand-alone surveys or questionnaires when no comparisons are being made. In such cases, the number of "agree" or top-box statements will be higher if placed on the left-side of a scale.[Read More]

Users, surveys, tasks and metrics: there

97 Things to Know about Usability

Jeff Sauro • September 7, 2010

Users, surveys, tasks and metrics: there's a lot to know about usability but here's a resource for experts and novices.

[Read More]

Maybe you already have heard-of and use Heuristics Evaluations. Here are six things you might NOT know about this popular usability method.

6 things you didn't know about Heuristic Evaluations

Jeff Sauro • August 31, 2010

Maybe you already have heard-of and use Heuristics Evaluations. Here are six things you might NOT know about this popular usability method.[Read More]

7 point scales tend to perform slightly better than 5 point scales. The benefit is too small to change your existing questionnaires if you have historical data. Having more points will provide the biggest benefit when you have only a few or one question in your questionnaire.  Focus more on what you

Should you use 5 or 7 point scales?

Jeff Sauro • August 25, 2010

7 point scales tend to perform slightly better than 5 point scales. The benefit is too small to change your existing questionnaires if you have historical data. Having more points will provide the biggest benefit when you have only a few or one question in your questionnaire. Focus more on what you'll do with the results than whether 5 or 7 points is better.[Read More]

Different usability teams find different problems in websites and applications. Would you get different diagnoses if different radiologists read your x-ray or MRI? It turns out you would. While there is a lot of room for improvement in the reliability of usability evaluations much of the variability is due to human judgment--a problem that also plagues the medical field.

Usability Evaluators: Reliable as Radiologists?

Jeff Sauro • August 18, 2010

Different usability teams find different problems in websites and applications. Would you get different diagnoses if different radiologists read your x-ray or MRI? It turns out you would. While there is a lot of room for improvement in the reliability of usability evaluations much of the variability is due to human judgment--a problem that also plagues the medical field.[Read More]

Love them, hate them, admire them or ignore them. These seven living legends aren

7 Living Legends of Usability

Jeff Sauro • August 10, 2010

Love them, hate them, admire them or ignore them. These seven living legends aren't one-hit wonders. Their work has had and will continue to have a large impact on the field of usability for some time.[Read More]

Confidence intervals, like statistics in general, are powerful because they are both consistent with our experience and provide a level of precision we can

Memory versus Math in Usability Tests

Jeff Sauro • August 4, 2010

Confidence intervals, like statistics in general, are powerful because they are both consistent with our experience and provide a level of precision we can't articulate. You should use them with your usability test data.[Read More]

A study conducted by Nielsen on reading speeds was criticized for going beyond the statistics to support the claim that books are faster than tablets. The crux of this point comes down to considering a finding "statistically significant" only when the p-value is below .05. This criterion is a convention not a commandment and context should always be considered when deciding the role of chance in applied research.

Books Faster than TabletsÖor not?

Jeff Sauro • July 27, 2010

A study conducted by Nielsen on reading speeds was criticized for going beyond the statistics to support the claim that books are faster than tablets. The crux of this point comes down to considering a finding "statistically significant" only when the p-value is below .05. This criterion is a convention not a commandment and context should always be considered when deciding the role of chance in applied research.[Read More]

Wondering about the origins of the sample size controversy in the usability profession?  Here is an annotated timeline of the major events and papers which continue to shape this topic from 1982-2010.

A Brief History of the Magic Number 5 in Usability Testing

Jeff Sauro • July 21, 2010

Wondering about the origins of the sample size controversy in the usability profession? Here is an annotated timeline of the major events and papers which continue to shape this topic from 1982-2010.[Read More]

This common question mixes two concepts: representativeness and sample size. It is more important to ask a few of the right people what they think than a lot of the wrong people. Once you

What is a Representative Sample Size for a Survey?

Jeff Sauro • July 15, 2010

This common question mixes two concepts: representativeness and sample size. It is more important to ask a few of the right people what they think than a lot of the wrong people. Once you're talking to the right people identify the highest margin of error you can tolerate to compute the right sample size.[Read More]

These five papers have had a large and lasting influence on the field of Usability and User Experience.

The Five Most Influential Papers in Usability

Jeff Sauro • July 7, 2010

These five papers have had a large and lasting influence on the field of Usability and User Experience.

[Read More]

It is tempting to just find, fix and forget usability problems. It is tempting to skip one milkshake and think you

Want to Improve Usability? Count Calories

Jeff Sauro • June 30, 2010

It is tempting to just find, fix and forget usability problems. It is tempting to skip one milkshake and think you're losing weight. Systematic measuring uncovers patterns and helps prevent major problems.[Read More]

Whether you

Five ways to make any usability test more credible

Jeff Sauro • June 23, 2010

Whether you're conducting an early stage test of a prototype or late validation, these five tips can make any usability test more credible. The tips both temper skepticism about small samples and help you avoid overstating your findings.[Read More]

Web analytics has transformed the problem of understanding user behavior from a puzzle to a mystery. Where we once didnít have enough information, we now can have too much to make sense of. Small sample user testing tells helps answer the "why" mystery. There will be a continued demand for user-researchers who can quantify observational data and make the most of analytic data.

What five users can tell you that 5000 cannot

Jeff Sauro • June 16, 2010

Web analytics has transformed the problem of understanding user behavior from a puzzle to a mystery. Where we once didnít have enough information, we now can have too much to make sense of. Small sample user testing tells helps answer the "why" mystery. There will be a continued demand for user-researchers who can quantify observational data and make the most of analytic data.[Read More]

Usability is not adhering to guidelines. Usability is measuring whether users are actually having a more usable experience. Users might want emotion-evoking software, but if they do there should be some evidence for it.

Is Usability a Science?

Jeff Sauro • June 10, 2010

Usability is not adhering to guidelines. Usability is measuring whether users are actually having a more usable experience. Users might want emotion-evoking software, but if they do there should be some evidence for it.[Read More]

Is the term UX becoming an overused dot-com term like B2C or clicks-to-mortar? The field of User Experience has matured enough now that it deserves its own BS generator replete with terms that have more use than meaning--as well as a PDF Report with an ROI graph.

User Experience BS Generator

Jeff Sauro • June 3, 2010

Is the term UX becoming an overused dot-com term like B2C or clicks-to-mortar? The field of User Experience has matured enough now that it deserves its own BS generator replete with terms that have more use than meaning--as well as a PDF Report with an ROI graph.[Read More]

Mechanical Turk is seeing a shifting of its demographics from stay-at-home moms in the US to younger males in India. Around 60% of the Mechanical Turk workforce will put forth a conscientious effort in completing HITs while around 12% are likely rushing through tasks indiscriminately.

The Faces in the Crowdsourcing

Jeff Sauro • May 26, 2010

Mechanical Turk is seeing a shifting of its demographics from stay-at-home moms in the US to younger males in India. Around 60% of the Mechanical Turk workforce will put forth a conscientious effort in completing HITs while around 12% are likely rushing through tasks indiscriminately.[Read More]

It takes about 2 to 3 times as long to send an email on the iPhone as on the Desktop. However, users were able to find percentages faster on the iPhone calculator, showing you can have both portability and efficiency for certain tasks.

iPhone vs. Desktop: Which is Faster?

Jeff Sauro • May 19, 2010

It takes about 2 to 3 times as long to send an email on the iPhone as on the Desktop. However, users were able to find percentages faster on the iPhone calculator, showing you can have both portability and efficiency for certain tasks.[Read More]

You don

Do you need a random sample for your usability test?

Jeff Sauro • May 12, 2010

You don't need a random sample to use statistics to make better decisions from your usability data. You do need to know if the users who aren't in your usability tests are different enough than those who are.[Read More]

The sample size formula for finding usability problems only works for a specific set of users and closed-ended tasks. With five users you will only find the more obvious problems.

Will five users really find 85% of all usability problems?

Jeff Sauro • May 6, 2010

The sample size formula for finding usability problems only works for a specific set of users and closed-ended tasks. With five users you will only find the more obvious problems.[Read More]

For most cases exclude the users who fail the task and call it the average task completion time. Keep the failed task times to report Mean Time to Failure and Average Time on Task.

What to do with task times when users fail a task

Jeff Sauro • April 29, 2010

For most cases exclude the users who fail the task and call it the average task completion time. Keep the failed task times to report Mean Time to Failure and Average Time on Task.[Read More]

For small sample sizes, the geometric mean provides a better estimate of the middle task time that the sample median or mean.

Average Task Times in Usability Tests: What to Report?

Jeff Sauro • April 21, 2010

For small sample sizes, the geometric mean provides a better estimate of the middle task time that the sample median or mean.

[Read More]

QA testers are not adequate substitutes for real users and usability tests are not adequate substitutes for good QA.

Usability testing is not QA testing

Jeff Sauro • April 7, 2010

QA testers are not adequate substitutes for real users and usability tests are not adequate substitutes for good QA.

[Read More]

Retrospective probing of user actions and intentions allows you to get a reliable benchmark and identify problems with an interface.

Can you measure task time if users think aloud during a usability test?

Jeff Sauro • April 1, 2010

Retrospective probing of user actions and intentions allows you to get a reliable benchmark and identify problems with an interface.[Read More]

While testing with five users might reveal 85% of problems that impact 31% of users (given a set of tasks and user-type), it doesn

Do severe problems affect more users than trivial ones?

Jeff Sauro • March 25, 2010

While testing with five users might reveal 85% of problems that impact 31% of users (given a set of tasks and user-type), it doesn't mean you're finding 85% of the critical problems. Assume that the severity of a problem is not related to how often it occurs.[Read More]

One consequence of analyzing user data is having to reconcile conflicting data-points. How would you display links to PDF files on a web-page?

How should you display links to PDF files?

Jeff Sauro • March 18, 2010

One consequence of analyzing user data is having to reconcile conflicting data-points. How would you display links to PDF files on a web-page?[Read More]

For finding usability problems with an interface, testing with five users is fine to find problems that affect 31% to 100% of all users. If a problem is more elusive (affects fewer than 31% of users) then you need to increase your sample size.  This sample size does not apply to comparing designs or generating a precise estimate of completion rates or task-times.

Why you only need to test with five users (explained)

Jeff Sauro • March 8, 2010

For finding usability problems with an interface, testing with five users is fine to find problems that affect 31% to 100% of all users. If a problem is more elusive (affects fewer than 31% of users) then you need to increase your sample size. This sample size does not apply to comparing designs or generating a precise estimate of completion rates or task-times.[Read More]

A single 7-point likert-type question asked after a task-scenario provides a quick but sensitive and reliable estimate of task level difficulty and ease.

If you could only ask one question, use this one.

Jeff Sauro • March 2, 2010

A single 7-point likert-type question asked after a task-scenario provides a quick but sensitive and reliable estimate of task level difficulty and ease.[Read More]

Post-task ratings capture satisfaction with task-performance and are great for identifying problem areas in an interface. Conversely, post-test questionnaires provide overall attitudes about the application and don

Performance Satisfaction and Perception Satisfaction

Jeff Sauro • February 24, 2010

Post-task ratings capture satisfaction with task-performance and are great for identifying problem areas in an interface. Conversely, post-test questionnaires provide overall attitudes about the application and don't provide much diagnostic information.[Read More]

At some point usability practitioners began using the terms quantitative and Summative interchangeably. That

Are the Terms Formative and Summative Helpful or Harmful?

Jeff Sauro • February 16, 2010

At some point usability practitioners began using the terms quantitative and Summative interchangeably. That's a bad thing as metrics and quantitative methods should be used when finding and fixing UI problems as well as establishing a usability benchmark.[Read More]

In a comparative test, satisfaction scores and completion rates from professional usability test-takers were nearly identical to lab-based users. However, time on task data differed significantly and showed much higher variability. For testing websites intended for a general audience the use of professional testers appears to provide mostly reliable data quickly and for a fraction of the price.

Can we trust data from professional usability test-takers?

Jeff Sauro • February 9, 2010

In a comparative test, satisfaction scores and completion rates from professional usability test-takers were nearly identical to lab-based users. However, time on task data differed significantly and showed much higher variability. For testing websites intended for a general audience the use of professional testers appears to provide mostly reliable data quickly and for a fraction of the price.[Read More]

Insurance companies do it, drug companies do it and so should usability testers. When you observe a problem from a small sample test, it is unlikely the problem only affects a tiny percentage of users.

If 1 of 5 users has a problem in a usability test will it impact 1% or 20% of all users?

Jeff Sauro • February 1, 2010

Insurance companies do it, drug companies do it and so should usability testers. When you observe a problem from a small sample test, it is unlikely the problem only affects a tiny percentage of users.[Read More]

Performance data such as task times and completion rates explain around 26% of the user

Do performance data and satisfaction data measure the same thing?

Jeff Sauro • January 25, 2010

Performance data such as task times and completion rates explain around 26% of the user's perception of the ease of use. Gather both performance data and satisfaction data to triangulate around the task-level user experience.[Read More]

The System Usability Scale (SUS) is the most popular standardized usability questionnaire because it

Can you use the SUS for websites?

Jeff Sauro • January 18, 2010

The System Usability Scale (SUS) is the most popular standardized usability questionnaire because it's free and short. It was designed over 20 years ago before the web existed. Should it be used on websites?[Read More]

I examined the relationship between customer loyalty as measured by the Net Promoter Score (NPS) and System Usability Scale (SUS) questionnaire from several usability tests. I found that perceptions of usability account for about 1/3 of the changes in customer loyalty. Increasing your usability will lead to increased loyalty.

Does better usability increase customer loyalty?

Jeff Sauro • January 7, 2010

I examined the relationship between customer loyalty as measured by the Net Promoter Score (NPS) and System Usability Scale (SUS) questionnaire from several usability tests. I found that perceptions of usability account for about 1/3 of the changes in customer loyalty. Increasing your usability will lead to increased loyalty.[Read More]

Is there such thing as usability? This might sound like a silly question considering the industry around usability testing and user experience consulting (not to mention this website). But you can

Does usability exist?

Jeff Sauro • January 4, 2010

Is there such thing as usability? This might sound like a silly question considering the industry around usability testing and user experience consulting (not to mention this website). But you can't touch usability and there is no usability thermometer to measure its presence or absence. While we can talk about usability and know it when we see it (or really, know it when we don't see it), what data is there that shows usability exists?[Read More]

Imagine a marketing department asking for more money to conduct a direct-mail campaign and their only justification was that marketing is a critical business advantage.

What is Quantitative Usability?

Jeff Sauro • December 17, 2009

Imagine a marketing department asking for more money to conduct a direct-mail campaign and their only justification was that marketing is a critical business advantage.[Read More]

Is it possible to get the same data from lab-based tests by having users test themselves?  Unmoderated testing appears to provide a cost effective alternative for gathering a lot more usability data with considerably less effort. Additional time is required to filter invalid data such as unrealistically short task times.

Is there a difference in usability data from remote unmoderated tests and lab-based tests?

Jeff Sauro • December 8, 2009

Is it possible to get the same data from lab-based tests by having users test themselves? Unmoderated testing appears to provide a cost effective alternative for gathering a lot more usability data with considerably less effort. Additional time is required to filter invalid data such as unrealistically short task times.[Read More]

Does a full-time PhD pay off in the Usability Profession?  The UPA Salary data from 2009 and 2005 is analyzed. It shows that while a PhD may open doors, being out of the work-force for five years is an opportunity cost that is unlikely to made up for.

How much is a PhD Worth?

Jeff Sauro • November 5, 2009

Does a full-time PhD pay off in the Usability Profession? The UPA Salary data from 2009 and 2005 is analyzed. It shows that while a PhD may open doors, being out of the work-force for five years is an opportunity cost that is unlikely to made up for.[Read More]

Only 14% of users who fail a task rate it at maximum level of satisfaction. In general there is an 80/20 rule of satisfaction and completion rates: 80% of users who rate at the maximum level of satisfaction will pass and 80% of users who rate at the minimum satisfaction level will fail the task.

Do users fail a task and still rate it as easy?

Jeff Sauro • October 9, 2009

Only 14% of users who fail a task rate it at maximum level of satisfaction. In general there is an 80/20 rule of satisfaction and completion rates: 80% of users who rate at the maximum level of satisfaction will pass and 80% of users who rate at the minimum satisfaction level will fail the task.[Read More]

How many users will complete the task and how long will it take them?  If you need to benchmark an interface, then a summative usability test is one way to answer these questions. Summative tests are the gold-standard for usability measurement. But just how precise are the metrics?

Margins of Error in Usability Tests

Jeff Sauro • August 6, 2009

How many users will complete the task and how long will it take them? If you need to benchmark an interface, then a summative usability test is one way to answer these questions. Summative tests are the gold-standard for usability measurement. But just how precise are the metrics?[Read More]

Used for comparing 2 small sample binary completion rates (it uses a statistical test called the Fisher Exact Test)

Compare 2 Small Sample Completion Rates (Fisher Exact Test)

Jeff Sauro • June 5, 2009

Used for comparing 2 small sample binary completion rates (it uses a statistical test called the Fisher Exact Test)

[Read More]

UPA 2008 Presentation References

Jeff Sauro • June 17, 2008



[Read More]

Time-on-task can be used as a valuable diagnosis and comparative tool during formative evaluations.

Task Times in Formative Usability Tests

Jeff Sauro • June 6, 2008

Time-on-task can be used as a valuable diagnosis and comparative tool during formative evaluations.

[Read More]

One Sample Proportion Calculator

Jeff Sauro • May 30, 2008



[Read More]

Use this interactive calculator to understand how the sample size changes will affect the confidence interval around a completion rate.

Sample Size Calculator for a Completion Rate

Jeff Sauro • January 4, 2008

Use this interactive calculator to understand how the sample size changes will affect the confidence interval around a completion rate.[Read More]

Use this interactive normal curve to understand how the z-score and the area of the curve are related. Offers both one and two-sided options.

Interactive Graph of the Standard Normal Curve

Jeff Sauro • December 14, 2007

Use this interactive normal curve to understand how the z-score and the area of the curve are related. Offers both one and two-sided options.[Read More]

Enter the area under the normal curve (a proportion between 0 & 1) and get the Z-critical value.

Percentile to Z-Score Calculator

Jeff Sauro • December 4, 2007

Enter the area under the normal curve (a proportion between 0 & 1) and get the Z-critical value.

[Read More]

Look up the area under the normal curve (1 or two-sided areas) from a standard score (Z-score).

Z-Score to Percentile Calculator

Jeff Sauro • December 3, 2007

Look up the area under the normal curve (1 or two-sided areas) from a standard score (Z-score).

[Read More]

The UsabilityScorecard web-application will take raw usability metrics (completion, time, sat, errors and clicks) and calculate confidence intervals, z-scores, quality levels and graph the results automatically. You can also combine any combination of the metrics into a 2, 3 or 4 measure combined score. Data can be imported from Excel (.csv) and exported to Word(.rtf).

Usability Scorecard

Jeff Sauro • June 1, 2007

The UsabilityScorecard web-application will take raw usability metrics (completion, time, sat, errors and clicks) and calculate confidence intervals, z-scores, quality levels and graph the results automatically. You can also combine any combination of the metrics into a 2, 3 or 4 measure combined score. Data can be imported from Excel (.csv) and exported to Word(.rtf).[Read More]

Use this calculator to determine the number of users you

Sample Size Calculator for Discovering Problems in a User Interface

Jeff Sauro • October 1, 2006

Use this calculator to determine the number of users you'd need to test given the probability of detecting a problem. If the probability of detecting the problem is unknown, this calculator also allows you to estimate the problem occurrence (p) from sample data.[Read More]

Visualizing your task time data is an essential step in understanding its distribution and computing accurate confidence intervals. This calculator creates a dot-plot of your task times, transforms the raw data to adjust for non-normality and computes the intervals.

Graph and Calculator for Confidence Intervals for Task Times

Jeff Sauro • February 6, 2006

Visualizing your task time data is an essential step in understanding its distribution and computing accurate confidence intervals. This calculator creates a dot-plot of your task times, transforms the raw data to adjust for non-normality and computes the intervals.[Read More]

If you

Confidence Interval Calculator for a Completion Rate

Jeff Sauro • October 1, 2005

If you've wanted to provide a confidence interval around a small sample completion rate but just didn't have time to do the math, this calculator does the work for you.[Read More]

Do you need to feel more confident about using statistics? Dismayed by overly complicated "introduction" courses that focus on theory and not application? Do the "basic" books assume you know where to look for your answer? The first module in this series is on using confidence intervals in usability testing.

Usable Statistics

Jeff Sauro • June 14, 2005

Do you need to feel more confident about using statistics? Dismayed by overly complicated "introduction" courses that focus on theory and not application? Do the "basic" books assume you know where to look for your answer? The first module in this series is on using confidence intervals in usability testing.[Read More]

This paper identifies the limitations of traditional usability metrics and presents a process to increase their meaning by adapting Six Sigma methods. We define how common usability metrics can be evaluated in terms of a standardized defective rate or quality level and explore the benefits of this data transformation. Use the <a href="scorecard/">Usability Scorcard</a> or the excel-based <a href="SUM/index.htm">SUM calculator</a> to standardize your metrics.

Making Sense of Usability Metrics: Usability and Six Sigma

Jeff Sauro • June 11, 2005

This paper identifies the limitations of traditional usability metrics and presents a process to increase their meaning by adapting Six Sigma methods. We define how common usability metrics can be evaluated in terms of a standardized defective rate or quality level and explore the benefits of this data transformation. Use the Usability Scorcard or the excel-based SUM calculator to standardize your metrics.[Read More]

SUM is a single usability metric that summarize the majority of variation in four common summative usability metrics. Download the calculator to convert raw metrics to a SUM score or read the <a href=CHI paper which explains the theoretical foundations.' title='SUM is a single usability metric that summarize the majority of variation in four common summative usability metrics. Download the calculator to convert raw metrics to a SUM score or read the CHI paper which explains the theoretical foundations.' />

SUM: Single Usability Metric

Jeff Sauro • April 17, 2005

SUM is a single usability metric that summarize the majority of variation in four common summative usability metrics. Download the calculator to convert raw metrics to a SUM score or read the CHI paper which explains the theoretical foundations.[Read More]

You can use measures such as confidence intervals, sample size calculations—and other statistics normally associated with more premium usability methods—without the high costs. These methods 
require no money to compute yet provide a wealth of information. Even better, you can still provide these quantitative qualifiers while using most discount methods. <a href="papers/Whiteboard_final.pdf">Pre-Published PDF Version</a>

Premium Usability: Getting the Discount without Paying the Price

Jeff Sauro • December 1, 2004

You can use measures such as confidence intervals, sample size calculations—and other statistics normally associated with more premium usability methods—without the high costs. These methods require no money to compute yet provide a wealth of information. Even better, you can still provide these quantitative qualifiers while using most discount methods. Pre-Published PDF Version[Read More]

Adding confidence intervals to completion rates in usability tests will temper both excessive skepticism and overstated usability findings. Confidence intervals make testing more efficient by quickly revealing unusable tasks with very small samples.

Restoring Confidence in Usability Results

Jeff Sauro • October 18, 2004

Adding confidence intervals to completion rates in usability tests will temper both excessive skepticism and overstated usability findings. Confidence intervals make testing more efficient by quickly revealing unusable tasks with very small samples.[Read More]

Conventional six sigma commonly adds a 1.5 sigma buffer to account for the shifting of a process over time. Does this make sense for software usability?

What's the 1.5σ Shift and Does it Apply to Software Usability?

Jeff Sauro • September 17, 2004

Conventional six sigma commonly adds a 1.5 sigma buffer to account for the shifting of a process over time. Does this make sense for software usability?[Read More]

The order in which a task is administered during a usability test can have an effect on the user

The Importance of Task Order Randomizing during a Usability Test

Jeff Sauro • September 17, 2004

The order in which a task is administered during a usability test can have an effect on the user's performance especially as measured by task time. By randomizing task order the effects of this lurking variable can be mitigated.[Read More]

Is attaining Six Sigma a reasonable or even attainable goal for usability? A product

What is an Acceptable Level of Quality for Usability?

Jeff Sauro • September 17, 2004

Is attaining Six Sigma a reasonable or even attainable goal for usability? A product's usability is the sum of several usability measures. Each Relative movement in your sigma value is good predictor of usability improvements.[Read More]

Many popular usability testing techniques are the right method to gather user data, however, their results alone will only scratch the surface of the true state of usability. Often their results can be misleading.

Current Usability Solutions are Unpredictable

Jeff Sauro • September 17, 2004

Many popular usability testing techniques are the right method to gather user data, however, their results alone will only scratch the surface of the true state of usability. Often their results can be misleading.[Read More]

The complexity and depth of this popular quantitative measurement is often only given cursory thought. There

Measuring & Analyzing Task Times

Jeff Sauro • September 17, 2004

The complexity and depth of this popular quantitative measurement is often only given cursory thought. There's much more to task times than using a stopwatch.[Read More]

Often the most reported measures of usability is task success. How does task success translate into a quality sigma value that can be compared to other reported sigma values?

Calculating a Sigma Level from Task Success

Jeff Sauro • September 17, 2004

Often the most reported measures of usability is task success. How does task success translate into a quality sigma value that can be compared to other reported sigma values?[Read More]

I

Relevant Publications for Measuring Usability

Jeff Sauro • September 17, 2004

I've begun to collect a list of articles and publications that relate to the quantitative measures of usability.

[Read More]

This common statistical way of describing data can be used in usability testing to standardize disparate data types to allow easy comparison between products or versions and providing a universal way of assessing quality.

What's a Z-Score and Why Use it in Usability Testing?

Jeff Sauro • September 17, 2004

This common statistical way of describing data can be used in usability testing to standardize disparate data types to allow easy comparison between products or versions and providing a universal way of assessing quality.[Read More]

One of the most common concerns about the six sigma methodology is that it cannot apply to something as byzantine as the interactions of humans with software. One of the major tenets of both Six Sigma and Human Factors is that the customer or user determine what

Why 6σ is Not Limited to Manufacturing Processes

Jeff Sauro • September 17, 2004

One of the most common concerns about the six sigma methodology is that it cannot apply to something as byzantine as the interactions of humans with software. One of the major tenets of both Six Sigma and Human Factors is that the customer or user determine what's considered "quality."[Read More]

The basics of z-scores are discussed plus an example of raw usability data converted into z-scores including three of Nielsen

How Do You Calculate a Z-Score/ Sigma Level?

Jeff Sauro • June 14, 2004

The basics of z-scores are discussed plus an example of raw usability data converted into z-scores including three of Nielsen's five usability attributes.[Read More]

Shows the history and computation of deriving a sample size for discovering problems in an interface.

Deriving a Problem Discovery Sample Size

Jeff Sauro • March 8, 2004

Shows the history and computation of deriving a sample size for discovering problems in an interface.

[Read More]

The discerning usability analyst should employ a mix of both qualitative and quantitative methods when discovering usability problems. The risks of relying heavily on a qualitative approach can lead to a severe misdiagnosis especially when usability problems are difficult to detect

The Risks of Discounted Qualitative Studies:

Jeff Sauro • March 8, 2004

The discerning usability analyst should employ a mix of both qualitative and quantitative methods when discovering usability problems. The risks of relying heavily on a qualitative approach can lead to a severe misdiagnosis especially when usability problems are difficult to detect[Read More]

Newsletter Sign Up

Receive bi-weekly updates.
[3795 Subscribers]

Connect With Us

Our Supporters

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Usertesting.com

Loop11 Online Usabilty Testing

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.