What is an Acceptable Level of Quality for Usability?

Jeff Sauro • September 17, 2004

Usability measurements involve human performance and because human behavior is inherently error prone, reaching the goal of 6σ isn't necessary to proclaim success. Manufacturing companies that are considered producing "high-quality" products are usually somewhere between 4σ and 5σ. The benchmarks and targets that we set in our tests will necessarily need to be more forgiving than in manufacturing. I prefer focusing on the movement of the sigma from one version to the next. Remember, going from 2σ to 4σ is an improvement of 50 times! I also consider the usability of a product to be the sum of a variety of measures of usability---more than just task completion. You can take any list of usability metrics and then derive a sigma value for each. Some measures reveal often overlooked but crucial aspects of what users truly perceive when using a product. The following are some measures and their target benchmarks found to be indicative of a product's usability: task completion, task time number of critical errors.

Task Completion Sigma

By far the most commonly measured and reported usability metric is task-completion. Unfortunately it is often one of the only metrics analyzed. A common benchmark for the percent of users that should complete a task is 90% or 90 out of 100 users that attempt a task should be able to complete it. There are at least two problems with this benchmark:

1. For "experienced" users--users that complete the task at least on a weekly or monthly basis, 90% is much too low.
2. Task completion by itself tends to only be a good indicator of a really unusable product. That is, if novice users are only able to complete 50% or 60% of the tasks, is it really necessary to investigate further?

The Bottom Line

In a sense, task completion then is a good preliminary test for detecting egregious usability problems or for first time or novice users. I'd continue to use 90% as goal for novice(never or rarely completed the task) and expect 99% for experienced(complete the task weekly) users.

Rate this Blog

Avg. Rating 66.33 (7)

Poor										Excellent
0	1	2	3	4	5	6	7	8	9	10

.

Posted Comments

There are 1 Comments

August 13, 2008 | Seshadri wrote:

we know the formula for calculating statistical control limits for 3sigma is X+(2.66*MR). Can you give me a similar formula for calculating 2sigma and 4 sigma conrol limits? 

---

Post a Comment

Comment:

Your Name:

Your Email Address:

.

To prevent comment spam, please answer the following :
What is 5 + 3: (enter the number)