Calculating a Sigma Level from Task Success

Jeff Sauro • September 17, 2004

Often the most reported measures of usability is task success. Success rates can be converted into a sigma value by using the discrete-binary defect calculation:

Proportion Unsuccessful= Defects/Opportunities

Where opportunities are the total number of tasks and defects are the total number of unsuccessful tasks. This calculation provides a proportion that is equivalent to a success or failure rate. For example, if 143 total tasks were attempted and 123 were successful(20 defects 143 opportunities) the equation would look like:

20/143 = .1399 By expressing .1399 as a percentage you get an unsuccessful rate of 13.9%. By subtracting the unsuccessful proportion from 1 you'll get approximately .86 or expressed in a percentage you could say that the task completion rate was 86%. Converting the proportion to a sigma value is a little more tricky.

Converting a Proportion to a Sigma Value

A sigma value is a description of how far a sample or point of data is away from its mean, expressed in standard deviations usually with the Greek letter σ or lower case s. A data point with a higher sigma value will have a higher standard deviation, meaning it is further away from the mean.

Standard deviations and sigma values assume your data is normally distributed from a continuous set of data. A normal curve has a mean of 0 and a standard deviation of 1. Success rates are Discrete-Binary (either a task was completed or it wasn't) not continuous(like time, weight or temperature) which means that binary data is not normally distributed.

Have no fear, for every statistical problem a statistical solution usually exits. We can treat binary data like continuous data because of the Central Limit Theorem. It states that as a binary sample gets larger, its distribution approximates a continuous distribution.

Approximating the Proportion to a Normal Distribution

An easy way to exploit the Central Limit Theorem solution and obtain a sigma value from the task proportion is by using the Excel function NORMSINV. NORMSINV approximates the area under a normal bell-shaped curve. Take the unsuccessful proportion .1399--the proportion defective and insert it into the following equation in Excel.

=NORMSINV(1-.1399)

 

 

 

Note: My sigma values DO NOT contain a 1.5σ adjustment or "shift."

Task Completion Benchmark

A common benchmark for the percent of users that should complete a task is 90% or 90 out of 100 users that attempt a task should be able to complete it. There are at least two problems with this benchmark:

1. For "experienced" users--users that complete the task at least on a weekly or monthly basis, 90% is much too low by definition. If the users complete the task frequently, why would we expect 10% of the tasks not to be completed?
2. Task completion by itself tends to only be a good indicator of usability for novice users or an easy measure of a really unusable product for all users. That is, if novice users are only able to complete 50% or 60% of the tasks then you already have a compelling reasons to conclude the product is unusable(for novice users) and don't need to spend as much time looking deeper for less obvious problems.

The Bottom Line

In a sense, task completion then is a good preliminary test for detecting egregious usability problems or for first time or novice users. I'd continue to use 90% as a goal for novice(never or rarely completed the task) and use 99%+ for experienced(complete the task weekly) users.

The next obvious question is: What is an acceptable sigma level for task completion? Is 6σ attainable?

Rate this Blog

Avg. Rating 51.05 (46)

Poor										Excellent
0	1	2	3	4	5	6	7	8	9	10

.

Posted Comments

There are 8 Comments

December 7, 2011 | Muhammad Nazar wrote:

This is good way to share six sigma methodology to belt participents 

---

July 16, 2010 | PP wrote:

It was really helpfull as i was not clear about the sigma level concept, thanks! 

---

May 20, 2010 | Judy wrote:

I need values that are above and below a mean based upon a target and upper and lower spec limits oon both side of the cureve.
Do you have an example for this? 

---

April 8, 2010 | Atul wrote:

good to implement 

---

September 14, 2009 | prabhakar wrote:

can i know the difference if my spred is closer and spred is larger , in both cases result is in specified limits. 

---

November 28, 2008 | Zafar wrote:

Hi, i found this realy a good and informative. I shall be thankful if u would help me in the following:
i. if want to find the chance of success between two values
ii. if expected value is greater than mean
iii. if expected value is less than mean

afer looking for corresponding value from Z table, how it will be interpreted.
Thanks
Waiting for an early response from ur side 

---

June 26, 2008 | Irfan wrote:

Let's say I have set up a standard to deliver pizza in 25 min. from the order to delivery. If I meet that target 68% of the time, is my process running at 1 Sigma. i.e. DPMO - 690,000 times I missed the target and did not meet my 25 min target or less thatn 25 min. Please explain this concept with example. 

---

March 28, 2008 | lavish wrote:

value of
1sigma
2sigma
3sigma
4sigma
5sigma
6sigma
 

---

Post a Comment

Comment:

Your Name:

Your Email Address:

.

To prevent comment spam, please answer the following :
What is 4 + 2: (enter the number)