Calculating a Sigma Level from Task Success

by Jeff Sauro | September 17, 2004 :: RSS

Often the most reported measures of usability is task success. Success rates can be converted into a sigma value by using the discrete-binary defect calculation:

Proportion Unsuccessful= Defects/Opportunities

Where opportunities are the total number of tasks and defects are the total number of unsuccessful tasks. This calculation provides a proportion that is equivalent to a success or failure rate. For example, if 143 total tasks were attempted and 123 were successful(20 defects 143 opportunities) the equation would look like:

20/143 = .1399 By expressing .1399 as a percentage you get an unsuccessful rate of 13.9%. By subtracting the unsuccessful proportion from 1 you'll get approximately .86 or expressed in a percentage you could say that the task completion rate was 86%. Converting the proportion to a sigma value is a little more tricky.

Converting a Proportion to a Sigma Value

A sigma value is a description of how far a sample or point of data is away from its mean, expressed in standard deviations usually with the Greek letter σ or lower case s. A data point with a higher sigma value will have a higher standard deviation, meaning it is further away from the mean.

Standard deviations and sigma values assume your data is normally distributed from a continuous set of data. A normal curve has a mean of 0 and a standard deviation of 1. Success rates are Discrete-Binary (either a task was completed or it wasn't) not continuous(like time, weight or temperature) which means that binary data is not normally distributed.

Have no fear, for every statistical problem a statistical solution usually exits. We can treat binary data like continuous data because of the Central Limit Theorem. It states that as a binary sample gets larger, its distribution approximates a continuous distribution.

Approximating the Proportion to a Normal Distribution

An easy way to exploit the Central Limit Theorem solution and obtain a sigma value from the task proportion is by using the Excel function NORMSINV. NORMSINV approximates the area under a normal bell-shaped curve. Take the unsuccessful proportion .1399--the proportion defective and insert it into the following equation in Excel.

=NORMSINV(1-.1399)

 

 

 

Note: My sigma values DO NOT contain a 1.5σ adjustment or "shift."

Task Completion Benchmark

A common benchmark for the percent of users that should complete a task is 90% or 90 out of 100 users that attempt a task should be able to complete it. There are at least two problems with this benchmark:

1. For "experienced" users--users that complete the task at least on a weekly or monthly basis, 90% is much too low by definition. If the users complete the task frequently, why would we expect 10% of the tasks not to be completed?
2. Task completion by itself tends to only be a good indicator of usability for novice users or an easy measure of a really unusable product for all users. That is, if novice users are only able to complete 50% or 60% of the tasks then you already have a compelling reasons to conclude the product is unusable(for novice users) and don't need to spend as much time looking deeper for less obvious problems.

The Bottom Line

In a sense, task completion then is a good preliminary test for detecting egregious usability problems or for first time or novice users. I'd continue to use 90% as a goal for novice(never or rarely completed the task) and use 99%+ for experienced(complete the task weekly) users.

The next obvious question is: What is an acceptable sigma level for task completion? Is 6σ attainable?

View All Articles | Subscribe to RSS | Follow on Twitter | Get Email Updates

Rate this Article

Avg. Rating 5.922 (28)

Poor										Excellent
0	1	2	3	4	5	6	7	8	9	10

---

Comments

Name

Email Address Not Published, sold or abused

To prevent comment spam, please answer the following question before submitting (tags not permitted) : What is 2 + 3: (enter the number)

---

Most Popular Content

GoDaddy Web Hosting for $1.99

Six Sigma Training

Featured Product