Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

What's a Z-Score and Why Use it in Usability Testing?

Jeff Sauro • September 17, 2004

The Power of Z

A common statistical way of standardizing data on one scale so a comparison can take place is using a z-score. The z-score is like a common yard stick for all types of data. Each z-score corresponds to a point in a normal distribution and as such is sometimes called a normal deviate since a z-score will describe how much a point deviates from a mean or specification point.

In Six Sigma parlance, z-score and process sigma are used interchangeably and are sometimes called z-equivelents. Strictly speaking, the process sigma and z-equivalents are loosely tied to the statistical z-score. The statistical z-score has very strict definitions derived from the rules of the normal distribution. For most applications in Six Sigma, ignoring some of those constraints is innocuous. In usability testing the benefit of the standardization from process sigmas allow us to meaningfully compare disparate measures like task completion and time on task.

The z-score/process sigma is calculated by subtracting your sample mean from a target data point and dividing by the target standard deviation. This value is a measure of the distance in standard deviations of a sample from the mean and is expressed using the Greek letter σ. If your sample is 3 standard deviations from the spec limit, you would describe your process as 3 sigma. or 3σ

The further away a sample is from the spec limit the higher the z-score and process sigma. A higher process sigma means a less defective process. The term Six Sigma originates from the z-score. 6σ means that six standard deviations lie between the mean of a sample and the nearest specification limit. To visualize the Z-score see the Interactive Graph of the Standard Normal Curve

Each process sigma has two equivalent values which provide a meaningful way to compare data and understand how defective a process is:

  1. DPMO: Each expresses the probability of a defect in terms of a defect per million opportunities or DPMO. That is, if a condition were to occur one million times, how many times out of that one million would a defect occur? A process sigma of .5 is equal to 308,000 defects per million opportunities. And a process sigma of 2.5 means that 6,210 out of 1 million times there will be a defect. For a sample that is 6σ, the DPMO is .0.001. Some organizations prefer to think in terms of defects per opportunities instead of the more abstract "standard deviations above the spec limit."
  2. Probability of a Defect: The process sigma can also be described in terms of a probability of a defect. A z-score of .5 means there is a 30% probability of encountering a defect. A z-score of .25 means there is a 40% probability of a defect. For a sample that is 6σ, the probability of a defect is .0000001%. Note: Values do not include a 1.5σ shift.

Why use a Process Sigma?

The process sigma is helpful in three ways:

  1. It allows you to compare disparate types of data (seconds, which are a continuous measurement with task completion which is binary with errors which are discrete count data)
  2. It provides you with a probability of a defect
  3. You can meaningfully compare two different products or processes:
    1. The process sigma for one release of a software product can be compared to subsequent versions
    2. You can compare two different products' process sigmas
    3. You can compare one module of the same product with a different module on the same product
    4. You can use the properties of the normal distribution to aide in assessing and improving your data set.

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar

Related Topics

Z-score
.

Posted Comments

There are 13 Comments

January 25, 2013 | Soma wrote:

I want to comparision of two Z score but I have not getting more idea about the comparision of the Z score. 


October 3, 2012 | anonymous wrote:

Comprehensive 


September 11, 2012 | Khalil wrote:

I am UX (QoE) Scientist and I found this post easy to understand and informative. Thanks 


July 5, 2012 | MaraCantu28 wrote:

Don't you recognize that this is the best time to get the <a href="http://goodfinance-blog.com">loans</a>, which can make your dreams come true.  


April 18, 2012 | pramod wrote:

It is very good to find these kind of learning material on web. it makes the learning easier. 


October 13, 2011 | anonymous wrote:

The Interactive graph drives the point home like a nail. 


February 24, 2011 | PALASH ROY wrote:

Very good thing 


November 23, 2010 | Dr. John wrote:

wonderful! simple and comprehensive. 


September 11, 2009 | Barbara wrote:

Discuss types of research where using the t statistic may be an appropriate alternative to using a z-score. 


August 25, 2008 | DERYA TERZIOGLU wrote:

what is the difference between z score and sigma value? 


May 28, 2008 | samar wrote:

that was great job thank you very much 


May 28, 2008 | samar wrote:

that was great job thank you very much 


March 22, 2008 | Michael wrote:

How do you compare two z scores to see if there is a significant difference between the two? would that be a t-test, or something else? 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 3 + 3: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4121 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.