Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

How Do You Calculate a Z-Score/ Sigma Level?

Jeff Sauro • June 14, 2004

The benefit of using a z-score in usability metrics was explained in "What's a Z-Score and why use it in Usability Testing?" this article discusses different ways of calculating a z-score.

The short answer is: It depends on your data and what you're looking for. If you've encountered the z-score in a statistics book you usually get some formula like:

The above formula is for obtaining a z-score for an entire population. Usability testing obviously samples a very small subset of the population and thus the following formula is used:

Where x-bar and s are used as estimators for the population's true mean and standard deviation. Both formulas essentially calculate the same thing:

Calculating a Z-Score Example

For example, lets say you took the GRE a few weeks ago and got scores of 630 Verbal and 700 Quantitative. How good are these scores? Which is better, the Verbal or Quantitative score? Using a z-score can tell you how far you are from the mean and thus how well you performed. If you know the mean and standard deviations for a set of GRE test takers you can compare your scores.

ETS publishes the means and standard deviations of a set of test takers on the GRE website.


Verbal Quantitative
Mean 469 591
StDev 119 148

By plugging in your scores you get the following:

Verbal z = (630 - 469) ÷ 119 = 1.35σ

Quantitative z = (700 - 591) ÷ 148 = .736σ

To convert these sigma values into a percentage you can look them up in a standard z-table, use the Excel formula =NORMSDIST(1.35) or use the Z-Score to Percentile Calculator (choose 1-sided) and get the percentages : 91% Verbal and 77% Quantitative. You can see where your score falls within the sample of other test takers and also see that the verbal score was better than the quantitative score. Assuming the sample data was normally distributed, here's how the scores would look graphically:

Figure 1: Verbal Score


Figure 2: Quantitative Score


Z-Scores and Process Sigma

An interactive Graph of the Standard Normal Curve similar to Figures 1 & 2 is available for you to visualize how the z-scores and the area under the normal curve correspond. The graphs also allow you to see the difference between one and two-sided (also called two-tailed) areas. In Six Sigma the process sigma metric is derived using the same method as a z-score. However, in Six Sigma you are measuring the distance a sample mean is above a specification limit--there can be an upper and lower spec limit that a sample must fall between as well. As in the z-score, you still use the same normal-deviates from the z-table to approximate the area under the curve. The process sigma metric is essentially a Z equivalent.

When testing software with users, task times are usually a good metric that will reveal the individual differences in performance. For task times there typically is only an upper spec limit. That is, it usually doesn't matter how fast a user completes a task, but it does matter if a user takes too long. For example, say you and your product team determined that a task should be completed in 120 seconds. 120 seconds becomes your Upper Spec Limit (USL). You sampled 10 users and got these task times:

Sample
100
99
101
125
100
123
96
90
98
116

USL: 120
Mean: 104
StDev: 12

To calculate the process sigma you subtract the mean (104) of the sample from the target (120) and divide by the sample standard deviation (12). For Sample 1 the process sigma is -1.32σ. The visual representation of the data can be seen below:



In the case of task times, a negative process sigma is ideal--as you want more people completing the task below the task time, not above it. You can simply drop the negative when communicating the results in the event it causes confusion. If you were to make radical improvements to the UI and then sampled another set of ten users, here are more results:

Sample 2
USL: 120
Mean: 75.8
StDev: 12.14


In the redesign, the average of the new sample is well below the spec limit and the process sigma is now very high. The corresponding defect area is now only .01% and the quality area is 99.98%



Of course having users perform that much below the spec limit is not very common due to the inherent variability in user performance.

If you need more help with z-scores, see the Crash course in Z-scores, a tutorial with plenty of pictures, examples and review questions for you to grasp this concept.

About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

.

Posted Comments

There are 74 Comments

May 13, 2014 | Gopalbhai N Desai wrote:

Ineed further example for Z score with out USL 


December 14, 2013 | yasini wrote:

good but why after read in z score value we + or - 


September 15, 2013 | G. Key wrote:

I get it now, Thanks to this site! 


September 12, 2013 | anonymous wrote:

it would be better if the equation was put into an example, rather than there just being an x-mew divided by sigma 


May 27, 2013 | Slim Shady wrote:

Thank you so much for this amazing explanation. I finally understand z-score, just in time for my Algebra SOL! 


April 25, 2013 | Hyung Seo wrote:

Doesn't help at all 


April 10, 2013 | Celeste wrote:

This is great! Thank you! 


April 9, 2013 | lynda wrote:

Very comprehensible, im not afraid anymore to know about statistics!!! 


April 3, 2013 | sam wrote:

Bob takes an online IQ test and finds that his IQ according to the test is 134. Assuming that the mean IQ is 100, the standard deviation is 15, and the distribution of IQ scores is normal, what proportion of the population would score higher than Bob? Lower than Bob?  


January 29, 2013 | Ashwini wrote:

Very simple and easy to understand ! 


January 19, 2013 | Dr Ashvini Sengupta wrote:

Very good for a beginner 


September 21, 2012 | Miguela wrote:

Congratulations and thank you for the information! May you richly be blessed! 


August 17, 2012 | bura wrote:

great job for layman 


July 12, 2012 | Liliana35Warren wrote:

The <a href="http://goodfinance-blog.com/topics/business-loans">business loans</a> seem to be useful for people, which want to ground their own business. In fact, it's not very hard to receive a auto loan.  


July 4, 2012 | Ajith p Perera wrote:

Thanx ! 


June 5, 2012 | abc wrote:

Thank-you! That really helped me. THANKS 


May 12, 2012 | Kerny wrote:

great info. 


February 1, 2012 | Carla Arce wrote:

Hi. I would like to ask, what could be the best formula to test the difference of two sets of scores. The number of the test takers are less than 20 students. Thanks :) 


January 31, 2012 | Ranmini wrote:

Is it right to pool two groups of student populations who sat for two different papers of the same subject with a combined mean and a combined standard deviation. 


January 31, 2012 | Ranmini wrote:

Is it right to pool two groups of student populations who sat for two different papers of the same subject with a combined mean and a combined standard deviation. 


December 19, 2011 | Sunil Chandrasiri wrote:

I got an idea regarding Z score. Is six sigma an alternative concept to Z score? 


December 6, 2011 | Jeff Sauro wrote:

Those are actually divided by signs ÷ . They might be rendering like plus (+) signs as it is a special character in HTML.  


December 6, 2011 | anonymous wrote:

In the example for computing z-scores for Verbal and Quantitative, you have "+" signs where you should have "/" signs. 


November 22, 2011 | Sangeeta Bose wrote:

Good description 


November 3, 2011 | Martine wrote:

Very well done. 


November 1, 2011 | Gilberto Murillo wrote:

I liked your explanation very much.
Thank you!! 


September 1, 2011 | Veena wrote:

Sir/Mam, it is not clear to me. I would like to ask you,if i get 164 toal out off 225 in IBPS ,,,,,,so what r my cut off marks? 


July 31, 2011 | sri wrote:

Thank you Very much..... 


June 8, 2011 | Hassan Simba Hassan wrote:

It helps me a lot in my statistics class. The site is among my major references in Quantitative Research Methodology class. 


May 3, 2011 | muhammed najeeb K wrote:

informable and simply explained 


January 14, 2011 | gabriel wrote:

wat do i do 2 calculate zscore with variance knwn 


November 27, 2010 | English major taking beginning stats class wrote:

This is not explained for non-math people. Very technical, lots of examples which make little sense, a nice little suggestion to just use the Excel formula =NORMSDIST(1.35) which I have no clue what is. This makes it more confusing, not less.  


November 4, 2010 | Six Sigma Student wrote:

The graphic helped tremendously. I now know how what is meant by looking at the left-tail and how to
diagram sigma.  


November 3, 2010 | ashish wrote:

its good 


August 23, 2010 | bhagyalakshmi wrote:

i want to know the use and way of caliculating z test for comparative studies 


June 9, 2010 | tamer wrote:

More Please 


June 7, 2010 | Obi wrote:

It was very good, but hard to understand. 


May 24, 2010 | nish wrote:

simple and clear... 


April 29, 2010 | CT wrote:

Good article 


April 3, 2010 | shakya wrote:

there are only one doubt how to calculate sigma is not cleared. 


March 11, 2010 | Mark Francis wrote:

I wanted to find info on z scores for a student. I found this article very informative. 


March 8, 2010 | Aditi M Sengupta wrote:

Thanks. 


March 7, 2010 | anonymous wrote:

how does one get from a sigma interval to percent? 


March 3, 2010 | John C. Beckerle, PhD, DAV wrote:

Use of words that need complete definitions, such as defect area is essential. Double blind statistical testing of a broad spectrum of individuals could give small defect areas, but that few persons fall into that region might mean that the actual test was not sensitive enough to properly evaluate the differences required to detect those with a benefit compared to those getting an identical looking substance but not the substance to be evaluated. I would seriously question results of a testing that gave small defect areas for many persons selected into two groups as conclusive evidence of a worthlessness of a treatment substance without detailed explanation of the testing procedure. This would be especially so if the testing of the same individuals took place over several years. The benefits of some herbal substances could have benefits that the medical profession, including research organizations might not want to see available to say the elderly who have declining brain function. So, my main point is that such testing takes lots of grant money, and that money could have a biased source. My question is how does a medical journal that reports results submitted evaluate whether or not the entire evaluation was not biased? Okay, we need to trust those involved in the evaluation. Why should that be needed? The testing process it self should prevent bias automatically! Why not? 


February 18, 2010 | ABHIK CHATTERJI wrote:

EXCELLENT 


January 24, 2010 | ali wrote:

that is very summerized unable to understand nad i raed difrent formula 


January 14, 2010 | Dr. Al-Ani wrote:

I am confused in the z score of body weight to age when every one z zcore equal to 1 standard deviation.how this can be decided? 


January 9, 2010 | Vijay Kumar wrote:

Please provide Z score calculation with standard daviation and Roll up machanism 


December 6, 2009 | Page wrote:

hello if you get a score wich is 22 what is the posabal level you would get? 


October 28, 2009 | maggie wrote:

z-score of z = +3.00 indicates a location that is the following.
1slightly above the mean
near the center of the distribution
the location depends on the mean and standard deviation for the distribution
far above the mean in the extreme right-hand tail of the distribution 


October 14, 2009 | Jessie Ratliff wrote:

Do you think Dr. Zak’s choice of scaling is appropriate 


October 14, 2009 | Jessie Ratliff wrote:

What scale of measurement is Dr. Zak using 


September 7, 2009 | Oliver wrote:

How do you find the Z sigma value without knowing the population size?
I have a mean of 5 and a standard dev. of 1.5 and I need to find out 95% of the breaking strengths. I know I should use the formula for z=(n-mean)/stand. dev but I don't know the population size. it is not given. I tried using the 95% but I'm not getting what my instructor says I should be getting. 


August 28, 2009 | SUMERA wrote:

HI I AM SUMERA,SCIENTIFIC OFFICER IN A RESEARCH ORGANIZATION.I FIND THIS WEB PAGE VERY INFORMATIVE AND LEARNING AS I HAVE PROBLEM IN CALCULATING Z-SCORE FOR MY RESULTS OF SPECTROMETER. 


July 12, 2009 | Theresa wrote:

200 students took a test. The scores were normally distributed. Your score was in the 60th percentile. How many people scored at or below your score? 


June 27, 2009 | jimboy wrote:

gud day. i would like to ask how a standard score (general scholastic aptitude) is transformed back to its raw score. thanks for the help 


June 25, 2009 | Tasha wrote:

What is the z score for each of the five numbers 1,2,3,4,5 if their standard deviation is 1.41? 


June 4, 2009 | jormin wrote:

for a given process, short term stanard deviation is 1.0, and mean is 4.5. what is short-term process Z score if upper specification limit for the process is 9.0? - thanks 


May 30, 2009 | Mona wrote:

which raw score corresponds to a z-score of +0.5? 


May 26, 2009 | EDITH SHERROD wrote:

Determine the area under the standard normal curve that lies between (a) z = -1.05 and z= 1.05. 


April 29, 2009 | Carmen M. Maldonado wrote:

I have a math problem I cannot solve. I used the same method, but it does not work. There are 60 girl: mean is 71,Std. 6, 50 boys: mean 66, Std, 5. Those students with a score greater than 75 are eligible to go to a field trip. What percent of those going to the field trip. I found the z-score for each one, but I cannot the answer. According to the book the answer is 88%. Can you explain to me how to solve this problem. Thanks 


March 29, 2009 | m d wrote:

The deviation is 20,000 people. What is the z-score 


February 7, 2009 | Vincent wrote:

I am confused on how to calculate a z-score. I am doing a six sigma project right now and am lost on doing the z-score. Here is the information I am provided:

Calculate the average student score necessary for the district to retain its federal funding.
(You may assume the standard deviation will not change). This will require some
thought. Think of the z formula. You will be solving for ì (average test score in the next
school year.)
Think about what you already know. You know the standard deviation (because we assume
that the variation in test scores stays approximately the same across years.)
You also know that 70% of students must meet the cut off score.

Data:
The following are the results from last year's MEAP scores for EHS
N = 1000
ì = 69.7
ó = 11.55

A student must score at or above 70% on the MEAP to meet the president's cut off score

Project Information:

The improvement program “No Child Left Behind” legislation has placed emphasis
on testing students, and has tied federal funding (and thus some matching state funds) to a certain
percentage of students meeting or exceeding cut off scores on state level achievement tests.
The challenges that face education are very interesting indeed: unlike in manufacturing
(where Six Sigma methodology was originally developed) the outcome is not a product that
meets stated specifications, rather it is a student who is “prepared” to take his/her place in society.
Unfortunately, the definition of “prepared” is elusive. Also unlike in manufacturing, public
educational institutions cannot reject the materials they receive (students) as being defective
and send them back – they must take students at whatever point they received them, and advance
their state of preparedness.
The challenge for this project revolves around Everyday High School in Normal, Michigan.
Last year only 49% of the students at the high school met the cutoff score of 70%
of questions answered correctly on the Michigan Educational Assessment Program (MEAP) –
Michigan’s state level test. This percentage of students meeting the cutoff score is well
below the 70% of students who must meet the cutoff score for the district to continue to receive
federal funding. Because losing the federal funding (approximately $2,000,000 per year)
would mean cuts in programs and maybe even staff that the board of education (BOE), parents,
community, students, and staff would find unacceptable, the board of education has directed
you (the Six Sigma Black Belt) to conduct a project to improve the scores on next year’s
MEAP tests.

Any help would be greatly appreciated. 


January 30, 2009 | Julie wrote:

There are two major tests of readiness for college, the ACT and the SAT. ACT scores are reported on a scale from 1 to 36. The distribution of ACT scores for more than 1 million students in a recent high school graduating class was roughly normal with mean = 20.8 and standard deviation = 4.8. SAT scores are reported on a scale from 400 to 1600. The SAT scores for 1.4 million students in the same graduating class were roughly normal with mean = 1026 and standard deviation = 209.
Jose scores 545 on the SAT. Assuming that both tests measure the same thing, what score on the ACT is equivalent to Jose's SAT score? 


December 6, 2008 | Carol Brazzier-Auguste wrote:

Using the raw score of 64, calculate the Z-score and the T-score.

I managed to work out the Z-score,since the mean for that score is 56 and the SD is 4 using the normal curve, but when you have to calculate the T-score I am using the formula 50 + 10(Z)/SD??
If so what figure amd I using for SD is 4 where the formula will be 50 + 10 (2)/4
or 50+10(2)/10. I theory it is said that In T-scores the mean is always 50 and standard deviation is always 10

By the way I got the Z-score of 2 by subtracting the raw score of 64 from the mean (the mean for that set of scores was 56 and the standard deviation was 4 


September 23, 2008 | Kelly wrote:

How can I calculate the z score for raw scores of 22 and 28 in with numbers 16,16,22,23,24,27,28,30,31 


September 8, 2008 | basua wrote:

How do I calculate 1Sigma = 68.26, 2Sigma = 95.44, 3Sima = 99.73 and so on? 


August 22, 2008 | ratan wrote:

interested in knowing all about basic statistics - i am a new learner. thanks 


August 18, 2008 | victoria wrote:

give the raw scores for a person with scores 2.4,1.5,0,-4.5 not sure how to do this 


July 31, 2008 | Karen wrote:

A union of restaurant and foodservice workers would like to estimate the mean hourly wage, , of foodservice workers in the U.S. The union will choose a random sample of wages and then estimate using the mean of the sample. What is the minimum sample size needed in order for the union to be confident that its estimate is within of ? Suppose that the standard deviation of wages of foodservice workers in the U.S. is about .

Carry your intermediate computations to at least three decimal places. Write your answer as a whole number (and make sure that it is the minimum whole number that satisfies the requirements). 


July 28, 2008 | Mina wrote:

Using the cumulative area from the left, locate the closest probability in the body of a table of normal distribution values and identify the z score corresponding to 0.99. Alternately, technology may be used to determine the most accurate z score.

How does the temperature for P subscript 99 come out to be approximately 2.33? 


June 19, 2008 | cochise180 wrote:

how do I calculate z score for a nominal of zero with usl 0.008" and lsl -0.008". i get good Cpk and Cp but bad Z score. 


May 9, 2008 | Tim wrote:

Am confused as to how to calculate a negative Z score into a percentage using a standard Z score table 


December 19, 2007 | anonymous wrote:

Why can't all math be explained this simple?
Thank you for this explanation! 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 3 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[4095 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Usertesting.com

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.