Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

Measuring Task Times without Users

Jeff Sauro • January 4, 2011

A key aspect of usability is efficiency. Users should be able to complete tasks quickly.

Efficiency is usually measured as time on task, one of the quintessential usability metrics.
 
For transactional tasks done repeatedly, shaving a couple seconds off a time can mean saving minutes per day and hours per week for users (think Accounting, Contact Management and Order Entry).

It can also mean saving all of us a lot of time (how many web-forms do you fill out a day?).

Does a new design reduce task time? 

One of the biggest challenges to driving usability improvements through measurement is that by the time usability is measured it's too late to change the interface. This is especially the case for task time which is typically only collected on fully-functioning and live applications.

At best, most early stage usability activities focus on finding and fixing problems (which is an essential activity). Often early stage prototypes don't have working functionality so even if you were able to test a few users on your designs it's hard to say how reliable the task-time estimates would be.

If you could know early on whether a design takes less time (or at least doesn't take more time) then it would be ideal. You could quickly see if a design concept isn't going to work or have some idea about how much an improvement you'll make (Marketing Departments likes these metrics).

Predicting Task Times using Keystroke Level Modeling

Turns out wanting to know how long it will take people to complete tasks is nothing new to the Web, software or even computers.  Since the industrial revolution there have been methods for estimating how long it will take people to complete repetitive tasks.

Almost thirty years ago researchers at XEROX PARC and Carnegie Mellon developed a way to estimate experienced user task-times on computer interfaces called Keystroke Level Modeling (KLM).

KLM is part of the cognitive modeling technique called GOMS and is detailed in the seminal book, "The Psychology of Human Computer Interaction" by Card, Moran and Newell (one of the classic usability books).

Card, Moran and Newell brought in hundreds of users and had them complete tasks repeatedly. They decomposed large tasks like typing a letter or using a spreadsheet into millisecond level actions (called operators). They found just a few of these operators can construct almost any task a user does on a computer.

  • Homing: Moving Hand to Keyboard or Mouse: 360ms
  • Clicking: the Mouse: 230 ms
  • Pointing: with the Mouse: 1100ms
  • Mental Operations: (Deciding what to Do): 1350 ms

In the ensuing decades several researchers have replicated these findings and tweaked some of the times by a few milliseconds. Using KLM you can predict a skilled user's task time (error-free) to within10-20% of the actual time.
 
To put this amount of error in perspective, it would take testing 80 users to have the same margin of error as using KLM.

How to Use KLM

If you're mocking up an interface and want to consider user efficiency, just roughly sketch out what interface objects a user would interact with and how they progress through the screen.

For example, let's say you want to know which survey design alternative would take users less time to completeóone with radio buttons or one with drop-down lists (an example inspired by a recent customer service survey I was asked to fill-out).

Let's assume that that the user already has his hand on the mouse.  It will take a user a few seconds to read the question and decide on the choice. But since this happens regardless of the input selection we can eliminate that from our comparisons.

First we estimate the time it takes for selecting from one drop-down list using the millisecond operators.

1. Point 2. Click  3. Point  4. Click
       
1100ms
230ms
1100ms
230ms
Figure 1: Selecting an option from a drop-down list will take an experienced user 2660ms (2.66 seconds).

So we'd expect a user who knew what choice they wanted to make from a drop-down list to take around 2.66 seconds.

Next we estimate the time it will take to click on a radio-button:

1. Point
   1100ms
2. Click
   230ms
Figure 2: Selecting an option from a radio-button group will take an experienced user 1330ms (1.33 seconds).

We'd expect a user who knew what choice they wanted to take around 1.33 seconds to make their selection from a radio-button group. The radio button option will probably take half the time of the drop-down list.

If we have 3 questions, the ratio of time saved will stay about the same but the number of seconds separating them starts adding up. If a task is done repeatedly you can see how small changes to an interface can save a lot of time.

You might have noticed I didn't take into account the distance I needed to move the mouse or the size of the target I was clicking on. If you're familiar with Fitt's Law you'll know that these two attributes are needed to predict task times. It turns our that 1100ms works pretty well for the typical sizes and distances objects are on a user-interface. Sometimes 1100ms over-predicts the time and sometimes it under-predicts the time but it tends to average out so that this is a good (and easy) estimate for pointing to objects with a mouse.

Try it yourself

Below I have the same three questions with the same response options. The only differences is one set uses drop-downs while the other uses radio buttons. KLM predicts that it'll take you around half the time to select the options using the radio buttons.

Click the Start button, select three choices then click the finish button to see how long it took you. Do it first on the drop-downs then again using the radio buttons. Go as fast as you can and try it a few times. The time difference will be shown after you stop the time on the radio button group.




I like traveling during the Holidays.
I think it's OK to use Facebook while at work.
I intend to spend less money in 2011.

Time :



 
Strongly
Disagree
1
 
2
 
3
 
Neutral 
4
 

5
6
Strongly
Agree
7
I like traveling during the Holidays.
I think it's OK to use Facebook while at work.
I intend to spend less money in 2011.

Time :


User-Efficiency isn't Everything

Keep in mind that KLM applies to experienced error-free user times. So the operator times don't work well for predicting first-time use or mistakes.  Also keep in mind that this is no replacement for traditional usability testing. You still need to find and fix problems early and design an interface that meets the user's needs.

Saving time is of course not all there is to usability and better designs. There are a lot of considerations such as screen real-estate, functionality and simplicity when considering which design is "better."

But for repetitive tasks, providing a more efficient design is certainly a big factor. KLM offers a quick and surprisingly effective way of estimating experienced user task times. It is ideal when comparing alternative designs on tasks done a lot by a lot of people.

Your users will thank you for all the time you will save them...so they can then check Facebook more at work.

Contact me if you're interested in learning more about using KLM for estimating efficiency for your designs.


About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Keystroke Level Modeling, Task Time
.

Posted Comments

There are 11 Comments

February 10, 2014 | xPnNYXfeQCJ wrote:

, <a href="http://thebestbinaryoptionstradingguide.com">binary options trading system</a>, [url="http://thebestbinaryoptionstradingguide.com"]binary options trading system[/url], http://thebestbinaryoptionstradingguide.com binary options trading system, ovao,  


April 17, 2013 | Goran Milovanovic wrote:

Thank you Jeff! You are the best ! <3 


August 7, 2012 | Oliver wrote:

Hi Jeff,
great article, thank you very much!

I'm currently writing my bachelor thesis in HCI about different modes of navigating between pages (especially swipes vs. buttons). Do you know of any scientific work detailing KLM in touch based interfaces, especially if there are established average times for (swipe) gestures? I will try to figure out these values in a small pretest myself but references would be great!
 


December 15, 2011 | Jeff Sauro wrote:

Lisa,
Great question and a common one. The short answer is yes. Iíve seen KLM applied to touch screens (e.g. GPS navigation devices) in a recent paper I reviewed. Tapping and swiping are just another way of interacting with devices and can be simulated in the same way a mouse click is.  


December 14, 2011 | Lisa Maurer wrote:

Jeff - Can KLM be adapted for the tapping behavior required for interacting with mobile devices? I can see many practical uses for this especially in the area of mobile testing. 


March 3, 2011 | Jamie wrote:

A great and informing article. Since my employers don't test content with users with any real frequency, this will at least give me a measurable process to work with. Thanks!

From trying your example above, it would also appear that the radio method is quicker by the fact that you can already see the response scale. 


January 12, 2011 | John Romadka wrote:

The Finish button on the second prototype using radio buttons doesn't work. It won't stop. (This only happens in IE8).

It works in Firefox.  


January 11, 2011 | Mary Fendrich wrote:

Very cool - will consider this in survey design for sure. The interactive tool was great. 


January 5, 2011 | Bob Schumacher wrote:

Great job (as usual)! Thanks Jeff!  


January 5, 2011 | David Travis wrote:

This is a great article -- and the interactive demo that you've included makes it a terrific teaching resource. Many thanks for taking the time to write and publish this article. 


January 4, 2011 | Susan Weinschenk wrote:

This is very interesting! And now I have proof of why I don't like drop-downs :) 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 1 + 2: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[3813 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.