Measuring Task Times without Users
Jeff Sauro • January 4, 2011
A key aspect of usability is efficiency. Users should be able to complete tasks quickly.
Efficiency is usually measured as time on task, one of the quintessential usability metrics
For transactional tasks done repeatedly, shaving a couple seconds off a time can mean saving minutes per day and hours per week for users (think Accounting, Contact Management and Order Entry).
It can also mean saving all of us a lot of time (how many web-forms do you fill out a day?).
Does a new design reduce task time?
One of the biggest challenges to driving usability improvements through measurement is that by the time usability is measured it's too late to change the interface. This is especially the case for task time which is typically only collected on fully-functioning and live applications.
At best, most early stage usability activities focus on finding and fixing problems (which is an essential activity). Often early stage prototypes don't have working functionality so even if you were able to test a few users on your designs it's hard to say how reliable the task-time estimates would be.
If you could know early on whether a design takes less time (or at least doesn't take more time) then it would be ideal. You could quickly see if a design concept isn't going to work or have some idea about how much an improvement you'll make (Marketing Departments likes these metrics).
Predicting Task Times using Keystroke Level Modeling
Turns out wanting to know how long it will take people to complete tasks is nothing new to the Web, software or even computers. Since the industrial revolution there have been methods
for estimating how long it will take people to complete repetitive tasks.
Almost thirty years ago researchers at XEROX PARC and Carnegie Mellon developed a way to estimate experienced user task-times on computer interfaces called Keystroke Level Modeling (KLM)
KLM is part of the cognitive modeling technique called GOMS
and is detailed in the seminal book, "The Psychology of Human Computer Interaction" by Card, Moran and Newell (one of the classic usability books)
Card, Moran and Newell brought in hundreds of users and had them complete tasks repeatedly. They decomposed large tasks like typing a letter or using a spreadsheet into millisecond level actions (called operators). They found just a few of these operators can construct almost any task a user does on a computer.
- Homing: Moving Hand to Keyboard or Mouse: 360ms
- Clicking: the Mouse: 230 ms
- Pointing: with the Mouse: 1100ms
- Mental Operations: (Deciding what to Do): 1350 ms
In the ensuing decades several researchers have replicated these findings and tweaked some of the times by a few milliseconds. Using KLM you can predict a skilled user's task time (error-free) to within10-20% of the actual time.
To put this amount of error in perspective, it would take testing 80 users to have the same margin of error
as using KLM.
How to Use KLM
If you're mocking up an interface and want to consider user efficiency, just roughly sketch out what interface objects a user would interact with and how they progress through the screen.
For example, let's say you want to know which survey design alternative would take users less time to completeóone with radio buttons or one with drop-down lists (an example inspired by a recent customer service survey I was asked to fill-out).
Let's assume that that the user already has his hand on the mouse. It will take a user a few seconds to read the question and decide on the choice. But since this happens regardless of the input selection we can eliminate that from our comparisons.
First we estimate the time it takes for selecting from one drop-down list using the millisecond operators.
Figure 1: Selecting an option from a drop-down list will take an experienced user 2660ms (2.66 seconds).
|1. Point ||2. Click || 3. Point || 4. Click |
| || || || |
So we'd expect a user who knew what choice they wanted to make from a drop-down list to take around 2.66 seconds.
Next we estimate the time it will take to click on a radio-button:
Figure 2: Selecting an option from a radio-button group will take an experienced user 1330ms (1.33 seconds).
|1. Point || || 1100ms |
|2. Click|| || 230ms |
We'd expect a user who knew what choice they wanted to take around 1.33 seconds to make their selection from a radio-button group. The radio button option will probably take half the time of the drop-down list
If we have 3 questions, the ratio of time saved will stay about the same but the number of seconds separating them starts adding up. If a task is done repeatedly you can see how small changes to an interface can save a lot of time.
You might have noticed I didn't take into account the distance I needed to move the mouse or the size of the target I was clicking on. If you're familiar with Fitt's Law
you'll know that these two attributes are needed to predict task times. It turns our that 1100ms works pretty well for the typical sizes and distances objects are on a user-interface. Sometimes 1100ms over-predicts the time and sometimes it under-predicts the time but it tends to average out so that this is a good (and easy) estimate for pointing to objects with a mouse.
Try it yourself
Below I have the same three questions with the same response options. The only differences is one set uses drop-downs while the other uses radio buttons. KLM predicts that it'll take you around half the time to select the options using the radio buttons.
Click the Start button, select three choices then click the finish button to see how long it took you. Do it first on the drop-downs then again using the radio buttons. Go as fast as you can and try it a few times. The time difference will be shown after you stop the time on the radio button group.
User-Efficiency isn't Everything
Keep in mind that KLM applies to experienced error-free user times. So the operator times don't work well for predicting first-time use or mistakes. Also keep in mind that this is no replacement for traditional usability testing.
You still need to find and fix problems early and design an interface that meets the user's needs.
Saving time is of course not all there is to usability and better designs. There are a lot of considerations such as screen real-estate, functionality and simplicity when considering which design is "better."
But for repetitive tasks, providing a more efficient design is certainly a big factor. KLM offers a quick and surprisingly effective way of estimating experienced user task times. It is ideal when comparing alternative designs on tasks done a lot by a lot of people.
Your users will thank you for all the time you will save them...so they can then check Facebook more at work.Contact me
if you're interested in learning more about using KLM for estimating efficiency for your designs.