Can you measure task time if users think aloud during a usability test?

Jeff Sauro • April 1, 2010

While it might seem like having users think aloud while they complete a task will increase the time on task—the literature is actually mixed. Some studies report no difference, some show longer task times for thinking aloud and others report faster task times for thinking aloud. It has been hypothesized that the reason users can perform faster while they think aloud is because this vocalization allows them to think more clearly through their approach for completing the task. It's like explaining a problem to someone when the actual process of explaining helps solve it (see Berry and Broadbent (1990) and Lewis 2006 p. 1282).

Probing

The key issue is whether you probe (ask for follow-up explanations after the user makes comments). Probing will increase the task time as users take longer to explain what they meant unless you are able to somehow subtract the probing time from the total task-time. If you need to probe, it doesn't necessarily render the task time metric useless. If you consistently probe users while they think aloud at different stages of testing then task time can still be used to compare the designs iteratively. There's a tradeoff between the information gains of probing (keeping in mind that people don't always know why they do things) and the increase in variability of time measurements, which is partly dependent on whether the study is primarily to find and fix problems or obtaining a benchmark.

Having it Both Ways: Retrospective Probing

As is often the case with my testing I want to make the most of user time. I want both a reliable benchmark of the website or application's usability and I want to know what the problems are and why.  So I ask my users to complete the tasks as quickly as they can without rushing. I don't say anything about thinking aloud. Some users will express their thoughts as they complete tasks and others are silent. When I see a problem that I want to know more about, I wait until after the task is complete and ask the user to elaborate more on what they did and why. I find this retrospective approach gets me the biggest bang for my user Buck (or Euro).

References

1. Berry, D. C., and Broadbent, D. E.  (1990).  The role of instruction and verbalization in improving performance on complex search tasks.  Behaviour & Information Technology, 9, 175-190.
2. Lewis, J. R. (2006). Usability testing. In G. Salvendy (ed.), Handbook of Human Factors and Ergonomics (pp. 1275-1316). New York, NY: John Wiley.

Rate this Blog

Avg. Rating 6.75 (8)

Poor										Excellent
0	1	2	3	4	5	6	7	8	9	10

.

Posted Comments

There are 2 Comments

April 21, 2010 | Abhay/ Cone Trees wrote:

This is what I was discussing with a number of eye tracking folks as well as professors from all over at the India HCI/ IDID 2010 conference and what you mention is just what I suggested- do an RTA (retrospective think aloud) if possible but since it isn't practical in most cases (how long can you hold a participant up really, especially if your test session takes more than half an hour), do a CTA (concurrent think aloud) and subtract probing and think aloud timings during data analysis to increase task time accuracy. 

---

April 2, 2010 | Alex Trofimovsky wrote:

There's a new experimental article that looks at different types of think aloud at CHI2010. 

---

Post a Comment

Comment:

Your Name:

Your Email Address:

.

To prevent comment spam, please answer the following :
What is 2 + 1: (enter the number)