Comparison of Usability Testing Methods

Posted Comments

There are 8 Comments

January 24, 2012 | Anne-Marie McReynolds wrote:

Thanks for your feedback. How many remote users do you recommend testing (i.e. UserTesting.com) for formative evaluation? Are three users (per their introductory offer of three users for $29) enough?  

---

January 18, 2012 | Jim wrote:

Another great User Testing service is http://www.usertesting.com 

---

January 18, 2012 | Jeff Sauro wrote:

Anne-Marie,

I'm impressed with how you tied those blogs together (even if it revealed some inconsistency or change in thinking).

One of the main reasons remote unmoderated testing gets a "Good" instead of Excellent rating for Metric Quality is because of the issue of task time reliability. If a user goes onto Facebook or answers the phone during an unmoderated session and we don't know it, it certainly affects the quality of our task time metric. The data suggests it doesn't seem to impact completion rates and task and test level satisfaction as much.

I also gave it "Good" instead of "Poor" because of the ability to record users for a reasonable cost. The last few unmoderated tests I've run I've had recordings of the users (from Usertesting.com and YouEye.com). This allows you to go back and qualify unusual times. The cost has gone down and technology has improved enough in just 2 years that even task-times (when properly qualified) are becoming more reliable from remote unmoderated tests. 

---

January 18, 2012 | Anne-Marie McReynolds wrote:

You previously stated (in "97 Things to Know about Usability") that "task time data from remote tests were an unreliable measure of actual user task time," citing a 30% difference between remote-unmoderated usability tests and lab-based tests. What changed your opinion of the metrics?  

---

January 18, 2012 | Jeff Sauro wrote:

Colin,

You're right, I added the source note to the table to make it clear the data came from the 2011 UPA Survey.  

---

January 18, 2012 | Colin McCormick wrote:

I like the comparison between the three methods but where do the statistics come from? 52% and 50% seem trivial without any context.  

---

January 18, 2012 | John Romadka wrote:

One other attribute that would be helpful in your chart of testing methods might be "Environment". Some methods (and tools) are better/worse suited for web sites vs. web apps (or even desktop apps). 

---

January 18, 2012 | John Romadka wrote:

Great article! I wanted to share it with all my circles on G+, but I couldn't find a "+1" button. It just so happens that I know what the URL is for adding that button: http://www.google.com/webmasters/+1/button/

Perhaps you could add it to your site? Me and the other "four" people on G+ could use it.

P.S. - Also, maybe you should make your spam equations more difficult so you can elevate the quality of comments, so people like me can't ask you to add a G+ button. <elbow, elbow, wink> 

---

Post a Comment

Comment:

Your Name:

Your Email Address:

To prevent comment spam, please answer the following :
What is 1 + 2: (enter the number)