Measuring Usability: Products

How to conduct a Quantitative Usability Test

Download a comprehensive guide for conducting Quantitative Usability tests. This 64 page report covers 72 of the most common questions and issues that arise when you measure usability.

Click for more details on this Screen Shot of

More...

If you will be making copies or distributing the product, please purchase the site license

Do you know how to measure usability? Do you have questions about the benefits and process for conducting a quantitative usability test? You're not alone, while many people have experience conducting qualitative usability tests, quantifying the usability and expressing it numerically is new. I've assembled answers to the 72 most common questions that arise from measuring usability. In this 64 page report I provide concrete examples and plenty of data from a dataset of 120 usability tests, the latest usability research and advice from my decade of experience conducting quantitative usability tests. That's only $1 per answer!

Features & Benefits

Addresses 72 of the most common questions and issues that arise from measuring the usability of an application: Sample Size, Questionnaires, Reporting, Analyzing, Tasks and more
Provides both a high-level overview of the benefits of quantitative usability testing and the mechanics involved in recording, analyzing and reporting usability data.
Filled with concrete examples and data from over 120 quantitative usability tests and survey results from other usability engineers
Includes nine figures, seven tables, fifteen- references in 64 pages

Who Should Buy

Anyone who needs to measure usability and has questions about conducting a usability test.
Usability Professionals who have experience conducting qualitative usability tests but have questions about gathering metrics

TESTING
1. Can you measure usability?
2. What is the difference between a formative and summative usability test?
3. What is a quantitative usability test?
4. What are the advantages of a Quantitative Usability Test?
5. What are the basic steps for conducting a quantitative test?
6. How long does it take to conduct a quantitative usability test?
7. How long should the test session last?
8. How do I select the users for testing?
9. What are common usability metrics?
10. How do I record the measurements?
11. Do you sit in the room with the user or behind a one-way mirror?
12. Do you compensate users for their time?
13. Do I need to conduct a pilot test?
14. Do you ask users to think aloud or have them complete the tasks silently?
15. Is there a difference in data from unattended versus lab-based tests?
16. Is user testing the only way to measure usability?

TASKS
17. How do you define the task-scenarios?
18. What are some examples of good task-scenarios?
19. How do I select the tasks for testing?
20. If users select their own tasks, can I still measure usability?
21. How long should a task scenario last?
22. How many tasks should I have the users attempt?
23. Should all the users get the tasks in the same or different order?
24. How do I counterbalance the order of the tasks to avoid a learning effect?
25. Should users repeat tasks or only attempt them once?

SAMPLE SIZE
26. Do I only need to test with 5 users?
27. What is the minimum sample size I should plan for?
28. How do I determine my sample size?

METRICS
29. How do you measure UI problems?
30. If 1 user encounters a problem will it impact 1% or 20% of all users?
31. How do you determine task success or failure?
32. How do you code task completion rates?
33. When do you start and stop the time?
34. What is an acceptable task completion rate?
35. What is an error?
36. Do I need to record errors?
37. How do you record errors?
38. How do you analyze errors?
39. What is an acceptable average number of errors per task?
40. Do I need to convert errors into an error rate?
41. How do you define an error opportunity?
42. What is an acceptable minimal error rate per task?
43. How long should a task take?
44. Is there a single usability score that I can report?
45. What is SUM?
46. How do you measure learnability?
47. Should you assist users during a usability test?
48. If you assist a user during a task, is it a task failure?

QUESTIONNAIRES
49. Do I administer a questionnaire after each task or at the end of the test?
50. What post-test satisfaction questionnaires do you recommend?
51. What is an acceptable SUS Score?
52. What post-task satisfaction questionnaires do you recommend?
53. Can I use my own questionnaire or should I use a standardized one?
54. How many scale points should a response item have: 3, 5, 7 or more?
55. Does it matter if the number of scale steps is odd or even?
56. Should I alternate positive and negative statements in my questionnaire?
57. Can I use statistical tests to analyze questionnaire data?

ANALYSIS
58. Why & how do I compute a confidence interval around a completion rate?
59. How do I compute a confidence interval around an average task time?
60. How do I compute a confidence interval around a mean satisfaction score?
61. How do I compute a confidence interval around a UI Problem?
62. Should I put the confidence intervals in a report?
63. What confidence level do I use?
64. Do my data need to be normally distributed?
65. What percent of all users can complete a task based on my sample?
66. Will at least 70% (or another percent) of all users complete a task?
67. Will the average user task time be less than 2 minutes (or another time)?

REPORTING
68. What should go into the report?
69. Should I use the Common Industry Format for reporting results?
70. Do you include failed task times in the reported average task time?
71. How should I report average task time?
72. Can I report percentages on completion rates from a small sample?

REFERENCES