8 Ways to Show Design Changes Improved the User Experience

Posted Comments

There are 7 Comments

December 14, 2011 | Jeff Sauro wrote:

Jeff

Good question and yes, in my experience for large enterprise software products that support different functions, for different roles, it's not uncommon to test a fraction of the functionality and use the SUS or other task-metrics to assess that --not just entire products or product suites. 

---

December 13, 2011 | Jeff Noyes wrote:

This seems like a viable solution for very small products. What about large ones with many many features. Are you breaking the SUS scores down by feature? 

---

March 21, 2011 | Jeff Sauro wrote:

José

You can use the SUS for any level of experience, it's not just for initial exposure. Experience matters, I've found on average long-term exposure increases SUS scores by 11% over initial use. But the effects of unusable software are typically so larger they over-ride differences in experience.

So there's nothing wrong with using experience users in late-stage analysis, just document their experience when making comparisons between groups. Ideally you're comparing users with similar experience with the application. 

---

March 21, 2011 | Jeff Sauro wrote:

Meredith,

Great question. It's a common misconception that sample size and quantitative/qualitative studies are related. The minimum sample size you need to use SUS or any quantitative metric is 2.

At a sample size of 8, you can show statistical differences between two iterations. The major difference is that you are limited to detecting large differences. But in early stage designs, you're largely concerned about those big differences that users notice.

As far as effective recruiting, you've really got the same issue whether you are quantifying your findings (as in a benchmark) or detecting problems. If you're using people who aren't representative of your user population, they you may be missing problems or finding false positives.

The notion that you need a large sample size for quantitative usability testing likely comes from the need for precision in benchmark testing. If you want to know how usable an application is, you can haul in some users and test them attempting representative tasks. Of course, any sample will fluctuate do to sampling error (just like a presidential poll does). To minimize the sampling error you need a larger sample size. As a general rule, just remember 20/20. At a sample size of 20, your metrics (completion rates, time and SUS) will have a margin or error of around +/- 20%.

The relationship between sampling error and sample size is an inverse square root relationship. So to cut your margin of error in half you need to quadruple your sample size. A margin of error of 10% would therefore need around 80 users (see http://www.measuringusability.com/test-margin.php).

So the number I test will depend on how much precision is needed or how large of a difference I'm hoping to detect. But for every test I collect metrics, you never know when they can help make better decisions. It's hard to show a small improvement with small sample sizes, but for large improvements, even very small sample sizes are statistically meaningful.
 

---

March 21, 2011 | Meredith Noble wrote:

Jeff, quick question – am I correct that any usage of the SUS, or the measurement of time completion, effectively changes a qualitative usability test to a quantitative one, and therefore requires a fairly large sample size (and extremely realistic recruiting) to be valuable? Normally we test 8 people at a time, but I can't imagine comparing SUS before and after without a much larger group of people. Not sure what would be required for statistical significance, but surely more than 8? How many do you test when using these measures? 

---

March 4, 2011 | José Tavares wrote:

Isn't the SUS questionnaire more suitable to use in the early stages of a product?
If I'm tracking experienced users that use the product for some years now, won't the users be biased by the previous usage of the product?  

---

March 4, 2011 | José Tavares wrote:

Thanks for this post.
I'll try immediately to sketch a plan to assess the improvements we are making in our product right now.
 

---

Post a Comment

Comment:

Your Name:

Your Email Address:

To prevent comment spam, please answer the following :
What is 3 + 4: (enter the number)