Jeff Sauro • July 27, 2010

Recently Nielsen conducted a study on the reading speeds between the printed book, Kindle and iPad. From 24 users the study concluded that the iPad took about 6.2% longer (p =.06) and Kindle about 10% longer (p <.01) to read than the same story on a printed book.From this data Nielsen concluded "Books Faster than Tablets" and while tablets have improved dramatically over the years, they are still slower than the book (albeit modestly).

Put another way, the data tells us we can be 94% sure a difference as large as 6.2% between the iPad and book isn't due to chance alone.

Would you think differently about the difference in reading speeds if the result said 96% instead of 94%? John Grohol PsyD did and wrote an article using this study as an example of "bad research." One of John's main criticisms is that Nielsen's statistics don't back up his claim that books are faster than tablets.

In statistics classes students are taught to conclude something is "statistically significant" if the p-value is less than .05. Under this criteria the Kindle is statistically significant (p =.01) and iPad is not (p=.06).

- It is a nice round number
- It accounts for around two standard errors in a normal distribution
- It intuitively seems about right
- It comes from a time when we relied on tables of values instead of software

Conventions are helpful because they remove some of the subjectivity that can "stack the data" in a way that favors the author's hypothesis. Conventions are bad when used as commandments without thought for context. Numbers are objective but interpretation always involves judgment about the context and consequences of being fooled by chance.

- The FDA will approve a new drug p=.06
- Autism is associated with immunizations p =.06
- Reading speed is faster on books than on tablets p=.06

With a large enough sample size almost any difference is statistically different. It's more interesting to have a difference of 6% at a p=.06 than a difference of 1 % at p =.01 when comparing reading speeds. Only the latter would make it into a peer-review journal.

In applied research you'll never be able to test enough people, explore every possibility, or address all the limitations of your data. Applied research is about making better decisions with data and with limited time and money. Fortunately life and death are rarely consequences of making wrong decisions in applied research.

By all means, conduct your research, summarize it on the web and report your p-values. Tell us what conclusion you drew from the data and see if your readers are convinced. If there is a compelling story it will be replicated or refuted in another web-article. If your p-value is less than .05 it might even make it in a peer-review journal ... although fewer people will read it.

For the interested reader see : Statistics as Principled Argument and Beyond ANOVA: Basics of Applied Statistics.

The Experiment Requires That You Continue: On The Ethical Treatment of Users

28 Resources for Getting Started In UX

Nine misconceptions about statistics and usability

What five users can tell you that 5000 cannot

A Brief History of the Magic Number 5 in Usability Testing

How common are usability problems?

97 Things to Know about Usability

The Five Most Influential Papers in Usability

Confidence Interval Calculator for a Completion Rate

5 Examples of Quantifying Qualitative Data

Should you use 5 or 7 point scales?

8 Ways to Show Design Changes Improved the User Experience

How to Conduct a Usability test on a Mobile Device

Does better usability increase customer loyalty?

.

Quantifying the User Experience: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.