Measuring Usability
Quantitative Usability, Statistics & Six Sigma by Jeff Sauro

10 Things to Know about Card Sorting

Jeff Sauro • April 3, 2012

Here are at least 10 things to know about using this popular user research method.
  1. Card Sorting involves having users (not designers, developers or VP's) sort a list of pages or categories together based on their understanding of the structure and relationship.

    Card sorting is useful for such things as finding out how users would group merchandise at an ecommerce website that has a huge selection of products and categories.

  2. The goal of card sorting is to generate a user-centered taxonomy which, among other things, should result in a better navigation structure for a website (less time to find products, higher conversion rates and a happier world for all of us).

  3. There are two types of card sorts:

    Open Sort: Users are asked to sort items into a group and make up their own groups and give the groups a name.
    Closed Sort: Users sort items into previously defined category names.

  4. Roughly half (52%) of UX professionals report using card sorting, about the same as in 2009 and up from 46% in 2005 (the 13% increase is statistically higher) [Source UPA annual Salary Survey].

  5. In a survey of 217 attendees of Usability Week 2008, Nielsen Norman Group reported that the average number of card sorts conducted per year was 2. 

  6. How long does it take to perform a card sort?  Well it depends on the number of cards and complexity of the categorization but William Hudson offers some guidance:
    • 20 minutes for 30 items
    • 30 minutes for 50 items
    • 60 minutes for 100 items

  7. Card sorting can be done by hand (hence the term card) but is now usually performed using software which allows for users from around the world to sort and generates fancy graphs and percentages. Popular packages include: OptimalSort, Websort and Userzoom. Note: We have business relationships with OptimalSort and Userzoom.

  8. Identifying groups of categories is often done using cluster analysis and results in a figure called a dendrogram. An example dendrogram from OptimalSort is shown below. Interpreting the groups in dendrograms is notoriously subjective, especially when there isn't high agreement between users. Using Factor Analysis may lead to better discrimination. 




  9. Confidence : Use confidence intervals around the percentage of items that were placed within a group to estimate the likely percentage of the total user population that would also group the cards that way. For example, if 32 out of 70 participants placed "Getting a New Person Started" in the "Hiring" category of an HR application, we can be 95% confident between 35% and 58% of all users also would place the card there (a margin of error or 11%).


  10. Sample Size: There is surprisingly little guidance on finding the right sample size for a card sorting study. Tullis and Wood (2004)[pdf] performed a resampling study with one large card sort involving 168 users and found the cluster results would have been very similar (correlations above .93) at sample sizes between 20-30.

    Another approach is based on finding the desired amount of precision around the percent of users who place cards in each category. It's basically working backwards from the confidence intervals.   For example, to have a margin of error of around 7% you should plan on having around 193 participants.

    Sample SizeMargin of Error (+/-)
    1027%
    2120%
    3017%
    3915%
    5313%
    9310%
    1159%
    1478%
    1937%
    2636%
    3815%
    5974%
    10643%


    The computations are explained in Chapter 3 and Chapter 6 of Quantifying the User Experience.




About Jeff Sauro

Jeff Sauro is the founding principal of Measuring Usability LLC, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 4 books on statistics and the user-experience.
More about Jeff...


Learn More


UX Bootcamp: Aug 20th-22nd in Denver, CO
Best Practices for Remote Usability Testing
The Science of Great Site Navigation: Online Card Sorting + Tree Testing Live Webinar


You Might Also Be Interested In:

Related Topics

Card Sorting, Information Architecture
.

Posted Comments

There are 2 Comments

April 1, 2013 | Candace wrote:

I am a graduate student at Arizona State University and am attempting to find a card sorting software to perform a closed sorting task using an image pool of 200 teapots. I would like the participants to sort 50 of the images only at a time, but to have these images randomly drawn from the total image pool for each participant. Does anyone know of a software that can rapidly perform this task? Thanks for your time and information.  


May 4, 2012 | S wrote:

It seems appropriate to mention here Remote Card Sorting (done online). There's a number of tools currently available to those interested in conducting this test remotely. One of such tools is: http://www.usabilitest.com/CardSorting 


Post a Comment

Comment:


Your Name:


Your Email Address:


.

To prevent comment spam, please answer the following :
What is 4 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[3789 Subscribers]

Connect With Us

UX Bootcamp

Denver CO, Aug 20-22nd 2014

3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis.Learn More

Our Supporters

Usertesting.com

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.