How to quantify comments

Jeff Sauro • February 21, 2011

Just because customer information is qualitative doesn't mean you can't use some quantitative methods to help interpret and prioritize your findings. 

Quantifying the frequency of comments with a confidence interval helps you estimate a sentiment in the total user population.


Analyzing and prioritizing comments is a common task for the user researcher. Open ended comments take all sorts of forms.

• Reasons why customers are promoters or detractors for a product
• Customer insights from field studies
  
• Product complaints to calls to customer service

Step 1: Group

Consolidating and grouping comments into categories has its own methodology (see for example Contextual Design Chapter 9). Some comments will be virtually identical and grouped easily. Others will differ and require additional layers of grouping.

There can be a high amount of variability between evaluators grouping items. When possible it is a good idea to independently categorize comments and assess the degree of agreement using a statistic like Kappa.

Step 2: Count

Once comments, insights or complaints have been categorized, count the number of comments that appear in a category and the total number of comments. You can then prioritize these comments by generating a confidence interval to understand what percent of all users likely feel this way.

How Likely are you to Recommend iTunes to a Friend?

For example, I recently asked 56 users to rate their experience with the iTunes software program. Users answered several questions about the ease of use and quality of iTunes.

I also asked them how likely they'd recommend the product to a friend on a 0 to 10 scale where 10 means extremely likely. By subtracting the number of responses from 0 to 6 (Detractors) from the number of 9 and 10 responses (Promoters) you can generate the Net Promoter Score.  Of the 56 respondents, 31 gave a 9 or 10 and 12 responded with a 0 to 6 which generated a NPS of 34% for iTunes (19/56).

To understand the key reason WHY users were Detractors, I categorized the open ended comments for Detractors.   In total there were 16 key points I pulled from the comments.

Category 1	Category 2	Rating	Comment
Conversion	Price	2	I used to use iTunes much more often before they raised their prices from 99 cents to $1.29 per song.  I still use the iTunes player to transfer songs to my iPod but I usually buy songs on Amazon.com now because it's a much better deal.
Conversion		2	It's okay at best, but I find a lot of the more obscure music I listen to is not available so I rarely use it to find music as a first source. Also, I have to convert anything from iTunes in order to put it on my Zune, which is a pain.
Difficult to Use		3	I dislike iTunes. It's unnecessary and can be annoying and difficult to use.
Necessary to Use		3	People generally say that iTunes kind of sucks. With ipods as ubiquitous as they are, though, iTunes is pretty necessary.
Necessary to Use		3	I don't really like it but since i have an ipod I'm obligated to use it
Proprietary		4	There are better software options out there, and I don't like the DRM of itunes.
Difficult to Use	Proprietary	4	It's cumbersome and too proprietary.
		4	<no comment>
Proprietary		5	I really don't like the way apple handles there control over your own item in which you purchased.
Difficult to Use	Installation	5	The software installer is very bloated, and there are better music library managers for some people.
Difficult to Use	Necessary to Use	5	iTunes is good, but it is unnecessarily complex. Only thing which makes someone to have it on their computer is their Apple devices.
Proprietary	Slow	6	It's a good program, but it's a little slow and I don't like the fact that itunes are incompatible with MP3 players.

I generated a 95% confidence interval around each group of comments using the online calculator to get an idea about how prevalent these reasons would be in the whole iTunes user population. For example, we can see that one of the primary reasons users are unlikely to recommend iTunes is because it is difficult to use. 

Category	# of Comments	Total Comments	%	95% CI Low	95% CI High
Difficult to Use	4	16	25	9	50
Proprietary	4	16	25	9	50
Necessary to Use	3	16	19	6	44
Conversion Process	2	16	13	2	37
Installation	1	16	6	1	30
Price of Songs	1	16	6	1	30
Slow	1	16	6	1	30

It turns out that ease of use is often a key driver of product loyalty and iTunes is no exception. We can be 95% confident between 9% and 50% of all iTunes Detractors feel ease of use is one of the main reasons for not recommending the product to a friend.

Ease of use isn't the only major reason for the detracting. By combining the "Proprietary" and "Necessary to Use" categories, we have 7 out of 16 Detractors (44%) who feel they have no choice but to use a proprietary product. We can be 95% confident between 23% and 67% of all iTunes Detractors feel this way.

While it's unlikely Apple's business model is going to embrace open source any time soon, there are probably some easy ways to fix the ease of use problems with iTunes.  Quantifying open ended comments and using confidence intervals can help you prioritize fixes and allow you to communicate your qualitative findings more confidently.
 
 

Learn More

Jeff Sauro hosted a live webinar on February 28th, 2012 on Best Practices for Remote Usability Testing. The event was overbooked so if you missed it you can now view a recording.

Rate this Blog

Avg. Rating 6.57 (7)

Poor										Excellent
0	1	2	3	4	5	6	7	8	9	10

Related Topics

Qualitative, Survey

.

Posted Comments