There is an erroneous perception in the UX community that if your method is qualitative, then numbers somehow cannot or should not be used.
These perceptions come from an informal practice that stems back to the beginning of the usability profession and continues through training programs and some UX experts.
Unfortunately, this perception is misguided and can prevent perfectly good data from being used to gain accurate views of the user experience.
Qualitative data can in fact be converted into quantitative measures even if it doesnt come from an experiment or from a large sample size.
The distinction between a qualitative study and a quantitative study is a false dichotomy. It doesnt cost more money to quantify or use statistics. It just takes some training and confidence - like any method or skill.
Here are five examples of how you can take common qualitative approaches to assess the user experience and convert them into numbers which can then be treated with a range of statistical procedures.
1.Converting a usability problem into a frequency
The quintessential usability activity is watching users attempt realistic tasks and identifying what in the interface is causing problems.
Simply categorize the problems, count the frequency, then use confidence intervals to estimate how common the problems likely are in the entire user population. For example, if 3 out of 11 users had a problem downloading the correct software product from a website, then we can be 95% confident at least 9% of all users would also have the problem (use the free web calculator or download the problem frequency calculator). It doesnt cost more money to generate those confidence intervals. This process also allows you to generate more accurate sample size estimates.
2. What problems are customers having?
It is sometimes difficult for customers to identify what they need in a product and where the shortfalls are. One effective approach is ethnographic research (a qualitative method), observing customers in their own setting encountering and solving problems.
Observe the problems customers encounter, categorize and count them. Then estimate the percent of all customers that likely share this behavior or problem to help prioritize product features. You can then estimate how many customers you need to visit based on the frequency of these issues.
3. Why is the product not being recommended?
When using the Net Promoter Score, it is valuable to ask open-ended, follow-up questions, especially for Detractors, such as, "Briefly describe why you gave the rating." Take the list of open-ended comments and group them into categories (content analysis). Count the occurrences, create a percentage of all comments, graph them and throw in some confidence intervals for good measure.
4. Why was that task so difficult?
If a user provides a low rating (below a 5), ask them to briefly explain why they gave a low rating. Take these open-ended comments, categorize them and add up the frequency in each group. This process can help you and your stakeholders make more informed decisions about the likely causes of the trouble.
5.Combining Net Promoter Scores and comments
A powerful way of making qualitative, open-ended comments more actionable is to combine them with a closed-ended question, like the Net Promoter Score. For example, quantify what users say they would improve on a website, then show what these customers Net Promoter Scores are
Quickly identifying the comments related to website navigation and product filters that are both high in frequency and come from users are likely to generate positive or negative word of mouth. In contrast, design/layout comments and advertisements while high in frequency appear to be minor issues for the users.
There are of course many software applications and websites which have never been exposed to any input from users. In such situations, there will likely be many obvious problems that just need to be fixed, regardless of how many users encounter the problem.
But once youve picked the low hanging fruit of a neglected interface, the benefits of structuring your activities and results lend themselves to quantification, where you can derive more meaning from your methods.
The advantage of converting qualitative data into quantitative data is that the source of qualitative data--a direct encounter of the users experience--can reveal nuances in usability, perhaps otherwise missed in more formal quantitative experiments and surveys.
Not only can qualitative data be categorized into quantities, but it can prompt further questions and discovery for usability improvement. This helps businesses understand their standing among their users.