Unlocking the Benefits of Qualitative vs Quantitative Research for UX

This article explores the differences between qualitative and quantitative UX research, with a focus on how these forms of research can be used together to gain an overall better understanding of user experience. We will discuss the advantages and disadvantages of each approach, examples of methods from both, as well as strategies for combining them.

Contents

• Intro to Qualitative Research
• Intro to Quantitative Research
• Advantages and Disadvantages of Both Research Approaches
• Combining Both for a Mixed Methods Approach to UX

Intro to Qualitative Research

Qualitative UX research focuses on collecting data about users’ behavior and motivations, such as emotions, opinions, and beliefs. Qualitative methods focus on understanding user behavior from research participants’ perspective. The data from these methods is not often measurable, but rather analyzed through themes. Qualitative research data is largely communication based, so results are usually in the form of text, audio, or video.

Examples of Qualitative Research Methods for UX

• User interviews
• Contextual inquiry (my personal favorite)
• Usability testing and prototype validation
• Ethnographic studies
• Landscape analysis
• Heuristic reviews
• Open-ended survey questions
• Diary studies
• Card sorting exercises with smaller groups, and especially moderated components.

Intro to Quantitative Research

Examples of Quantitative Research Methods for UX

• Experience Analytics like:
  • Bounce Rates
  • Time on Task
  • Goal success and conversion rates
  • Number of repeat and new visits to a view or page
• Analytics Frameworks such as HEART or AARRR
• Usage Analytics like:
  • Device usage
  • OS Platform and version usage
• A/B Split Testing
• Numerically scored or closed-ended questions in surveys, such as SUS
• Heatmap analysis for page view scrolling or click analysis.
• User flow analysis
• Card Sorting with large groups
• Tree Sort

Advantages and Disadvantages of Both Research Approaches

What are the Differences Between Qualitative and Quantitative Methods?

This is often the what vs why of the two groups of research methods. Quantitative methods often show what behaviors are happening, but not why, and qualitative methods discover the why people behave a certain way, but can be challenging to objectively discover the what.

Quantitative = The What

Quantitative and qualitative research are two distinct methods of gathering and analyzing data in user experience research (UXR). Quantitative research is based on numerical data and is not very useful until the quantity of participants increases, while qualitative research focuses on non-numerical data and seeks to understand the ‘why’ behind the numbers. Qualitative research does not need nearly as many participants. Quantitative research involves collecting numerical data through methods such as analytics, surveys, polls, or A/B split testing. This type of research can provide valuable insights into user behavior (the what), allowing UX designers to make informed decisions about product design. For example, a survey might ask users to rate their satisfaction with a product feature on a numeric scale. The results can then be used to identify areas for improvement.

Although quantitative research can provide very useful insights, it doesn’t inform on how to solve a problem, or why purchases are dropping, insights are limited by behavior we think to track, and what we can track. It really lacks the context around the data.

Some examples from my experience:

• A high rate of support requests around password recovery on a mobile application in an enterprise setting. The quantitative data showed the presence of a problem and where user drop off is in the password workflow, as well as the related support requests. The data also showed that this was a frequent and expensive problem that was worthwhile to prioritize. However, quantitative data could not show why users preferred to contact support over following the reset process. This was discovered through a handful of usability tests showing since users’ were on their personal devices, they couldn’t receive the reset email that was going to their work email. With just a small qualitative “study” the problem was quickly resolved.
• Android notifications being ignored. Our quant data showed that our enterprise users were rarely opening the Android app from our push notifications for priority alerts important to their work. A/B testing different messages didn’t significantly improve our open rates. During a contextual inquiry (a ride-along and qual study) for another project, we noticed that many of our users didn’t pay attention to any notification icons in the Android system tray. Many had 20+ notifications stacked up and ignored. When asked they didn’t even know what the notification icons indicated. Without the qualitative insights we wouldn’t have had a clue what was going on, and kept assuming it was a design issue on our end. This pattern was repeated in many iOS and Android native features. Our team gets excited by a new mobile feature, burn prod cycles supporting it and… crickets from users.

Many organizations fall into the trap of relying exclusively on quantitative data to inform their product strategy and user experience. Especially at high tech firms. Quantitative research is important and a powerful tool for UX, but it is severely limited when not supplemented by qualitative insights. Quantitative research is seen as faster and cheaper than performing qualitative research, which is an incorrect way to look at research practices in general. Quantitative data is only part of the picture to inform strategy, and not necessarily the biggest part. Determining the why from your quant data is an exercise in black box guessing your way to a solution.

Qualitative = The Why

Qualitative research is more exploratory in nature and seeks to uncover deeper insights into user behavior by looking at underlying motivations, emotions, and mental models (the why). This type of research often involves interviews and other moderated methods where participants are asked open-ended questions about their experiences with a product or service or researchers observe human behavior (contextual inquiry). Qualitative research can help product teams gain an understanding of how users think and feel about their products, which can be used to inform design decisions. More importantly, qual studies can be used to map the mental models of users to goals of the business. Users in different industries, countries, regions, and roles all think differently about their workflow and goals. Quantitative research is poor at uncovering these mental models, but well targeted and relatively small qualitative studies can easily and quickly uncover them.

Some examples from my experience using qualitative research methods in the past:

• Contextual inquiry: Observing paramedics simulate caring for a stroke patient in their ambulance. My team was surrounded by excellent in-house subject matter experts (SMEs), but observing one two-way radio call outside in the weather while caring for a critically injured person highlighted many UX challenges we never understood from our SMEs.
• Moderated Testing Prototypes: Early in my UX journey, about 2008 I had the opportunity to attend a conference my client put on for their customers. I opted to setup a curtained off testing room to have volunteers run through prototypes of a new feature pushed by some senior execs. I had no expectations what current customers would think, but had assumed the execs had a basis for their assumptions. In a 4 hour afternoon and about 2 hours of designing the prototype, I collected enough feedback (mostly negative) that the exec abandoned their push for this feature without any argument. A feature that would’ve took my team about 2 months to implement. Assumptions like this are something I’ve seen frequently in the course of my career (20+ years in software and UX engineering). It is what I think about anytime someone says qualitative research is expensive. Not doing user research is more expensive.

Even with the power of qualitative research, there are weaknesses. Qualitative research is definitely an active pursuit, not a passive one. You will always find out new and interesting insights from your users. However, without some quantitative data to inform your research plan you may not discover the most important insights you need at the moment. Meaning, qualitative research by itself can be poor to objectively discover problems in your user experience. It is difficult to not zero in on one off-hand critique by a single or small group of outspoken users. That’s not necessarily bad data, but should be paired with data that is at least close to being statistically significant.

Overall, quantitative and qualitative research are both important tools for product teams as they provide different types of information that can be used together to eliminate risky assumptions, or improve existing products.

Combining Both for a Mixed Methods Approach to UX

What is Mixed Method Research?

Specifically, mixed methods research is a type of research that combines both quantitative and qualitative data collection and analysis techniques. As I have heavily hinted at in the disadvantages of quant and qual research above, often for UX research the best approach is a mixed methods approach. In the context of UX research, mixed methods research allows researchers to gain a deeper understanding of user behavior and preferences by collecting both numerical data (e.g., click-through rates, time spent on page) and subjective feedback (e.g., user opinions, experiences).

Even in teams with specialized skill sets it is important for researchers to understand how data is collected and analyzed from both types of research. One, so you can objectively understand the quality of any conclusions before adding them to your own research. Two, by adding even basic quantitative or qualitative skills to your core competencies you can speed up your research planning, and even be able to do research without waiting on other teams. It is very common that a data science team may be a roadblock to quantitative data for a UX research project.

Mixed methods research involves three main stages: design, data collection, and analysis. During the design stage, researchers determine which quantitative and qualitative methods they will use to answer their research questions. This may involve surveys or questionnaires to gather quantitative data, as well as interviews or focus groups to collect qualitative data. In the data collection stage, researchers collect both types of data simultaneously. For example, they may conduct user testing sessions with participants while also collecting demographic information through surveys. Finally, during the analysis stage, researchers integrate the quantitative and qualitative data to gain a comprehensive understanding of user behavior. This may involve using statistical techniques to identify patterns in the quantitative data while also analyzing interview transcripts for relevant themes.

When would you use Mixed Methods in UX Research?

Really, you use mixed methods when it fits your research plan and will answer your research questions. Mixed methods research involves the gathering of both open-ended and close-ended data (ie, qualitative and quantitative), methods from each, as well as the combination of the research data sets into a single analysis. When set up properly, this approach provides the missing context around numerical data.

Some good rules of thumb when to use a mixed methods approach are:

• When it limits the weaknesses of a singular approach.
• When the researcher has access and can collect both types of data to answer their research questions. Can you track parallel variables in both types of data?
• When an organization’s culture leans too heavily or exclusively towards quantitative or qualitative research. Mixed methods can help with buy-in for other kinds of research by introducing holdouts.
• You need better context around analytics.
• You need more perspectives to invalidate/validate assumptions.
• You need to explain numerical analysis in depth for stakeholders.

Mixed methods is a large and extensive topic that we will be covering in depth in other articles, along with techniques for quantitative and qualitative research.

Conclusion

In conclusion, there are many differences between quantitative and qualitative methods for UX research that contribute to their effectiveness in different scenarios. While quantitative studies allows us to empirically measure large groups of people, qualitative studies give us deeper glimpses into the mental models of a smaller set of participants. Both have their advantages and disadvantages, which is why it’s important to consider both when conducting research. In many cases, a combination of quantitative and qualitative research into a mixed methods approach can provide some of the most robust results for understanding user experience issues. Additionally, this strategy can also add depth to quantitative results by grounding them within the context of the participant's experiences. Choosing which type of research methodology to employ depends on what question you’re asking – is there a description to be made or you need a more analytical approach? When utilizing either single methods or combined, success largely comes down to trusting the process and designing an efficient system to obtain valid results.

References & Further Reading

• Research Design: Qualitative, Quantitative, and Mixed Methods Approaches by John W. Creswell, J. David Creswell
• UX Research by Brad Nunnally, David Farkas
• Design patterns for mixed-method research in HCI