Designing Effective Research Designs for User Studies: A Guide to Mixed Methods Approaches

Mixed methods research is a research approach that combines quantitative and qualitative research methods. This approach is used to collect and analyze both types of data in order to gain a more complete understanding of a research problem.

Quantitative research methods are used to collect numerical data that can be analyzed using statistical methods. This type of data is often used to describe the frequency, distribution, and relationships between variables. Qualitative research methods are used to collect non-numerical data, such as text, images, and audio recordings. This type of data is often used to understand the meaning and context of human experiences.

There are many different types of mixed methods research designs. Some of the most common designs include:

• Convergent mixed methods research: This design involves collecting both quantitative and qualitative data and then integrating the data to answer the research question.
• Explanatory sequential mixed methods research: This design involves collecting quantitative data first, followed by qualitative data. The qualitative data is then used to explain the quantitative findings.
• Exploratory sequential mixed methods research: This design involves collecting qualitative data first, followed by quantitative data. The quantitative data is then used to test the findings of the qualitative data.

Mixed methods research can be a powerful tool for gaining a more complete understanding of complex research problems. This approach is increasingly being used in a variety of fields, including education, health care, and social work.

Using Mixed Methods Research in UX

There are many reasons why you might want to use mixed methods research in UX. Here are a few:

• To get a more complete picture of user needs and behaviors. Quantitative research can be helpful for getting a general sense of how users are interacting with a product or service, but it can't tell you why they're doing what they're doing. Qualitative research, on the other hand, can provide you with insights into the user's thoughts, feelings, and motivations. By combining quantitative and qualitative data, you can get a more complete picture of how users are experiencing your product or service.
• To identify and address user pain points. By understanding users' needs and behaviors, UX designers can identify and address the pain points that users experience when interacting with a product or service. This can lead to the design of more user-friendly and effective products and services. Quant data helps find potential problems and pain points that affect a large number of users while qual data will help narrow in on what the user was trying to accomplish when encountering the pain point.
• To identify and address research biases. All research is biased to some extent. Quantitative research can be biased by the way that the data is collected and analyzed, and qualitative research can be biased by the researcher's own interpretation of the data. By combining quantitative and qualitative data, you can help to identify and address these biases. [See below]
• To improve the usability of products and services. Mixed methods research can be used to test the usability of products and services. Again, quantitative data like analytics or logs can help identify when and where problems are occurring. Then using qualitative methods you can know more about the context and specifics of the usability problem. This data can be used to identify and fix usability problems, which of course leads to better user experiences.
• To make better design decisions. By understanding users' needs, behaviors, and pain points, product teams can make better design decisions. This can lead to the design of products and services that are more user-friendly, effective, and engaging.
• To enhance the credibility of your research findings. Unfortunately in tech organizations, quantitative research is often seen as being more credible than qualitative research, but this is not always the case. Qualitative research can provide you with rich insights into user experiences that quantitative research cannot. By combining quantitative and qualitative data, you can enhance the credibility of your research findings with skeptical stakeholders.
• It can help to develop new theories and explanations. Quantitative research can be helpful for testing existing theories, while qualitative research can be helpful for developing new theories. By combining the two methods, researchers can develop new theories and explanations that are based on both quantitative and qualitative evidence.

If you're looking for a way to get a more complete and credible understanding of user needs and behaviors, then mixed methods research is a great option.

A Note about Research Bias

There are a few ways that combining qualitative and quantitative data can help to identify and address research bias.

• By providing a more complete picture of the research problem. Quantitative research can be helpful for getting a general sense of how users are interacting with a product or service, but it can't tell you why they're doing what they're doing. Qualitative research, on the other hand, can provide you with insights into the user's thoughts, feelings, and motivations. By combining quantitative and qualitative data, you can get a more complete picture of how users are experiencing your product or service. This can help you to identify and address any biases that may be present in your research.
• By providing a way to cross-check the results of different data collection methods. Quantitative and qualitative data collection methods can be biased in different ways. By combining these methods, you can cross-check the results of each method and identify any potential biases. For example, if you conduct a survey and find that 75% of users are satisfied with your product, you can follow up with qualitative interviews to get a deeper understanding of why users are satisfied. This can help you to identify any potential biases in your survey data.
• By providing a way to triangulate the results of your research. Triangulation is a research method that involves using multiple data collection methods to study the same research problem. By triangulating your research, you can get a more reliable and accurate understanding of the research problem. For example, if you conduct a survey, interviews, and usability testing to study how users interact with your product, you can use the results of these methods to triangulate your findings. This can help you to identify and address any potential biases in your research.

Overall, combining qualitative and quantitative data can be a powerful way to identify and prevent research bias. By using a mixed methods research approach, you can get a more complete and credible understanding of the research problem.

Types of Mixed Methods Research

Convergent Research Design

Convergent mixed methods research is a research design in which both quantitative and qualitative data are collected and analyzed concurrently, and the results of the two data sets are integrated to answer the research question. This type of research design is often used when the researcher wants to gain a more complete understanding of a research problem by using the strengths of both quantitative and qualitative data.

For example, a researcher might use convergent mixed methods research to study the impact of a new educational intervention on student achievement. The researcher might collect quantitative data on student achievement scores, as well as qualitative data on student and teacher perceptions of the intervention. The researcher would then integrate the results of the two data sets to answer the research question of whether the intervention had a positive impact on student achievement.

Explanatory Sequential Research Design

Explanatory sequential mixed methods research is a research design in which quantitative data is collected and analyzed first, followed by qualitative data. The qualitative data is then used to explain or elaborate on the quantitative findings. This type of research design is often used when the researcher wants to gain a deeper understanding of the quantitative findings or to find potential problems with the quant data.

For example, a researcher might use explanatory sequential mixed methods research to study the factors that contribute to student achievement. The researcher might collect quantitative data on student achievement scores, as well as qualitative data on student and teacher perceptions of student achievement. The researcher would then use the qualitative data to explain why some students had different achievement scores than others.

Reference

• Using Mixed-Methods Sequential Explanatory Design: From Theory to Practice by Nataliya V. Ivankova, John W. Creswell, and Sheldon L. Stick

Exploratory Sequential Research Design

Exploratory sequential mixed methods research design is a type of mixed methods research design in which qualitative data is collected and analyzed first, followed by quantitative data. The quantitative data is then used to test the findings of the qualitative data. This type of research design is often used when the researcher wants to gain a deeper understanding of the qualitative findings by using the strengths of quantitative data.

For example, a researcher might use exploratory sequential research design to study the experiences of students who have been bullied. The researcher might first conduct qualitative interviews with students who have been bullied to understand their experiences. The researcher might then collect quantitative data on the prevalence of bullying and the impact of bullying on students. The researcher would then use the findings of the quantitative data to test the findings of the qualitative data.

• It can help to deepen the understanding of the qualitative findings by providing quantitative data to support them.
• It can help to validate the qualitative findings by providing additional evidence to support them.
• It can help to identify any potential biases in the qualitative findings.
• It can help to identify new research questions and hypotheses.
• It can help to develop new research instruments.
• It can help to refine the findings of quantitative research.

Reference

• A sequential mixed model research design: Design, analytical and display issues by Roslyn Cameron

Challenges of Mixed Methods Research

We covered why you should use mixed methods and the benefits, but there are some challenges. There are a number of challenges to using a mixed methods research design for product design teams. Here are a few:

• It can be time-consuming and expensive. Mixed methods research can be more time-consuming and expensive than using a single research method. This is because it requires collecting and analyzing both quantitative and qualitative data.
• It can be difficult to integrate the qualitative and quantitative data. Qualitative and quantitative data are collected and analyzed in different ways. This can make it difficult to integrate the two types of data in a meaningful way.
• It can be difficult to get buy-in from stakeholders. Mixed methods research can be more complex than using a single research method. This can make it more difficult to get buy-in from stakeholders, who may not be familiar with this type of research.

Despite these challenges, mixed methods research can be a valuable tool for product design teams. By combining quantitative and qualitative data, product design teams can gain a more complete and credible understanding of user needs and wants. This can help them to design products that are more user-friendly and successful.

Here are some tips for overcoming the challenges of using a mixed methods research design for product design teams:

• Start by clearly defining your research questions. This will help you to determine which methods are best suited for answering your questions.
• Start with sequential research designs instead of convergent. I found these the easiest to plan in my experience when starting out. Usually I would identify some data from current research or sources like analytics or customer support tickets and need to answer questions and validate/invalidate assumptions the product team was developing.
• Start small. Just like any other project or problem, if you can, start with a smaller one, or a smaller piece of the puzzle. You can still tackle the riskiest assumptions, but make sure the data is easier to collect.
• Work with someone who is experienced with the research type you aren’t. This will help you to integrate the data and make sense of it and avoid making mistakes early on. If you primarily do qualitative research, get help from your data science team for the quantitative part of the project.
• Be prepared to spend time and money on collecting and analyzing the data. Mixed methods research can be more time-consuming and expensive than using a single research method. This is another reason to start small.
• Be patient. It takes time to develop a deep understanding of your users and their needs. Mixed methods research can help you to gain this understanding, but it takes time and effort. This is also a good reason to think about a way to store your research data and insights. Whether it is a spreadsheet or a full on research repository.

How to Plan a Mixed Methods Research Project

Here are some good steps for planning a mixed method approach:

1. Define Your Research Questions

What do you want to know about your software design? What are your specific research questions?

• Start with a general question. What do you want to know about your software design?
• Narrow down your question. What are the specific aspects of your software design that you want to know about?
• Make sure your question is answerable. Can you collect data that will answer your question?
• Make sure your question is relevant. Is your question important to your research?
• Make sure your question is feasible. Do you have the resources to answer your question?

It is important to note that research questions are not set in stone. As you conduct your research, you may find that your questions need to be modified or even replaced. This is perfectly normal. The important thing is to start with a clear idea of what you want to know and then be flexible enough to adapt your questions as needed.

2. Choose Your Research Methods

What methods will you use to collect and analyze your data? What quantitative and qualitative methods? How does the data collected from each fit together?

Some examples combinations that work well:

• Quantitative surveys and qualitative interviews: Quantitative surveys can be used to collect data on a large number of people, while qualitative interviews can be used to get in-depth insights from a smaller number of people.
• Quantitative experiments and qualitative observations: Quantitative experiments can be used to test the effects of different variables, while qualitative observations can be used to capture the lived experiences of people.
• Quantitative content analysis and qualitative discourse analysis: Quantitative content analysis can be used to identify patterns in large amounts of text, while qualitative discourse analysis can be used to explore the meaning of text.

3. Justify Your Research Design

Here are some good approaches to justifying your chosen research design:

• Start by explaining the purpose of your research. What do you hope to achieve by conducting your research? Why is it important? How does this match project and business objectives?
• Explain the advantages of using a mixed methods approach. What can you learn from using both quantitative and qualitative methods? How will this help you to achieve your research goals?
• Address any potential concerns that stakeholders may have. For example, some stakeholders may be concerned that a mixed methods approach will be too time-consuming or expensive. Be prepared to address these concerns and explain why a mixed methods approach is the best way to achieve your research goals.
• Be clear about your research methods. What data will you collect? How will you analyze your data? How will you ensure the reliability and validity of your findings?
• Be open to feedback and be prepared to answer questions. Stakeholders may have valuable insights that can help you to improve your research design. Be open to their feedback and be willing to make changes as needed. This is an opportunity to build trust and credibility with stakeholders.

Here are some additional tips for justifying your chosen research design to stakeholders:

• Use clear and concise language. Stakeholders may not be familiar with the technical jargon of research, so it's important to use language that they can understand.
• Use examples to illustrate your points. If you can provide examples of how a mixed methods approach was used successfully in other research projects, this will help to make your case. If you don’t have direct examples, give examples of the type of problems your approach prevents or questions it answers.
• Be persuasive. Ultimately, you need to convince stakeholders that your research approach is the best way to achieve your research goals, whether it’s mixed methods or a singular approach. It’s fair for stakeholders to need persuasion, they have a vested interest in making projects succeed as well, be confident in your research design and be prepared to answer any questions that stakeholders may have.

4. Collect Your Data

Administer your surveys, conduct your interviews, or collect your other data.

5. Analyze Your Data

How do your quantitative and qualitative findings relate to each other? What do they tell you about your software design? Here are some suggestions for analyzing data in mixed methods research designs:

• Use a mixed methods data analysis framework. There are a number of different mixed methods data analysis frameworks available. These frameworks can help you to organize your data and to identify patterns and themes.
• Consider the strengths and limitations of each data type. Quantitative data can be used to test hypotheses and to make generalizations, while qualitative data can be used to explore experiences and to develop theories. When you are analyzing your data, it is important to consider the strengths and limitations of each data type.
• Integrate your findings. The goal of mixed methods research is to integrate the findings from the quantitative and qualitative data. This can be done by comparing and contrasting the findings, or by using the findings from one data type to inform the interpretation of the other data type.
• Be transparent about your data analysis process. It is important to be transparent about your data analysis process. This means documenting your decisions and explaining your reasoning. By being transparent, you will increase the credibility of your research.
• Use a variety of data analysis techniques. There is no one-size-fits-all approach to data analysis in mixed methods research. The best approach will vary depending on the research questions, the data collection methods, and the data analysis framework.
• Be flexible and adaptable. The data analysis process is often iterative. This means that you may need to go back and forth between the quantitative and qualitative data, and that you may need to modify your data analysis plan as you learn more about your data.
• Get feedback from others. It is helpful to get feedback from others on your data analysis plan and on your findings. This can help you to identify any potential biases or errors in your analysis.

We’ll go into more detail in another article about analyzing research results from different designs.

6. Integrate Your Findings

How do your quantitative and qualitative findings relate to each other? What do they tell you about your software design?

• **There are multiple ways to integrate results. **Side-by-side comparison, data transformation or data merging, and joint display are common approaches.
• Start by identifying the similarities and differences between the two sets of findings. What do the quantitative and qualitative findings have in common? What are the key differences?
• Use the similarities and differences to develop a more comprehensive understanding of the research question. The quantitative findings may provide general insights, while the qualitative findings may provide more in-depth insights. By integrating the two sets of findings, you can develop a more complete understanding of the research question.
• Be careful not to overstate the level of agreement between the two sets of findings. It is important to remember that quantitative and qualitative methods are different, and they may produce different results. Do not assume that the two sets of findings are completely consistent.

7. Write your research report/share your findings

Summarize your research questions, methods, findings, and insights. Some tips for sharing your findings:

• Know your audience. Who are you writing for? What are their interests? What do they already know about your topic? Tailor your summary to your audience's level of knowledge and interests.
• Be clear and concise. Get to the point quickly and avoid jargon. Use plain language that everyone can understand.
• Highlight the key findings. What are the most important things that you want your audience to know? Focus on the findings that are most relevant to them.
• Use visuals. Visuals can help to make your summary more engaging and easier to understand. Use charts, graphs, and images to illustrate your points.
• Be objective. Avoid stating your own opinions or interpretations. Stick to the facts and let your audience draw their own conclusions.
• Use citations. If you are summarizing someone else's work, be sure to cite your sources. This will give your summary credibility and help your audience to find the original sources if they want to learn more.
• Proofread carefully. Before you share your summary, be sure to proofread it carefully for errors in grammar, spelling, and punctuation. A well-written summary will make a good impression on your audience and help them to understand your research findings.
• Share your summary in a variety of formats. You can share your summary in a research paper, a blog post, a presentation, or even a social media post. The format you choose will depend on your audience and your goals.
• Get feedback from others. Ask friends, colleagues, or mentors to read your summary and give you feedback. This can help you to identify any areas that need improvement.
• Be patient. Sharing your research findings can take time. Don't get discouraged if you don't get immediate results. Just keep sharing your work and eventually you will find an audience that is interested in what you have to say.

It is important to note that these are just general steps. A main advantage of the mixed methods approach is flexibility and a larger set of research methods. The specific steps you take will vary depending on the specific research questions you are asking and the methods you are using.

Conclusion

In conclusion, mixed methods research is a valuable tool for researchers who want to gain a more comprehensive understanding of their research topic. By combining the strengths of both quantitative and qualitative methods, mixed methods research can provide a more complete and nuanced understanding of the research question.

Mixed methods research is a powerful tool that can be used to gain a deeper understanding of user needs and behaviors. When used effectively, mixed methods research can help to improve the design and usability of products and services.

When conducting mixed methods research, it is important to carefully consider the research questions, the data collection methods, and the data analysis methods. It is also important to be transparent about the research process and to be respectful of different perspectives. By following these tips, researchers can increase the rigor and credibility of their mixed methods research.