As UX professionals, we gather the information needed to make informed, data-driven decisions. We have a multitude of tools and research approaches to choose from, and knowing how to obtain valuable insights as accurately and efficiently as possible is a critical first step. One of the first decisions researchers must make is their methodological approach. Researchers must determine if the research questions or hypotheses would be better answered with quantitative or qualitative approaches.
Unfortunately, we at Usability Sciences find there is often confusion regarding the use and utility of quantitative research methods. This leads to common misconceptions, ranging from perceptions that quantitative research is too expensive or time-consuming, that it is too complex, to fundamental misunderstandings about quantitative research and how it relates to scientific inquiry and differs from qualitative approaches. These perceived limitations lead CX/UX professionals to shy away from incredibly valuable approaches. This piece is designed as an introduction for CX/UX professionals to equip them to make better research decisions.
Misconception #1: Quantitative and qualitative research are basically the same
Quantitative and qualitative research are not simply different flavorings of the same concept, they are entirely different philosophies. The differences in these research approaches go directly to the fundamental differences in how people consider and gain knowledge (the branch of philosophy that deals with the theory of knowledge is called Epistemology). What do you know, how do you know it, how can you be sure what you know is accurate? It is about how you understand and make sense of the world around you and what level of evidence is sufficient in answering the questions you pose.
There’s a brilliant article that goes into this topic in great detail, but very simply: There are three main research paradigms (philosophies):
Positivism – Positivism states there are objective natural phenomena that can be studied scientifically and known. Positivists tend to believe that there are “real”, objective truths, which are discovered by hypothesis and scientific evaluation. This philosophy encourages the use of quantitative methods.
Constructivism – Constructivism states natural phenomena are subjective and must be interpreted by an observer. Constructivists tend to believe that “truth” is relative, subject to our observations and interpretations. This philosophy encourages the use of qualitative methods.
(We’ll get to the third a bit later…)
It may be tempting to think of this as a trivial, academic distinction; certainly, we do not sit around discussing epistemology every day! But recognizing and having a general understanding of the philosophical underpinnings of the methodologies establishes a firm foundation for exploring and choosing the right approach.
Misconception #2: Quantitative research is scientific
While quantitative research certainly can be scientific, it is not inherently so. Let us look at the basic scientific method:
Step 1: Ask a question
Step 2: Do background research
Step 3: Construct a hypothesis
Step 4: Test your hypothesis
Step 5: Analyze your data
Step 6: Report your results
Steps three and four, constructing and testing the hypothesis, are the critical pieces that make a research process scientific. A hypothesis is a discrete, definitive statement about reality, as opposed to a question. It allows you to construct an approach that will provide incontrovertible evidence for or against that specific, pre-defined finding. A hypothesis is fundamentally different from a broader research question. Research questions are what are asked when you do not know something; a hypothesis is formed based on your existing research to assert what you suspect to be true.
A research question might be something like: “Why are customers abandoning my site before making a purchase?” Conducting background research may include qualitative research, like a usability test, and/or quantitative research, such as a survey asking people why they leave. This is exploratory research. From there, you may learn several things: people cannot find price, they do not like the look-and-feel, they do not trust the site, etc. You will not confidently know how these factors are interacting, but you can observe them.
From there you can establish formal hypotheses, such as “Increasing the size of the price displayed will increase conversion.” These hypotheses can be supported or disproven with scientific research. This is confirmatory research. This is typically when quantitative research becomes scientific.
Again, it may feel trivial to say, “the hypothesis has to be a statement instead of a question.” It may feel like a grade-school technicality, but consider the research paradigms; if you lean towards positivism and feel there is an objective truth out there, you want to validate it and verify it if possible. When you are working from a hypothesis, you can create an approach that will ensure you either provide evidence to substantiate it or disconfirm it. A scientific approach will rule out alternative explanations and isolate the individual effects of the variables tested.
Misconception #3: Quantitative research is highly complex and complicated
This misconception is often associated with the thought quantitative research implies scientific research. While scientific, confirmatory research does entail more complexity than exploratory research, neither necessarily must be particularly complex. Some simple descriptive quantitative statistics are among the simplest research approaches available, far simpler than usability testing. And, even scientific experimental designs can be simple (depending on the hypothesis and skill of the researcher) and conducted quickly and easily. Of course, some quantitative research can be highly complicated, but it is not always necessary.
A simple descriptive quantitative survey could be obtaining your System Usability Scale (SUS). This is an existing measure and only requires you build a survey, distribute it to your users, and analyze the results.
A simple scientific quantitative survey could be comparing your market segments or personas on the System Usability Scale (SUS) to see how the aggregate of each type of customer/user differs on their rating.
Scientific research becomes far more complex when our questions change from "are these variables related" to, "are these variables related given these other variables, and can we make predictions about people based on the relationship between the variables". The complex research is valuable but is needed less than simple correlation style studies.
Misconception #4: Quantitative research is more rigorous than qualitative
This is more of a misperception regarding qualitative research, but it is an easy mistake to make even (or especially) among CX/UX researchers. Qualitative research can, and often should, be just as rigorous as quantitative research. To an outside observer, it may look like a usability test, for example, is nothing more than a casual conversation or interview. That “casual conversation”, though, is the planned result of a well-trained researcher, adept at guiding the participant through the desired session flow, without influencing or leading the participant themselves. The researcher balances friendly casualness to keep the participant engaged with targeted and direct questions to elicit the desired opinions and reactions. Let alone all the rigorous recruiting, screening, research plan creation, and analysis that goes into these projects.
The distinction between quantitative and qualitative is not about the rigor of the research approach, rather it is about how representative and predictive the information obtained from the research is.
Qualitative researchers focus on transferability. Transferability is how well the findings can be transferred to other contexts. A good qualitative researcher conducting a usability test will identify how the user’s behavior is influenced by the non-natural testing process and how it may differ in a more normal (non-lab) environment.
Quantitative researchers focus on generalizability. Generalizability is how well the findings fit (i.e., are representative of) the population and settings. A good quantitative researcher conducting a system usability scale (SUS) analyses will ensure the sample obtained is representative of the population and reduces threats to internal validity while balancing ecological validity.
Misconception #5: One approach (quantitative or qualitative) is always better than the other
Let us return to our philosophical research paradigms. What happens if you feel like both positivism and constructivism have some merit? Well, there is a happy medium between these two philosophies called pragmatism.
Pragmatism – Pragmatists are all about practicality. If an approach provides a solution such that we can be reasonably confident the findings are accurate enough to proceed, we use the approach. Generally, a pragmatist uses the method that best fits the question(s) we have.
At Usability Sciences, we operate as pragmatists, and we believe other UX professionals should as well. A pragmatist will weigh the parameters (limitations) of a project to determine the best approach. The best approach is one that provides us reasonable confidence in the accurate generalizability of our research given, limitations like time, cost, sample size availability, and even the philosophical leanings of stakeholders (e.g., if stakeholders predominately prefer quantitative data (positivists), they may not respond well to qualitative (constructivist) research findings and suggestions, and vice versa). A pragmatic UX researcher will weigh the costs of different research methods against the value those different methods would provide, to determine the best research approach.
Consider this scenario: your team is in the early stages of working on a redesign to your e-commerce experience, and you want to validate the changes thus far and the overall direction. While you could conduct controlled quantitative research to determine the degree to which each component is impacting the user experience, you do not need to. Nor should you! Such an investment significantly outweighs the value answering those questions would provide. A qualitative usability test would provide you the insights you desire, allowing you to adjust direction if necessary, without a significant monetary or time investment.
Now consider some of the key learnings from your qualitative study were specific to unique features some users expected but were not included. Adding any of these features would incur significant development costs and may impact your launch window. From your qualitative research, you can be confident that some users want and need these features, but you cannot be certain how many, or how critical the inclusion of the feature is to be successful. Quantitative, scientific research would be able to tell you, with a high degree of confidence, which features are critical, and how much of an impact on your business the inclusion of those features would have. This allows you to make an informed development decision, confident that you can predict the potential value return on investment (ROI)from market opportunity and development cost.
Misconception #6: Quantitative research is prohibitively expensive and/or time-consuming
The price tag and timeline attached to quantitative research can vary dramatically, just like qualitative research. But it does not have to be expensive or time-consuming. This misconception often arises when attempting to compare quantitative and qualitative research solutions and apply them to the same problem. Again, though, these are fundamentally different research paradigms. Consider the qualitative and quantitative research solutions in the scenario above. Both would likely be relatively quick and inexpensive research projects. A quick qualitative usability test will provide clear insights to validate design direction of a massive and technically complex site. Similarly, a conjoint analysis, used to determine the optimal bundle of product or service offerings and price points, can offer efficient, predictive, actionable insights with a quick, well-crafted survey.
Flip them, try to answer those questions with the other methodology (qualitative <-> quantitative), and while technically doable, both projects would have huge timelines and costs. The research paradigms are fundamentally different, so a mismatch between the approach and the research question can lead to runaway costs. The approaches are so ridiculously different that trying to think about either solution using the other paradigm borders on non-sensical!
This is why pragmatism is so important within the CX/UX space; being able to understand the needs of the research and how best to gain the insights necessary leads to much more efficient and insightful research, without extraneous and unnecessary impacts to your budget and timeline.
In conclusion…
Do not limit yourself to one research approach; be pragmatic in how you answer your research questions or hypotheses. Quantitative and qualitative research are similar in that there is a great deal of complexity and nuance in the ways the approaches can vary based on the needs of an organization. Quantitative research is a valuable tool to have at your disposal, and we encourage you to slowly start adding it to your toolkit. By utilizing both qualitative and quantitative research methods, and taking a pragmatic approach to determine the best method for a given need, you can make research decisions that allow you to easily and efficiently gather the information needed to inform the decisions that drive success.