All too often, PMs follow this line of thought: idea first, then validate with data later. But when PMs use customer research data as a way to support, approve, back up, or validate their presuppositions, the integrity of that data comes into question.
Of course, plenty of worthwhile ideas come to us in the middle of the night based on no existing data. When that’s the case, it’s still worth turning to customer research to investigate the idea. But to make the most of user research, you must approach it with curiosity, not just to secure a “data-backed” thumbs-up.
PMs must be wary of confirmation bias clouding their user research. When this happens, they become blind to unexpected insights that would enhance their customer understanding and, by extension, help them make better products.
Here, we’ll spell out some of the signals that show confirmation bias is polluting user research and offer some tips to bias-proof your research process. But first, let’s look the beast straight in the eyes: confirmation bias.
Confirmation bias is a natural trap
Confirmation bias describes the natural tendency to interpret information in a way that’s aligned with preconceived notions. In other words, it’s when our held beliefs persuade how we search for and analyze information.
In research, even the most prestigious scientists fall victim to two major categories of confirmation bias: biased collection and biased interpretation.
- Biased collection: A researcher uses research methods that will skew data to validate their hypothesis (i.e., they only survey individuals who are more likely to confirm their hypothesis)
- Biased interpretation: A researcher puts lopsided importance on certain data in order to validate their hypothesis (i.e., they only notice or feature survey results that confirm their hypothesis)
Confirmation bias isn’t exclusive to research, either; it crops up in our personal lives. For instance, it’s been suggested that it’s a major factor for political polarization: a 2016 study found that Facebook users preferred reading news that confirmed their worldview and avoided news that didn’t.
Warren Buffet’s success is credited, in part, to his ability to sidestep confirmation bias. He sums up the phenomenon well by saying, “What the human being is best at doing is interpreting all new information so that their prior conclusions remain intact.”
In fact, Buffet is spot-on when he describes this as an innate issue for human beings. Confirmation bias is an age-old phenomenon that’s programmed into our brains. It stems from the posterior medial prefrontal cortex (the pMFC), a decision-making center of the brain. The pMFC is more sensitive to your own opinions than the opinions of others, which is what causes a consistent natural bias.
Now, how does confirmation bias impact customer research, in particular?
5 ways confirmation bias creeps into user research
Here are five common ways PMs fall into the trap of confirmation bias during user research:
- You cherry-pick certain statistics and pretend you didn’t see others. Don’t approach user research like an opportunity to shop for affirming stats. In the long run, you’ll do yourself a favor by collecting, analyzing, and responding to a complete picture of customer feedback, even (especially!) if that includes some surprises.
- You stop research as soon as you find a key insight. This is another form of cherry-picking data. Again, a robust set of user research data offers you much more than a simple thumbs-up or a simple thumbs-down, so don’t pull the plug as soon as you hear an answer you like.
- You use leading/close-ended questions during customer surveys. When it comes to survey writing, small decisions (like question order and word choice) can make a big difference. Check out our practical guide to avoiding survey bias for more information and advice.
- You use a small or skewed sample size. Just because one demographic might love your idea doesn’t mean they make a good survey group. If you’re trying to gauge sentiment about the most popular film franchise of all time, you probably shouldn’t conduct a survey at a Star Wars fan convention.
- You gather data out of context. Sometimes, PMs gather skewed data by surveying customers out of context (i.e., when they’re not using the software or product). Customers are particularly susceptible to giving you skewed responses — usually, the responses they think you want to hear — when they don’t have hands-on access to the product in question.
“What the human being is best at doing is interpreting all new information so that their prior conclusions remain intact.”
PMs, let’s be clear: you should strive for data-backed decision-making. Strong customer research data helps you align product development with customer needs and business goals. Plus, it can often make a difference when it comes to budget allocation. But if you’re using data to defend decisions you’ve already made, you might be shooting yourself in the foot.
Data collected to validate an existing idea could steer you to launch a feature that nobody uses or one that doesn’t meet business goals. Plus, approaching research with the sole aim of validating one idea could blind you from improving it or discovering an even better idea altogether.
3 tips to help you steer clear of confirmation bias
Confirmation bias might be a natural instinct and a common trap, but we’re not powerless to prevent it. PMs: circle, underline, and highlight these tips to ensure you don’t let confirmation bias pollute your research.
1. Use research to surface new insights, not validate old ones
If you set out to find data to confirm you’re doing the right thing, you’ll find it. But if that’s your approach, you won’t see the full benefits of strong customer research: discovering nuance, sparking new ideas, and identifying potential problems before they drag you down.
Instead, use data as a tool to prompt new questions and spark new ideas. Approach customer research with a curious mindset about what new questions might arise. Trust yourself to turn those questions — even if they weren’t what you expected — into insight and action in the long run.
2. Make research continuous, not one and done
A single round of customer research might be enough to “data-back” a hypothesis, but it doesn’t capitalize on all the potential information great user research can provide. Plus, that single round of research won’t keep up with new developments in your product and market.
Instead, make research a continuous part of your product development process. If you establish research to be a regular part of the process, you’ll enjoy a stream of up-to-date, raw data that tells you to hold steady, pivot slightly, or try new directions altogether.
3. Include others in data interpretation
Numbers might seem objective, but collecting and interpreting them is not. If one single person is defining research methods, collecting data, and interpreting results, you’re far more susceptible to bias.
Instead, try and diversify your data interpretation; don’t let the decision-maker be the sole interpreter. For many teams — especially those that don’t have the bandwidth or user research expertise — AI Analysis can help you turn raw data into insights without the potential for human error.
User research should spark new ideas, not validate existing ones
The best user research starts conversations; it doesn’t end them.
All too often, PMs undercut the value of customer research by mining for data that appears to validate their own hypothesis, be it for themselves, investors, or a stakeholder with power over the budget. And while user research can and should embolden product teams that are on the right track, that isn’t the end-all-and-be-all of user research.
At its best, research helps you spot issues ahead of time, consider new nuances, and better align your product with customer experience and needs. But, like anything, it’s all in the execution.
Want to learn more about where user research goes wrong and how to make it right? Check out The 5 Mistakes that are Ruining Your Survey Data.