This blog post is based on a webinar Sprig hosted with Noon Academy. If you prefer to watch the full webinar, check it out here!
What’s an example of recall bias?
"Describe a time when…”
It’s a question you may ask your users frequently to understand what they were thinking or experiencing when they were interacting with a particular feature or during a specific event.
But turn the tables on yourself for a second. Do you remember how efficient the process was when you signed up for that grocery delivery service two weeks ago? Or whether you were able to easily register for a fitness class on Monday?
Not so easy, right? It’s impossible to perfectly recreate an experience at a later date, and even more importantly, users usually cannot recall why they made a specific decision. This is a perfect example of recall bias.
What is Recall Bias?
According to N/N group, recall is the retrieval of related details from memory. Three key factors influence recall:
- Practice: How many times something has been used — the more you do something, the more likely you are to remember it
- Recency: How recently something has been used — you’re more likely to remember the most recent events
- Context: What is present in the person’s focus of attention — is there something that helps activate a memory through association or cues?
Recall bias occurs when you’re specifically missing the recency piece. Additionally, users may be influenced by experiences that took place afterwards.
Let’s say you ask a user about a particular feature. They mention they had a hard time figuring it out, but when asked where they ran into trouble, they can’t remember specifics. They answer, “I just remember I had to go to the help center, and then I talked to my coworker, and we managed to figure it out, but it was annoying.”
Why is that?
Your users have a hundred different things running through their mind when they click into your app or interact with a feature. Their experience with your product could be secondary to any number of things on their long to-do list, and that leads to forgetting specific details and actions days or even moments later. This also comes into effect with the peak-end rule, where users only remember the intense peaks of an experience, as well as the end. Their recall isn’t perfect, and recall bias is in effect.
Here’s When and Why Recall Bias Matters
Because of recall bias, the feedback you get will likely not be as specific or actionable. It may not uncover the root of your users’ problems or reveal opportunities for improvement.
Let’s look at a few examples of recall bias.
In the first example, there is an educational platform that relies on user-generated content to provide answers to submitted questions. In response, users are able to cast upvotes and downvotes, providing trust and legitimacy to the answers. But the platform wasn’t sure how users were using the upvotes and downvotes.
Some users downvoted because they disliked the answer. For example, the answer said cats only like their owners because they feed them, and the user, as a cat lover, doesn’t like that answer, so they downvoted. In other cases, they downvoted because they didn’t like the topic. For example, the question and answer were related to whether raw shrimp are safe to consume and the user doesn’t like shrimp.
In both of these cases, it’s a quick, in-the-moment reaction. A week later, they may not remember why they downvoted. So, if the platform is interested in finding out what motivates an upvote or a downvote, they need to ask why as soon as the user votes.
In the second example, a real estate tech company wants to understand what users are experiencing in their onboarding flow. They were seeing a drop-off towards the end, but needed more information to determine why. Asking them several days later wouldn’t arrive at a detailed or actionable answer — the user may not remember a small annoyance that caused them to navigate away. Instead, they needed to ask users right after they clicked away — that’s what revealed that users were dropping off because they weren’t far enough in the discovery process to provide contact info that was requested.
Why has Recall Bias Continued to Persist?
Recall bias is an intuitive concept to understand. Our memories aren’t steel traps, and neither are our users. So why does it continue to be an issue?
The biggest obstacle to reducing recall bias is the lack of an in-product research tool. Some organizations aren’t aware there’s an option, or they’ve been limited by a lack of effective tools and instead rely on building this functionality in-house (which comes with its own set of challenges).
You need to work hard to reduce recall bias, and not all companies have dedicated user research teams to address it head on. Additionally, advanced research methods like diary studies and contextual inquiry are not always easy or available options. In-product surveys are the fastest and easiest way to reduce recall bias.
Three Ways to Reduce Recall Bias
When you think about ways to provide context and reduce recall bias, you might immediately think of qualitative research methods, but there are also ways to get quick, scalable, accurate feedback while a user’s memory is fresh. Let’s talk through each of these, and compare the pros and cons.
Contextual Inquiry
A contextual inquiry means going to where your users are to see their real-time, behind-the scenes, first-hand experience. For example, if you are an e-commerce platform researching sellers’ shipping processes, you could go to their offices and follow them around during a typical day, researching how they ship an order or take a specific action.
Pros of Contextual Inquiries:
- Real-time
- First-hand experience
Cons of Contextual Inquiries:
- Time intensive, resource intensive
- The action a researcher wants to see may not come up that day
Contextual Diary Studies
With contextual diary studies, users commit for a specific duration of time and during that time, take a series of tests or activities that are related to the context they’re going through at that moment in their user journey. These are not to-the-minute, but they should be timed closely based on the user’s actions.
Pros of Contextual Diary Studies:
- Can get more depth and qualitative insights
- Uncover emotions, who they’re talking to, and a lot of detail on what’s going on in their relationship with the product
Cons of Contextual Diary Studies:
- Not at scale and not in the moment the same way in-product is
- Not quite as granular, specifically around small, quick actions
- Can’t always tie thoughts to specific actions
- Have to hope they’re going through the user journey at the pace you assume they’re going through
In-Product Surveys
In-product surveys are launched, as you might assume, in-product, capturing users in-the-moment as they’re completing a specific action (though placed at the right time to avoid disrupting the flow!). These are designed to ask the right questions to the right users at the right time to understand the user experience as it’s happening.
In-product surveys help reduce recall bias by asking targeted questions when tasks are completed. By asking the user the question right after they’ve taken an action, there is little room for error in recall.
Additionally, in-product surveys allow you to capture context that's not possible with other out-of-context methods. With these surveys, you can run research that is tactical or strategic, generative or evaluative, provide quick insights or company-wide metrics, or rapidly capture quantitative insights and/or validate qualitative hypotheses.
Pros of In-Product Surveys:
- Understand what’s happening at the time of action
- Surprising amount of granularity from well-placed open-ended questions
- Scale means you can capture a lot of data in a short amount of time
- Ability to add a recruiting prompt to the end of your in-product survey to ask follow-up questions to a user!
Cons of In-Product Surveys:
- You can’t go quite as deep as you could in-person
- Short and sweet questions perform best — however, you can ask for their email to follow up later or ask open-ended questions and analyze using AI analysis
How In-Product Surveys Tackle Recall Bias
In our recent webinar, Ale Argenio and Garrett Gronzko from Noon Academy joined Sprig to share how they’ve used Sprig’s in-product surveys to reduce recall bias and make changes in their users' experience. Here are two examples from their recent work.
Example One: Homepage Research
When Noon Academy set out to conduct research on their homepage experience, they wanted to tap into their users’ reactions in-the-moment using in-product surveys.
Website pages are one of the hardest things to research, especially if you have a lot of new users. This is an example of when “practice,” one of the three tenants of recall that we described earlier, is low. Users have not used the website before, so they're likely not going to remember as much from the experience later on.
To avoid interrupting the experience, they targeted users as they were about to exit the app, and specifically targeted a nuanced group of users.
They asked users why they came to Noon Academy, what’s working, how they’re using it, and whether or not the current version of the app is working for their needs.
The results showed that the Noon Academy app was delivering on its mission of having live classes right in front of students, but it was lagging in showing groups for its students to join. This gave the research team more questions to dig into and revealed opportunities for improvement in the experience. Had they asked students via an email survey days after their in-app experience, their results could have been skewed by recall bias. A student may have remembered there was something missing, but not quite able to recall exactly what that was.
Example Two: Feature Experience
Noon Academy also wanted to understand whether students were satisfied with the class session experience. This is an example of when recency is important. That’s because right after students complete the class, their recall will be highest, and it will decline over time.
For this in-product survey, they targeted new users of the app in grades 10 to 12 as they exited their class session.
When they asked how satisfied students were with their session on a scale of 1-5, 34% rated it a 1, or "not at all satisfied," and 6% rated it a 2. Clearly, there was room for opportunity.
The research team dove in with an open-text follow-up question for those who gave the app a poor rating and used Sprig’s AI analysis to group the answers. They saw there was a need to report more bugs and glitches, but also a need for more coaching moments and guides throughout the product. They made improvements that included a more collaborative, guided product experience for students with helpers throughout the app, resulting in a 74% increase in participation and an over 192% increase in social interfacing.
In-Product Surveys are Critical to Reducing Recall Bias
When trying to understand specific actions or decisions users are making about your product, especially when they only took seconds to make those decisions, you can’t wait to ask questions. You need to intercept them in that moment to understand exactly what they were thinking at the time, and why they are behaving in certain ways.
Often, these actions weren’t the result of planned behavior, but they did take critical decision making. To address frustrations and opportunities, you need to understand the user’s reasoning and, in the moment, you would be able to reveal that.
In-product surveys require less time and resources on your end, yet they guarantee you’ll capture the user as they’re completing the desired action to ensure your insights are specific, detailed, and actionable. They account for practice by asking when tasks are done infrequently or for the first time, recency by asking right after the action has taken place, and context by asking while the user is in the same context.
Want to reduce recall bias with in-product surveys? Create an account and we’ll connect you with a Sprig research advocate so you can learn how to launch an in-product survey today.