How to Run: Competitive Benchmarking Survey

Introduction

A competitive benchmarking survey measures your product against named competitors on identical questions, asked of comparable respondents in the same field window. To run one, recruit every brand’s users from one frame, set a quota for each brand, have each brand rated by its own users, and report only gaps that clear an adjusted interval. A five-brand set needs about 557 users per brand to read a 10-point gap.

Last updated October 2026.

Why Most Competitive Benchmarks Mislead

Most competitive benchmarks fail at the design stage, not the analysis stage. A team surveys its own customers in-product, buys a panel of competitor users, asks everyone to rate every brand they know, and then reads a ranking that mostly reflects brand size.

Four choices prevent that: one sample frame for every brand, a quota for each brand’s users, each brand rated by its own current users, and gaps reported only when they clear an interval adjusted for the number of comparisons.

Mixing a survey run inside your product with a panel of competitor users confounds brand with method. Asking everyone to rate every brand they know typically produces a ranking of brand size, because attitudes largely follow usage.

The sample is set by the smallest brand: reading a five-brand set for a 10-point gap takes about 557 users of each brand when you compare only your product against each competitor, and about 665 when every pair is compared.

When to Run a Competitive Benchmarking Survey

The clearest case is a positioning or roadmap decision that turns on where you stand against specific competitors.

A product team deciding whether to invest in onboarding or in reporting needs to know which one trails the competitor set, measured among people who actually use each product.

A competitive benchmark fits when the team keeps asserting a comparison nobody has measured.

“Our support is better than theirs” is a claim a benchmark can test, provided both support experiences are rated by their own users under identical conditions.

It fits before a messaging or sales enablement push. A claim that ships in a sales deck is often challenged by a buyer, and a benchmark with its frame, cell sizes and comparison plan stated gives the sales team something a buyer can check.

It fits when a new competitor has entered and the team needs a first read on where that competitor is strong. The same study can carry the incumbents, so the new entrant is read against the full set rather than against your product alone.

And it fits as a baseline. A first benchmark, run with a frozen instrument, becomes the comparison point for the next one.

Where a competitive benchmark doesn’t fit is as a quick answer to “are we winning?” The question is too broad for the method, and the next section names what to run instead.

When Not to Run a Competitive Benchmarking Survey

Eight situations, each naming the instrument that’s actually right. The first two are the ones most likely to change what you do.

You need to know why you win or lose specific deals. A benchmark scores the population of users, not the decision a buyer made last month. The better instrument is a win-loss study, which interviews recent buyers and lost prospects close to the decision and asks what tipped it.

You need to know whether closing a gap would move share. A benchmark correlates ratings with brand choice, and attitudes largely follow usage, so the direction of any link can’t be read from the survey. The better instrument is an experiment: message testing for claims, and conjoint analysis for feature trade-offs.

You need to know what share you actually hold. A stated primary brand is self-report, and it commonly drifts from purchase or usage records. The better instrument is sales data, consumer purchase panel data, or product analytics for your own users.

You need to know whether anything changed. A single benchmark is a snapshot, and two snapshots built on different instruments aren’t a trend. The better instrument is a brand tracker carrying a frozen subset of the benchmark’s measures.

You need to know how well people complete tasks in each product. Ratings aren’t performance. A user can rate a product’s ease highly and still fail the task in front of them. The better instrument is a task-based usability benchmark, which the design section covers as its own fork.

You need to know where your price sits against competitors. Satisfaction with value isn’t willingness to pay. The better instrument is Gabor-Granger or conjoint analysis.

You need to know which competitor feature matters most. Attribute ratings typically compress toward the top of the scale, which makes a long list hard to rank. The better instrument is MaxDiff, which forces trade-offs between items.

You need a size estimate for the whole market. Quota cells are filled to equal sizes by design, so the raw benchmark file overstates small brands and understates large ones. The better instrument is a usage and attitudes study weighted to the category population, or the benchmark’s own screener data weighted back to each brand’s true share.

Each of these is a question executives commonly bring to a benchmark readout. Rather than stretching the benchmark to answer them, name the better instrument in the room and scope a follow-up study. That keeps the benchmark credible for the question it was built to answer.

Hold the first two limitations hardest when the results are presented. They’re the two that executives most want a benchmark to answer, because “why we lose” and “what would win share” are the easiest things to put on a slide.

State both limitations on the readout’s first page, next to the frame, the field dates and the comparison plan. A benchmark slide that travels without them is commonly quoted as an answer to the question it can’t answer, often months after the study closed and by someone who never saw the method.

The pattern across all eight is the same. A competitive benchmark is the right tool for relative position among each brand’s own users, and the wrong tool for causes, levels, trends and prices.

Designing the Study, and the Three Forks

Every competitive benchmark takes three design decisions, and they interact. Listing competitors and KPIs without settling these three is how a benchmark ends up producing a share chart with a satisfaction label on it.

The three forks are what you benchmark, who rates which brand, and which frame and mode each brand is measured through. The committed default comes first, and the evidence follows.

The committed default. Recruit every brand’s users, yours included, from one frame, in one mode, in one field window, with the sponsor blinded. Screen on bounded current use, and quota each brand on a single-select “primary brand” question. Each respondent rates their primary brand on an identical block.

Ask perception as a pick-any grid of everyone aware, and read every perception score against that brand’s user base. Report a gap only when it clears an interval adjusted for the number of comparisons.

Fork 1: What You Benchmark

Choose between experienced performance, perception, and task-based usability before writing a single question.

Experienced performance is how a brand’s own users rate it on satisfaction, attributes, effort and likelihood to switch. It’s typically the core of most competitive benchmarks and the default in this guide.

Perception is which brands people associate with which statements, whether or not they use them. It describes reputation, and it’s generally dominated by brand size.

Task-based usability is how well people complete the same tasks in each product. It needs participants working inside each product, which makes it a usability study with its own sample rules rather than a survey.

Fork 2: Who Rates Which Brand

Have each brand rated by its own current users: route each respondent to the rating block for the brand they use most, and don’t ask every respondent to rate every brand they’ve heard of.

Asking everyone to rate every brand produces a ranking of brand size, because expressed attitudes depend on how recently respondents used the brand (Bird, Channon and Ehrenberg, Journal of Marketing Research, 1970). The usage and attitudes guide carries the full argument and the replication across 45 datasets. The critique section below applies it to a competitor set.

The user-rated design has a bias of its own. People who chose a brand are often predisposed to rate it well, and the selection applies to every brand in the set, yours included.

That’s a reason to read results relative to one another rather than as absolute levels, and it isn’t a reason to abandon the design.

A third option is the dual-user design, in which people who use two brands rate both. It controls for the respondent, but dual users are commonly a distinct group with their own habits, so their ratings describe dual users rather than each brand’s user base.

Why Comparing Users against Users Is Fair

Competing brands' user bases generally look alike.

Anesbury, Winchester and Kennedy, in Marketing Letters in 2017, examined “700 brands in more than 60 consumer packaged goods categories, for more than 160 variables” and found that brand user profiles “seldom differ” and “seldom change much over 3 to 6 years.”

That evidence comes from consumer packaged goods, and it shouldn’t be stretched to every business software category without checking.

Where it holds, it means a user-rated benchmark is mostly comparing experiences rather than comparing different kinds of people. Check it in your own data by comparing the profiling variables across brand cells before reading any score.

Fork 3: One Frame and One Mode for Every Brand

Recruit every brand’s users from the same source, through the same mode, in the same weeks.

The tempting design is the cheap one: survey your own customers in-product or by email, and buy competitor users from a panel. That design confounds brand with method, and nothing in the analysis can separate them.

de Leeuw, in the Journal of Official Statistics in 2005, described the same problem for surveys that change mode over time: “Time effects and mode effects in the results are fully confounded.”

Swap time for brand and the sentence describes the mixed-frame benchmark exactly.

Your customers answering inside your product, while logged in and mid-task, are typically in a different context from a competitor’s users answering a panel invitation at home.

So field every brand, including yours, through a panel or another single external source. Identify your own users with the same primary-brand screener everyone else answers.

A boost of your own customers can still be useful for depth. Report it separately, and never pool it with the panel cell for your brand.

Blinding the Sponsor

Blind the sponsor. A survey that announces it comes from one of the brands being rated invites respondents to answer toward that brand.

No peer-reviewed estimate of that effect’s size was found for this guide, so treat blinding as practitioner judgment rather than a sourced rule. It generally costs almost nothing, which is why it’s the default.

In-product surveys can’t be blinded, because they run inside your product. That’s one more reason the in-product frame fails Fork 3 for a competitive read.

The Task-based Fork

A task-based competitive benchmark is a usability study, and it has its own sample rules.

Sauro and Lewis at MeasuringU, in their 2020 sample-size recommendations for benchmark studies, advise that to detect differences of 12 percent or greater against a competitor, “you should plan for a sample size of 213 in each group for a between-subjects study or 93 total for a within-subjects study,” at 90 percent confidence and 80 percent power.

That is MeasuringU’s recommendation for usability metrics, not a universal rule, and it answers a different question from the survey design in the rest of this guide.

Run it on unmoderated usability tooling that can send participants into each live product. Don’t try to fold tasks into a ratings survey. Run the two studies separately and read them together.

The Design Tree

Work down the questions in order until a branch fires.

Do you need to know how people perform tasks in each product? Yes, run a task-based usability benchmark on tooling that supports tasks in live products. No, continue with a survey benchmark.

Do you need experienced performance or reputation? Experienced performance, use the user-rated design with a quota per brand. Reputation, add a pick-any perception grid and read it against usage.

Are your own users and competitors' users coming from different sources or modes? Yes, stop and redesign, because brand and method will be confounded. No, continue.

Will the survey announce who is sponsoring it? Yes, blind it. If blinding is impossible, as with in-product delivery, use a different channel for the benchmark.

Does the rating block need to pipe in the respondent’s brand name and randomize attribute order? Build one block per brand with display logic instead of piping, so randomization stays available.

Is the smallest competitor you plan to read below about 10 percent of category users? The 10 percent line is a working convention. Yes, read the sample-size tables before anything else, because filling that cell will set the cost of the study.

Sprig Study Logic: response piping and randomization can’t run in the same survey. The study logic documentation states that “if one has been added to the survey, the other will be disabled,” so piping each respondent’s brand into the rating block would switch off attribute-order randomization everywhere.

The workaround is one display-logic block per brand, shown on the primary-brand answer. Display logic is an Enterprise feature. Recorded Task is documented for prototypes only, so the task-based fork isn’t something the platform runs against a competitor’s live product.

Writing the Instrument

A competitive benchmark instrument has a fixed order, and the order does as much work as the wording. Behavior comes before evaluation, the scored block is identical for every brand, and profiling comes last.

The Instrument Spec

This is the instrument in fielding order, ready to lift. Replace the category wording, the brand list and the attributes, and keep the order.

Screener. Category use within a bounded period: “In the last 90 days, have you used [CATEGORY] for work?” Screen out anyone who hasn’t.
Primary brand. “Which of these do you use most for [CATEGORY]?” as a single-select from a randomized list, with “Other” and “None of these.” This is the quota question.
Tenure. How long the respondent has used their primary brand, in bands.
Repertoire. Which other brands the respondent has used in the last 12 months, as a multi-select.
Rating block for the primary brand, one block per brand, shown by display logic. Overall satisfaction, five to eight attribute ratings, effort, and likelihood to switch in the next 12 months.
Switching reason. “What would make you switch from [BRAND] to something else?” as open text with a follow-up probe.
Perception grid. Brands by statements, pick-any, asked of everyone for every brand they recognize.
Profiling. Company size, role, industry, and any variable used to check that brand cells look alike.

Four rules sit under the list:

Use identical wording, scales and question order in every brand’s rating block without exception
Keep brand-specific features out of the scored block, because a competitor’s users can’t rate them
Randomize attribute order within each block and brand order within every list
Blind the sponsor in the invitation, the survey header and the thank-you page

The first two rules are what make the benchmark a comparison. The last two follow from the design section.

Choosing the Attributes

Choose attributes that every brand in the set can be rated on. List the jobs the category is used for, not the features your product has.

“Setting up a new project quickly” can be rated for any project-management tool. “Using the timeline view” can only be rated for products that have one, and it quietly turns the benchmark into a feature audit.

Five to eight attributes is a common working range, though it’s a convention rather than a sourced figure. Each extra attribute typically adds length to every block and another comparison to the analysis.

Draw the list from the category’s own vocabulary, which generally differs from the product team’s. Support tickets, sales call notes and the switching-reason text from a previous study are typically better sources than an internal brainstorm.

Keeping the Blocks Identical

Build one block per brand and make every block a copy of the first.

Build the first block, freeze it, and duplicate it for each brand before changing only the brand name, instead of editing each block by hand.

A single changed word between blocks, such as “easy” in one and “simple” in another, commonly produces a gap that belongs to the wording, not the brand.

Compare the block wording in a text diff before launch rather than by eye. A diff catches a changed word that a side-by-side read typically misses.

Why Perception Is Asked of Everyone

The perception grid is the one place the benchmark asks every respondent about every brand they know.

That’s deliberate. Perception describes reputation among the category, and reputation includes people who have never used the brand.

Ask it pick-any, meaning respondents tick any brands that fit each statement. The grid collects associations for every brand in one pass, which rating each brand on each statement would multiply in length.

Then read the grid against usage, as the core-calculation section shows, because a raw pick-any ranking is largely a ranking of brand size.

The Switching Question

Ask likelihood to switch as a bounded behavior. “How likely are you to switch from [BRAND] in the next 12 months?” is more useful than an open-ended loyalty question, because it names a period and an action.

Stated intent predicts behavior imperfectly, and the critique section returns to that point. Treat switching intent as a comparison between brands, not as a forecast of churn.

Sprig Design Agent: the Design Agent can draft the full instrument from a written brief, including logic and randomization. It won’t know your category definition, your attribute list or the one-block-per-brand structure unless the brief states them, so check all three before fielding.

Scale and Scoring Choices

Use the same scale, the same labels and the same scoring rule for every brand. The choice of scale matters less than holding it constant.

Rating Scales

A fully labeled five-point scale is frequently the default for satisfaction and attribute ratings.

Use one scale for the whole rating block so every attribute can be read the same way. A new scale for each attribute breaks that.

Report each attribute as a top-two-box share, meaning the percentage choosing the top two points, alongside the mean.

Lead with the top-two-box share, because it reads as a share of users and compares across brands directly. Report the mean beside it as the check that uses every scale point.

Which Headline Metric to Use

Pick one headline metric before fielding and hold it.

This guide’s default headline is overall satisfaction, because every brand’s users can answer it on the same terms. Effort and likelihood to switch are strong second measures, and each has its own guide on method. Customer satisfaction and customer effort are covered in depth there, including their own scale debates.

Recommendation metrics are also frequently used as competitive headlines. The Net Promoter Score guide covers that metric, and this guide doesn’t repeat it.

Whichever headline you pick, the benchmark rules are the same: identical wording, identical scale, and a result read with its interval.

Scoring the Perception Grid

Score the perception grid as the share of respondents aware of a brand who tick it for each statement.

Use only respondents who recognized the brand as the base, because an unrecognized brand can’t be associated with anything.

Report the base with every score. A brand recognized by 90 percent of respondents and a brand recognized by 20 percent have very different bases, and the second one’s scores are typically much less precise.

Sample Size and Precision

Size a competitive benchmark from the smallest competitor you intend to read and from the number of comparisons you plan to run. Those two inputs change the answer more than any other decision in this guide.

No methodology source found for this guide publishes a sample-size rule for survey-based competitive benchmarks.

MeasuringU publishes one for task-based usability benchmarks, covered in the design section. For the survey, the arithmetic below is the rule.

The Derived Tables

The tables are computed, not copied, and every figure was recomputed a second way with the simplified closed-form formula. The two methods agree within about 0.3 percent at every point.

They assume 95 percent two-sided confidence, 80 percent power for detection, simple random sampling, no design effect and no weighting.

Table A. Margin of error for one brand’s score

| n per brand | Margin at 50% | Margin at 20% | |:-----------:|:-------------:|:-------------:| | 100 | ±9.8 points | ±7.8 | | 200 | ±6.9 | ±5.5 | | 300 | ±5.7 | ±4.5 | | 400 | ±4.9 | ±3.9 | | 600 | ±4.0 | ±3.2 |

‍‍Table B. Respondents per brand needed to detect a true gap between two brands on a 50 percent metric, adjusted for the number of pairwise comparisons

| Brands in set | Pairwise comparisons | 5-point gap | 10-point gap | 15-point gap | 20-point gap | |:-------------:|:--------------------:|:-----------:|:------------:|:------------:|:------------:| | 2 | 1 | 1,569 | 392 | 174 | 97 | | 3 | 3 | 2,093 | 523 | 232 | 130 | | 4 | 6 | 2,421 | 605 | 268 | 150 | | 5 | 10 | 2,662 | 665 | 295 | 165 |

Table C. Category users you must qualify to fill a 200-person cell, by the brand’s share of category users

| Brand's share of users | 40% | 20% | 10% | 5% | 2% | |:----------------------:|:---:|:-----:|:-----:|:-----:|:------:| | Qualified users needed | 500 | 1,000 | 2,000 | 4,000 | 10,000 |

The Formulas

The margin of error is 1.96 times the square root of p times one minus p divided by n, with the whole fraction under the root. To lay the brand cells out side by side before testing them, a simple cross-tab is enough.

The per-brand n in Table B uses the standard two-proportion formula at a significance level of 0.05 divided by the number of comparisons, which is the Bonferroni adjustment. Table B centers each gap on 50 percent, so a 10-point gap is 45 percent against 55. Table C is the cell size divided by the brand’s share of category users.

Why the Number of Comparisons Matters

Every added brand adds comparisons, and every comparison is another chance to find a gap that isn’t there.

A two-brand benchmark runs one comparison. A five-brand benchmark comparing every pair runs 10.

At a 5 percent significance level per test, 10 independent tests carry roughly a 40 percent chance of at least one false positive. The adjustment in Table B holds that chance near 5 percent, and it costs sample.

Comparing You against Each, or Every Pair

Decide before fielding whether you will compare only your product against each competitor or every brand against every other.

Run you-versus-each instead of every pair when the decision is about your product. A five-brand set then needs four comparisons, not 10, which lowers the adjustment and the sample.

Table B2. Respondents per brand when you compare only your product against each competitor

| Brands in set | Comparisons | 5-point gap | 10-point gap | 15-point gap | 20-point gap | |:-------------:|:-----------:|:-----------:|:------------:|:------------:|:------------:| | 2 | 1 | 1,569 | 392 | 174 | 97 | | 3 | 2 | 1,900 | 474 | 210 | 118 | | 4 | 3 | 2,093 | 523 | 232 | 130 | | 5 | 4 | 2,229 | 557 | 247 | 138 |

Write the comparison plan down before fielding. A plan chosen after seeing the data isn’t an adjustment. It’s a search.

What the Tables Mean in Practice

Reading a five-brand set for a 10-point gap takes about 557 users of each brand with you-versus-each comparisons, and about 665 with every pair compared.

If the smallest brand holds 5 percent of category users, Table C’s arithmetic puts the qualified users needed to fill a 557-person cell at about 11,140.

That’s why most competitive benchmarks should include fewer brands than the team wants. Each added brand raises both the per-cell n and the screening volume in Table C.

No Minimum Floor

This guide publishes no minimum cell size. Pick the gap your decision needs to detect, read the n from Table B, and fill every cell to it. Don’t borrow a floor from elsewhere.

Where a total market figure is also needed, the quota cells have to be weighted back to each brand’s true share before reporting any all-brand number.

Audience, Targeting and Screening

Screen on category behavior, then quota on primary brand. The screener decides who counts as a category user, and the primary-brand question decides which cell they fill.

Write the category definition down before building anything. “Used any project-management software for work in the last 90 days” and “is responsible for choosing project-management software” define two different populations with two different sets of brand cells.

Business categories often need one more decision: whether the respondent is the user, the buyer or both. Screen for the role the decision cares about, and record the other role as a profiling variable.

Quotas per Brand

Set a quota for every brand cell at the n chosen from Table B. Hold every cell open until it reaches the target, instead of letting cells fill in proportion to market share.

The largest brands will typically fill first. The smallest will often set the field time and the cost, which Table C shows before fielding begins.

Close “Other” and “None of these” cells at a small number, or screen them out. They add cost without adding a readable brand.

Checking That Cells Look Alike

Compare the profiling variables across brand cells before reading any score.

If one competitor’s cell is heavily weighted toward small companies and yours toward large ones, part of any score gap often belongs to company size.

Where the cells differ materially, report the comparison within the largest shared segment as well, or weight the cells to a common profile and say so.

Sprig Panels and Quotas: Panels put every brand’s users in one external frame, and response quotas can close each brand cell on a single-select primary-brand question.

Quotas and Panels are both Enterprise features. A single quota can match several answers at once, but only on single-select, rating-scale or NPS questions. Panels documents recruitment filters such as location, age, job title and industry, not brand-usage targeting, and it documents no country coverage or fielding turnaround. Confirm that a primary-brand screener can be fielded to the panel, and confirm reach for each competitor’s users, before planning cell sizes.

Fielding and Delivery

Field every brand in one window, through one mode, with one invitation. A benchmark fielded in stages commonly mixes events into the brand comparison, and nothing in the analysis can separate them afterwards.

Pilot with a small soft launch before the full field. Twenty to 50 completes, a practitioner convention, will typically show broken display logic, a confusing screener and a completion time longer than planned.

Check in the pilot that every respondent saw exactly one rating block and that it matched their primary brand. A display-logic error that shows the wrong block is invisible in the results and fatal to them.

Hold the brand list, the attribute list and the wording fixed once fielding starts. An edit mid-field typically splits the data into two instruments, and the two halves can’t be pooled.

The Comparability Checklist

Work through this list in order before the full launch. Most items can’t be fixed once responses start arriving.

Confirm every brand’s users come from the same source and the same mode.
Confirm the field window is the same for every brand.
Confirm the sponsor is blinded in the invitation, header and closing screen.
Confirm the screener bounds category use to a stated period.
Confirm a quota exists for every brand cell at the planned n.
Compare every rating block side by side and confirm the wording is identical.
Confirm attribute order randomizes inside each block.
Write down the comparison plan and the adjustment before any data arrives.
Soft launch, confirm each respondent saw one correct block, and record the median completion time.

Items 1, 6 and 8 are the ones that can’t be repaired after launch. A mixed frame, a changed word or an unwritten comparison plan commonly surfaces only at the readout, when the fix means fielding again.

Sprig White-Labeling and Follow-Ups: link surveys can be white-labeled so the sponsor stays blind (confirm with Sprig whether panel surveys can be), and AI Follow-Ups (Field Agent) can add a short probe after the switching question. A follow-up is still self-report and not a win-loss interview, and in-product surveys always carry your brand, so they can’t be used for the benchmark itself.

Timing and Cadence

Run a competitive benchmark when a positioning, roadmap or sales decision is open and the comparison is missing or stale.

No source found for this guide publishes a repeat cadence for competitive benchmarks.

Refresh the benchmark when something structural changes, not on a calendar: a competitor launch, a major release of your own, a pricing change in the category, or a new entrant.

Keep the field window as short as the smallest cell allows, and state its dates in the report. A window of two to three weeks is a common working choice, though it’s a practitioner convention.

Avoid windows that straddle a known event. A benchmark fielded across a competitor’s launch week often measures two versions of that competitor.

When you do refresh, keep the competitor set, the frame and the rating block identical. A refreshed benchmark that changes any of the three is a new study.

Quality Control and Data Hygiene

Four checks matter most, and the first two are specific to competitive benchmarks.

Wrong block shown. Confirm that every respondent’s rated brand matches their primary-brand answer. Any mismatch is a logic error, and every affected response is removed rather than recoded.

Brand cell drift. Compare each cell’s profile against the others as fielding runs. A cell that skews toward one company size or region mid-field will commonly distort that brand’s score.

Straight-lining. Flag respondents who give every attribute the same rating. In a five-to-eight attribute block, identical answers on every item are rare enough, as a practitioner heuristic, to review respondent by respondent.

Speeders. Take a third of the pilot median completion time as a working floor, and review before deleting anything. That fraction is a convention, so check what it actually excludes first.

Report exclusions by brand cell. State the starting count, the number removed, the rule used and the final count for each brand, because unequal exclusions across cells can frequently create a gap by themselves.

The Core Calculation

A competitive benchmark produces three numbers that belong together: each brand’s score with its interval, the adjusted gap between your product and each competitor, and each brand’s position against what its size predicts.

Report all three side by side, not a ranking.

Scores with Intervals

Compute each brand’s top-two-box share and its 95 percent interval.

Here is a worked example with illustrative numbers: four brands, 400 users in each cell, and overall satisfaction as the headline.

| Brand | Top-two-box satisfaction | 95% interval | |:------------:|:------------------------:|:------------:| | Your product | 62% | 57.2 to 66.6 | | Competitor A | 55% | 50.1 to 59.8 | | Competitor B | 66% | 61.2 to 70.5 | | Competitor C | 58% | 53.1 to 62.7 |

The intervals are Wilson score intervals, which generally behave better than the simple formula near the ends of the scale.

Gaps with the Adjustment

Compare your product against each competitor with a two-proportion test at the adjusted significance level.

The comparison plan in this example is you-versus-each, which is three comparisons, so the adjusted level is 0.05 divided by 3, or about 0.017.

Your product leads Competitor A by 7 points. The unadjusted p-value is 0.045, which looks significant, but it doesn’t clear the adjusted level.

The adjusted interval on that gap runs from minus 1.3 to plus 15.3 points, and it crosses zero.

The gaps against Competitor B, minus 4 points, and Competitor C, plus 4 points, are further from significance.

At 400 per cell, this example is also below the 523 per brand that Table B2 gives for a 10-point gap with three comparisons. The honest readout is that this study can’t separate your product from any of the three competitors on satisfaction. That’s a real finding, and it’s the one an unadjusted slide would often have hidden.

Reading against Brand Size

Read loyalty-type measures against what each brand’s size predicts.

Ehrenberg, Goodhardt and Barwise, in the Journal of Marketing in 1990, described the pattern as double jeopardy: “a small brand typically has far fewer buyers than a larger brand. In addition, its buyers tend to buy it less often.”

Smaller brands typically score lower on loyalty measures for reasons of arithmetic rather than failure.

Fit a line across brands relating each brand’s share of category users to its “would choose again” score, and report how far each brand sits above or below that line. The example below uses illustrative numbers.

| Brand | Share of users | Would choose again | Expected from size | Deviation | |:------------:|:--------------:|:------------------:|:------------------:|:---------:| | Competitor A | 40% | 78% | 78.4% | minus 0.4 | | Your product | 25% | 70% | 71.5% | minus 1.5 | | Competitor B | 20% | 74% | 69.2% | plus 4.8 | | Competitor C | 15% | 64% | 66.9% | minus 2.9 |

The raw ranking puts Competitor A first by 4 points. The deviation column tells a different story: Competitor B sits 4.8 points above what a brand of its size would normally get.

Competitor B’s observed 74 percent carries a margin of about ±4.3 points at 400 respondents, a margin that ignores the uncertainty in the fitted line itself. Treat the 4.8-point deviation as directional, not as a finding.

The expected values come from an ordinary least-squares line with a slope of 0.457 and an intercept of 0.601. A leave-one-out refit, predicting each brand from the other three, moves Competitor B’s deviation to plus 7.1 and Competitor C’s to minus 6.3, which shows how much a four-brand line depends on each point.

Perception against Usage

Read the perception grid the same way. A brand’s association share on each statement is compared with the share expected from its usage, and the deviation is the finding.

The logic is the same as the loyalty line, and it has the same cause: attitudes largely follow usage.

Cutting any of these numbers by segment belongs to cross-tab analysis, and coding the switching reasons into themes belongs to the theming workflow.

Benchmarks and What a Good Result Looks Like

There’s no published cross-industry benchmark for a competitive benchmarking survey, and there shouldn’t be one: the benchmark is the competitor set, measured in the same study, on the same frame, in the same field window.

That absence is structural. A score from another company’s study used a different frame, a different wording and a different competitor set, so it can’t be compared with yours.

The brand tracking guide makes the same argument for awareness: the competitive read is the one comparison that transfers, because it holds the confounds constant.

Norms That Exist for Standardized Instruments

Three standardized instruments do carry external norms, and each comes with limits.

The System Usability Scale (SUS) has the best-documented public norm. Lewis and Sauro, in the Journal of Usability Studies in 2018, describe a curved grading scale built from “241 industrial usability studies and surveys” in which “a SUS score of 68 is at the center of the range for a ‘C.’” That norm comes from usability studies, not competitive benchmarks, and describes products in general.

The SUPR-Q® is a registered mark with a licensed normative database of over 200 websites (MeasuringU product page, retrieved September 25, 2026). Using those norms requires a license from MeasuringU, and this guide reproduces none of its items or figures.

The American Customer Satisfaction Index (ACSI®) publishes scores for more than 400 companies in over 40 industries on a 0 to 100 scale (ACSI industries page, retrieved September 25, 2026). ACSI states that its data may not be used for advertising or promotional purposes without written consent, and its scores come from its own model, so they aren’t comparable with a satisfaction question in your survey.

Figures in Circulation That Do Not Hold

Ranges for a “good” satisfaction score circulate in vendor guides on this topic, typically without a source attached.

A satisfaction percentage typically depends on the scale, the wording, the cut point and who answers. Compare your score with the competitors measured beside it instead of borrowing a range like that.

Build Your Own

Your benchmark is your own previous benchmark, on the same competitor set, frame and instrument.

Archive every study with the instrument version recorded, so the next benchmark has something valid to compare against.

This guide doesn’t publish a minimum number of prior studies before a trend is meaningful. No source supports one.

Interpreting and Acting on the Result

Work through the output in a fixed order, starting with the comparison plan and ending with the open text.

Start with the comparison plan. Restate which comparisons were planned and at what adjusted level, before anyone looks at a gap.

Then check the cells. Confirm each brand’s cell reached its target n and that profiles look alike. A cell that fell short is reported with its wider interval, or not at all.

Then the headline metric with intervals. Report which gaps clear the adjusted interval and which don’t. A gap that doesn’t clear it’s reported as not distinguishable, however large it looks.

Then the attributes. Attribute gaps are where a benchmark generally becomes actionable, because they point at a job rather than at the whole product. Apply the same adjustment across attributes, since eight attributes times three competitors is 24 comparisons.

Then the size line. Read loyalty and perception against brand size. A brand above its line is doing something its size doesn’t explain.

Then the switching reasons. The open text explains the patterns above rather than standing alone. Read the reasons given by each competitor’s users about their own brand, because those often name what that competitor would need to fix.

Turning a Gap into an Action

A cleared attribute gap supports a hypothesis worth testing, not a conclusion.

Suppose your product trails Competitor B on “setting up a new project quickly” by a margin that clears the adjusted interval, and Competitor B’s users mention templates in their switching reasons.

That chain points at onboarding templates as a candidate investment. The test belongs to a concept test or message test, not to the benchmark.

What the Result Licenses

A competitive benchmark licenses statements about relative position: where you lead, where you trail, and where the study can’t tell the difference.

A benchmark doesn’t license a claim about why buyers choose, a forecast of share, or a public claim of superiority without the method stated beside it.

Sprig Theming: AI open-text theming (Synthesize Agent) codes the switching reasons into themes with response-level traceability. Themes regenerate and no reproducibility claim is published, so theme counts aren’t comparable between two benchmark waves, and a refresh should apply the original theme list rather than generating a new one.

Running the Analysis with Claude or ChatGPT

The benchmark calculations are typically simple statistics applied to a reshaped file. The reshape from one block per brand into one comparable column is where a model most often misassigns respondents.

Intervals, the adjustment and the size line are run by the connected AI client, because none of the three is a documented Sprig feature. Sprig’s Analyze Agent is described as coming soon (its page also carries a preview banner). Its pages list crosstabs, t-tests and ANOVA but don’t document confidence intervals, a multiple-comparison correction or weighting, so don’t plan the benchmark around it.

This chapter covers only what’s specific to a competitive benchmark. Theme creation belongs to the advanced cross-tab guide, cutting any number by segment belongs to the simple cross-tab guide, and connector setup belongs to the MCP documentation.

What the Analysis Produces

The analysis produces four outputs in one pass: each brand’s scores with Wilson intervals, the planned gaps with adjusted tests, each brand’s deviation from the size line, and a plain-language readout that refuses to rank brands whose planned comparison fails the adjusted test.

Each output is saved as its own named file. Separate files make each number checkable on its own, which one long summary doesn’t.

Prompt One: Reshape, Score and Compare

You’re analyzing a competitive benchmarking survey exported as CSV.

WHAT THIS PROMPT DOES: reshapes the file to one row per respondent with a brand
column, then computes each brand’s scores with intervals and tests the planned
comparisons at an adjusted significance level.
WHAT IT RETURNS: a reshaped data file, a scores table, a comparisons table and
a short method note.

Files: [PATH TO EXPORT FILE, required, for example survey_export.csv]
Primary brand question: [PRIMARY BRAND QUESTION COLUMN, required, for example Q2_Response]
Rating block columns, by brand: [LIST EACH BRAND AND ITS BLOCK COLUMNS, required]
Headline metric: [HEADLINE QUESTION, default overall satisfaction]
Top-two-box values: [VALUES, default 4 and 5 on a 5-point scale]
Your brand: [YOUR BRAND NAME, required]
Comparison plan: [YOU VERSUS EACH or ALL PAIRS, default YOU VERSUS EACH]
Adjustment: [BONFERRONI or HOLM, default BONFERRONI]
Minimum cell size: [N PER BRAND YOU PLANNED FROM TABLE B2, default 557]

Use code to calculate this, not estimation. Don’t compute any value by
reasoning about it in prose.

Rules:
1. Reshape so each respondent has one row, a brand column taken from the
   primary brand answer, and one column per rated item. Report any respondent
   whose populated block doesn’t match their primary brand, and exclude them.
2. Exclude a respondent from any calculation where the relevant answer is
   blank, null or a non-response code. Blank isn’t the lowest scale point.
3. Any brand cell with fewer respondents than the minimum cell size,
   including zero, is reported as below threshold and not interpreted or
   tested.
4. For each brand and each rated item, report n, the mean, the top-two-box
   share, and a 95 percent Wilson score interval on the share.
5. Run only the comparisons named in the comparison plan. For each, report
   the gap, the unadjusted p-value from a two-proportion z-test, the adjusted
   significance level, and whether the gap clears it.
6. Don’t rank brands. Describe two brands as distinguishable only when their
   planned comparison clears the adjusted level in rule 5.

Verify every interval and every test a second way by bootstrapping
[ITERATIONS, default 5000] resamples of respondents within each brand cell,
and compare the bootstrap intervals and gaps with the analytic ones. These
are different methods and should broadly agree.

If any result disagrees between the two methods by more than [TOLERANCE,
default 1.5 points], say that you can’t reconcile them and show me the
inputs. Don’t correct a mismatch by choosing the answer that looks more
reasonable.

Save as cb_reshaped.csv, cb_scores.csv, cb_comparisons.csv and
cb_method_note.md.

Prompt Two: Read Brands against Their Size

WHAT THIS PROMPT DOES: reads loyalty and perception scores against the level
expected from each brand’s share of category users.
WHAT IT RETURNS: one table per measure with observed, expected and deviation
for every brand, plus the fitted line.

Files: cb_reshaped.csv and [PATH TO EXPORT FILE, required, for example survey_export.csv]
Brand share source: [PRIMARY BRAND SHARE FROM SCREENER DATA, or EXTERNAL
SHARES I SUPPLY, required]
Loyalty measure: [QUESTION COLUMN, default would choose again]
Perception grid columns: [LIST THE GRID COLUMNS, required, for example Q7_Ease, Q7_Support]
Brands: [LIST OF BRANDS, required, for example Acme, Birch, Cobalt]
Minimum base per brand: [SMALLEST BASE WHOSE TABLE A MARGIN YOU ACCEPT, no default, you set it]

Use code to calculate this, not estimation.

Rules:
1. Compute each brand’s share of category users from all screened category
   users, before quotas were applied, not from the quota cells.
2. Exclude any respondent whose loyalty answer is blank, null or a
   non-response code. Blank isn’t the lowest scale point. For the perception
   grid, a respondent counts toward a brand’s base only if they recognized
   that brand. Blank cells for recognized brands mean not
   associated. Blank cells for unrecognized brands are excluded.
3. Any brand with a base under the minimum, including zero, is reported as
   below threshold and left out of the fitted line.
4. For each measure, fit an ordinary least-squares line of the score on brand
   share across brands. Report slope, intercept, expected value and deviation
   per brand.
5. Report a 95 percent prediction interval from the fitted line for each
   brand, and mark any observed score inside its interval as not
   distinguishable from the line.
6. State the number of brands in the fit, and label every deviation that
   changes sign in the leave-one-out refit below as directional.

Recompute the slope and intercept a second way, from the closed-form
covariance of share and score divided by the variance of share, and compare
them with the fitted line. Then run a leave-one-out refit as a stability
check: drop each brand, fit the line on the remaining brands, and predict the
dropped brand. Report both deviations. If any brand changes sign, label it
unstable and show the inputs rather than choosing one.

If the two fits disagree on slope, intercept or any prediction interval by
more than [TOLERANCE, default 1.0 point], say that you can’t reconcile them
and show me the inputs.

Don’t describe any brand as strong or weak from the raw score. Describe it
only from the deviation.

Save as cb_size_line.csv.

Setup and Getting Your Data in

Export the responses as CSV, or pull them through the Sprig MCP connector. The documented export columns are:

surveyId, visitorId and userId for the survey and the respondent
createdAt for the timestamp
Q#_Question_Text and Q#_Response for each question and its answer
Themes for coded open-text themes
Attributes_# for any attributes passed in

With one display-logic block per brand, each brand’s ratings sit in different question columns. The reshape step in prompt one is what turns those into one comparable set, and it’s the step most worth checking by hand.

Keep the screener data. The size line needs brand shares from everyone who qualified, before quotas closed any cell, and a quota’d file on its own overstates small brands.

Put the data above the instructions in the prompt, and split very large files by brand cell.

Reading the Output

Read the method note before the results. It states the exclusions by cell, any block mismatches, and the adjusted significance level, and a high mismatch count means the display logic needs fixing before anything is reported.

Read the comparisons table from the column showing whether each gap clears the adjusted level. A gap that doesn’t clear it has no finding, however large the point estimate looks.

Read the size line from the deviation column, and treat any deviation that changes sign in the leave-one-out refit as directional. No source sets a minimum number of brands for this fit, and a line through four points can often move a lot when one brand’s score moves a little.

What to Verify before Reporting

Check five things before any number leaves the analysis:

Confirm the reshaped file has one row per respondent and one brand each.
Confirm the bootstrap and analytic intervals agree.
Confirm the comparisons run match the plan written before fielding.
Confirm no cell below the minimum is interpreted.
Recompute one gap and its p-value by hand from the reported counts.

If the hand calculation disagrees with the model, stop and rerun rather than reporting either number.

Researchers remain responsible for the comparison plan, the brand shares and every number that leaves the analysis. The model does the arithmetic, and it doesn’t decide what the arithmetic means.

Pitfalls

Six documented failure modes apply to any survey analysis run in a language model, and one more applies to competitive benchmarks specifically.

Discovering themes is harder than applying them. Hill and colleagues, in PLOS Digital Health in April 2026, found models matched human analysts on coding against an existing codebook, 93.5 percent against 92.7, and were materially worse at inductive discovery. Theme the switching reasons inductively once, rebuild the list yourself, then have the model apply your list.

Rare themes get over-predicted. Ashwin, Chhabra and Rao, in Sociological Methods and Research in May 2025, found non-random bias in 10 of 19 codes, with sparse codes systematically over-predicted. A small competitor’s switching reasons are a set of rare themes.

Outputs vary between runs. Thinking Machines Lab reported in September 2025 that 1,000 completions at temperature zero produced 80 unique outputs. Neither chat client exposes a seed, so produce any reportable number twice.

The middle of a long input gets lost. Liu and colleagues documented it in the Transactions of the Association for Computational Linguistics in 2024. Put data above instructions and split large files.

Models tend to agree with the user. Sharma and colleagues documented sycophancy at ICLR in 2024, and OpenAI withdrew a model update in April 2025 for being overly agreeable. Never ask a model to confirm that your product is winning.

Verbatims contain personal information nobody asked for. Switching reasons frequently name colleagues, employers and vendors. Whether that data trains a model depends on the vendor’s terms and your plan tier.

Asking a model which competitor you’re beating. A model asked that question will rank point estimates and ignore overlapping intervals, because a ranking is what the question asks for. Specify the comparison plan and forbid a ranking where intervals overlap.

Sprig MCP: the MCP connector gives Claude or ChatGPT governed access to benchmark responses without a manual export. It returns at most 1,000 responses per call, so a five-brand study with 665 users per cell, 3,325 completes in total, needs several calls or a CSV export. Researchers remain responsible for confirming the pulled count matches the completes in every brand cell.

The Critique You Should Know

Five objections apply to competitive benchmarks specifically, and each one aims at a different part of the design. The defense is real, and it’s narrow.

Perception Is Largely a Usage Echo

A competitive perception grid is substantially a market-share chart.

Bird, Channon and Ehrenberg showed in 1970 that expressed attitudes depend on how recently respondents used the brand, and Romaniuk and colleagues confirmed the pattern across 45 datasets in 2012.

A brand that leads on “innovative” or “trusted” often leads because more respondents use it. Presenting that lead as a perception advantage tells the team nothing it didn’t already know from market share.

Double Jeopardy Makes Challengers Look Like Failures

Smaller brands have fewer users, and those users are typically a little less loyal.

Ehrenberg, Goodhardt and Barwise documented the pattern across categories in 1990. The brand tracking guide applies the same arithmetic to funnel conversion. A benchmark that reports raw loyalty scores will almost always show the challenger trailing, and a team reading it will often conclude the challenger has a loyalty problem it doesn’t have.

Individual Answers Are Unstable

Brand attitude answers commonly change more than most teams expect.

Dall’Olmo Riley, Ehrenberg, Castleberry, Barwise and Barnard, in the International Journal of Research in Marketing in 1997, found that when respondents were re-interviewed, “on average only about 50% gave the same attitudinal Yes or No response as before.”

That finding is from the paper’s abstract, and it covers yes-or-no attitude statements rather than rating scales. It still explains why small gaps between brands are so often noise, and why the adjustment in the sample-size section isn’t optional.

User-rated Designs Inherit Selection

Every brand’s users chose that brand, and people generally rate their own choices favorably.

That bias doesn’t cancel perfectly. A brand whose users had few alternatives, such as a tool mandated by an employer, may carry less self-selection than a brand users sought out, and the benchmark can’t separate the two.

The practical response is to record how each respondent came to use their brand, chosen or assigned, and to report the headline split by that variable where the cells allow it.

Multi-country Benchmarks Carry Response-style Contamination

Response styles differ by country, so cross-country comparisons of brand scores are contaminated.

Baumgartner and Steenkamp, in the Journal of Marketing Research in 2001, studied representative samples from 11 European Union countries and wrote that “response styles are a source of contamination in questionnaire ratings.”

The practical rule follows directly. Compare brands within each country and report countries separately. Don’t compare a brand’s score in one country with another brand’s score in a different country.

The Counter-critique

Three responses. The first two carry published evidence, and the third is this guide’s design judgment. None of the three answers the instability or response-style critiques, which stand.

Cross-brand satisfaction scores have carried financial signal. Fornell, Mithas, Morgeson and Krishnan, in the Journal of Marketing in 2006, used ACSI data and concluded that “satisfied customers are economic assets with high returns/low risk.”

That finding has been challenged. Jacobson and Mizik, in Marketing Science in 2009, reexamined the same relationship and found the evidence of excess returns “limited to firms in the computer and Internet sector.” Read together, the two papers say satisfaction differences between brands can matter commercially, but the effect isn’t uniform across industries.

User-rated comparisons are fairer than they look. Anesbury and colleagues found that competing brands' users seldom differ, at least in consumer packaged goods. Where that holds, a user-versus-user benchmark mostly compares experiences, which is the thing it claims to compare.

The size objection has a design answer. Reading each brand against the level expected from its size, as the core-calculation section does, separates the part of a score that size explains from the part it doesn’t. That’s this guide’s judgment about how to respond to Ehrenberg’s work, not a sourced rebuttal of it.

Where This Guide Lands

Trust a competitive benchmark for relative position among each brand’s own users, measured under identical conditions and read against brand size.

Distrust it for perception rankings, for small gaps, for cross-country comparisons, and for any claim about why buyers choose.

Common Mistakes

Mixing frames. Your customers in-product and competitors' users from a panel confound brand with method. Recruit every brand from one source.

Letting everyone rate every brand. An all-aware ratings grid typically ranks brands by size, often by a wide margin. Route each respondent to the block for the brand they use.

Reporting unadjusted gaps. A five-brand set with every pair compared runs 10 tests. Adjust for the comparisons, and write the plan down before fielding.

Changing wording between blocks. One changed word creates a gap that belongs to the wording. Duplicate a frozen block rather than editing each one.

Including brand-specific features. A feature only your product has can’t be rated by a competitor’s users. Rate category jobs instead.

Sizing from the total instead of the smallest cell. A 2,000-complete study with a 5 percent brand reads that brand on about 100 people. Quota every cell to the planned n.

Announcing the sponsor. Respondents who know which brand is asking can often answer toward it. Blind the invitation and the survey itself.

Reading raw loyalty as a verdict on the challenger. Double jeopardy predicts lower loyalty for smaller brands. Read the deviation from the size line instead.

Comparing countries against each other. Response styles differ by country. Compare brands within a country.

Treating a benchmark as a win-loss study. A benchmark scores users, not decisions. Run win-loss interviews for why deals were won or lost.

Mistakes in the Readout

Ranking brands on point estimates. A ranking generally implies differences the adjusted tests didn’t find. Report any pair that fails its planned test as not distinguishable.

Publishing a superiority claim without the method. A benchmark claim in a sales deck is commonly challenged by a buyer. State the frame, the cell sizes and the adjustment beside the claim.

Comparing theme counts across waves. Regenerated themes are a different instrument each time. Apply the original theme list to the refresh instead.

Leading with the one gap that cleared. A readout that opens with the single significant attribute out of 24 overstates the study. Report how many comparisons were run and how many cleared, then show the gap.

Hiding the cells that fell short. A competitor whose cell reached 150 instead of 557 still appears on the slide, often with the same visual weight as the others. Mark short cells and show their wider intervals.

Reporting the quota file as market share. Equal cells make every brand look the same size. Weight to true shares before quoting any all-brand number.

Presenting a leave-one-out-unstable deviation as a finding. A size-line deviation that flips sign when one brand is dropped is a property of the line, not of the brand. Label it directional.

Adding brands at refresh. A new competitor added to the set changes the comparison count and the adjustment. Treat the refreshed benchmark as a new study for that reason alone.

Dropping a brand at refresh. Removing a competitor that scored well makes every other brand look better by comparison. Keep the set fixed, or report the change beside every refreshed number.

Synthetic Respondents

Synthetic respondents aren’t usable for brand scores in a competitive benchmark.

A language model’s view of a named brand is its training-data reputation, which is exactly the size-driven perception this guide warns against. A synthetic user of a small competitor has never used that competitor.

The general evidence points the same way. Bisbee and colleagues, in Political Analysis in 2024, compared real respondents with synthetic ones and found that averages corresponded closely.

But 48 percent of regression coefficients differed significantly from the human data, and 32 percent of those flipped sign, with artificially deflated variance. Gaps between brands and deviations from a size line are relationships, which is exactly what that study found synthetic data gets wrong.

Pre-testing is a legitimate and cheap use. Running a draft instrument past a model can surface an attribute that only one brand can be rated on, or a screener that admits the wrong people, before any real respondent sees it.

Treat those outputs as editorial notes on the questionnaire, not as data. Researchers remain responsible for every change made to the instrument.

How Sprig Supports Competitive Benchmarking

The main fit is the frame. One study can reach a panel for every brand’s users, including yours, which is what the comparability rule requires.

The second fit is the fielding and analysis layer: white-labeled link surveys for a blind sponsor, AI Follow-Ups for the switching probe, theming for the open text, and the Design Agent for the first draft of the instrument.

Attribution matters here. Theming, AI Study Reports and AI Follow-Ups are Sprig agents, while intervals, the adjustment, the size line and any weighting are run by the connected AI client. The Analyze Agent is described as coming soon and documents none of those four.

What Happens Outside the Platform

Intervals, the adjustment, weighting and the size line happen outside, because the documentation publishes none of them. The Analyze Agent is described as coming soon and lists t-tests and ANOVA, so check its status before relying on it for the pairwise tests.

Recorded Task is documented for prototypes only, so a task-based benchmark against a competitor’s live product needs usability tooling built for that.

Panels documents no country coverage or turnaround, so confirm reach for each competitor’s users before planning cells.

Synthetic respondents aren’t offered, and nothing here replaces a researcher’s judgment on the competitor set, the comparison plan or interpretation. Researchers remain responsible for all three.

Alternatives and Adjacent Methods

A brand tracker when the question is whether awareness or consideration moved. Freeze a subset of the benchmark’s measures and field them in waves.

A win-loss study when the question is why specific deals were won or lost. It interviews buyers close to the decision, which a population survey can’t do.

A usage and attitudes study when the question is the shape of the whole category: who uses it, how often, on which occasions, and with which brands.

A task-based usability benchmark when the question is how well people complete tasks in each product. It needs participants working inside each live product.

Conjoint analysis or MaxDiff when the question is which features drive choice, or which of a long list matters most.

Message testing and concept testing when the question is causal. Both manipulate the stimulus, which a benchmark never does.

Market research tooling more broadly when the question is which platform should run the study. That guide compares vendors, and this one doesn’t.

Frequently Asked Questions

What Is a Competitive Benchmarking Survey?

A competitive benchmarking survey is a head-to-head measurement of your product against named competitors, using identical questions asked of comparable respondents in the same field window. It typically scores satisfaction, attribute performance, effort and likelihood to switch, with each brand rated by its own current users.

How Do I Run a Competitive Benchmarking Survey?

To run a competitive benchmarking survey, follow six steps. First, screen respondents on category use and ask which brand they use most. Second, recruit every brand’s users from one frame and mode. Third, set a quota for each brand cell. Fourth, route each respondent to one identical rating block for their brand.

Fifth, blind the sponsor and field every brand in the same window. Sixth, report only gaps that clear the adjusted interval.

How Is Competitive Benchmarking Different from Brand Tracking?

A competitive benchmark scores experienced performance against named competitors among each brand’s own users. A brand tracker measures awareness and consideration across the category in repeated waves. Run the benchmark to learn where you stand on the product experience, and the tracker to learn whether your brand’s position in the market is moving.

How Many Respondents Does a Competitive Benchmark Need?

A competitive benchmark needs enough respondents in each brand cell to detect the gap your decision cares about, adjusted for the number of comparisons. Detecting a 10-point gap on a 50 percent metric takes about 392 per brand for two brands. For five brands it takes about 557 per brand when you compare only your product against each competitor, and about 665 when every pair is compared.

How Many Competitors Should a Benchmark Include?

A competitive benchmark should include only the competitors the decision needs, because every added brand adds a cell to fill and more comparisons to adjust for. Three to five brands is commonly the working range, though it’s a convention rather than a sourced figure. The smallest brand typically sets the cost.

How Do I Recruit Competitors' Customers?

Recruit competitors' customers through an external panel, screened on category use and asked which brand they use most. Recruit your own users the same way through the same panel, rather than through your product or email list, so every brand is measured through one frame and one mode.

Should the Survey Reveal Who Is Sponsoring It?

A competitive benchmarking survey should blind its sponsor. A survey that announces it comes from one of the brands being rated invites respondents to answer toward that brand. No peer-reviewed estimate of the effect’s size was found, so treat blinding as practitioner judgment, and note that in-product surveys can’t be blinded.

Why Do Big Brands Score Highest on Every Attribute?

Big brands score highest on most attributes in an all-aware ratings grid because attitudes largely follow usage, and more respondents use big brands. Have each brand rated by its own users, and read perception scores against the level expected from each brand’s share of users rather than as a raw ranking.

How Do I Know if a Gap between Brands Is Real?

A gap between two brands is real enough to report when a two-proportion z-test clears a significance level adjusted for your planned comparisons. The Bonferroni adjustment divides 0.05 by the number of comparisons, so three comparisons use about 0.017. A gap with an unadjusted p-value of 0.045 can still fail that level, and a gap whose adjusted interval crosses zero is reported as not distinguishable.

Can I Compare My In-product Survey Results with a Competitor's Panel Results?

In-product results for your users and panel results for a competitor’s users can’t be compared fairly, because brand and survey method are confounded. Respondents answering inside a product they’re using are in a different context from panel respondents. Field every brand, including yours, through the same panel.

What if My Competitor’s Users Are Hard to Reach through a Panel?

If a competitor’s users are B2B buyers or a niche a consumer panel can’t supply, the benchmark can’t be run with that brand in the same frame. Check feasibility first by asking the panel vendor for incidence on your primary-brand screener. If a cell can’t be filled, drop that competitor or report it as not measured. Don’t fill it from a different source, because that confounds brand with recruitment method.

How Do I Present Benchmark Results to Executives?

Present a benchmark as one table of planned gaps, each with its adjusted interval, and label every other difference as not distinguishable. Lead with the two or three gaps that clear the adjusted level, state the limits of self-reported switching intent, and show the brand-size reading alongside raw scores. Don’t rank brands, because a ranking implies differences the sample can’t support.

Is There a Benchmark for Competitive Benchmarking Results?

There’s no published cross-industry benchmark for competitive benchmarking results, because the benchmark is the competitor set measured in the same study. Standardized instruments such as the System Usability Scale carry their own norms, but those norms describe products in general, not your category.

How Long Does a Competitive Benchmark Take to Field?

A competitive benchmark’s field time is often set by its smallest brand cell, because the largest brands typically fill first. A field window of two to three weeks is a common working choice, though it’s a practitioner convention. Run a soft launch of 20 to 50 completes before the full field, and keep the window as short as the smallest cell allows.

How Often Should a Competitive Benchmark Be Repeated?

No source found for this guide publishes a repeat cadence for competitive benchmarks. Refresh the benchmark when the category changes structurally, through a competitor launch, a major release, a pricing change or a new entrant, and keep the competitor set, frame and rating block identical so the two studies compare.

Can a Competitive Benchmark Include Usability Tasks?

A competitive benchmark survey can’t substitute for a task-based usability benchmark, which needs participants completing the same tasks inside each live product. MeasuringU recommends 213 participants per product for a between-subjects comparison or 93 in total within subjects, to detect a 12 percent difference. Run it separately and read the two together.

Can a Benchmark Support a Public Claim That We Are Better than a Competitor?

A competitive benchmark can support a public comparison claim only when the gap clears the adjusted interval and the method is stated beside the claim. State the frame, the field dates, the cell sizes and the comparison plan.

What Does a Competitive Benchmark Cost to Run?

A competitive benchmark’s cost is driven mostly by the number of brands, the target cell size and the share of category users the smallest brand holds. A brand used by 5 percent of the category needs about 4,000 qualified users screened to fill a 200-person cell, so the sample-size tables are the best early guide to scale.