MaxDiff Analysis with Claude: A Practical Guide

Example H2

Introduction

A plain prompt gets the ranking right, but misses part of the picture. Asked with no engineering, Claude correctly ranked all 12 items in a MaxDiff study and caught the more obvious of two anchor-question mismatches (a hygiene factor). It missed the second, opposite mismatch (a false differentiator) — not because of a wrong calculation, but because the summary statistic it reached for (a correlation, then a top-five overlap) structurally couldn't see that direction of disagreement. An engineered prompt, using a properly fitted choice model, bootstrapped confidence intervals, and independent recomputation, catches both. Code execution must be turned on for the verification step to run.

A note on what's real here: the "Ideal AI Tool" in this guide is a fictional composite, not a real product. The 400 survey responses are synthetic, built to behave like a real MaxDiff study. Nothing here is a real finding or real usage data — it's a worked example.

What is MaxDiff and how does it work?

MaxDiff (best-worst scaling) ranks a long list of items without asking anyone to rate each one on a 1-to-10 scale — a format most people grade generously and inconsistently. Respondents see four or five items at a time, pick which matters most and which matters least, and repeat with a different handful each round. The pattern of picks resolves into a full ranking built from real trade-offs, not everyone calling everything a 9.

Key mechanics:

Items are shown in small, rotating subsets (typically 4–5 per screen)
Each respondent completes multiple rounds (e.g., 12 rounds of 4 items)
The design ensures every item appears a set number of times across the sample
Picks are converted into a preference share via a choice model

When should you use MaxDiff instead of a rating scale?

Use MaxDiff when you have more candidate features, benefits, or messages than a single survey question can responsibly compare — and a rating scale would just confirm that respondents are generous. It answers one specific question: given a long list, which items win head-to-head when people are forced to choose? It does not tell you what price to charge or what people would trade off between two specific items.

The limitation to know before you start: MaxDiff is purely relative. A list of 12 mediocre features produces a confident 1st-to-12th ranking just as easily as a list of 12 genuinely important ones — because the method can only ever report what beat what, never whether anything matters much on its own. That's the gap an anchor question is built to close (see below).

What's the difference between base and anchored MaxDiff?

Base MaxDiff fits a choice model to best/worst picks and returns a preference share for each item — how much more one item gets chosen relative to the rest of the list. That's a complete analysis on its own, and it's where most MaxDiff studies stop.

Anchored MaxDiff adds one independent question per item: is it a "must-have," "nice-to-have," or "not important"? This directly fixes MaxDiff's blind spot — telling you not just what beat what, but how much anything matters in absolute terms.

Which version do you need?

Want only the base version? Skip the anchor paragraph in the setup and skip step 6 below. Steps 1, 2, 3, 5, 7, 8, 9, and 10 run a complete basic MaxDiff analysis on their own.
Want to compare across a segment (job level, company size, etc.)? That's step 4 — a separate, independent option addable to either version.

What data does this example use?

This guide's example uses 12 features and benefits for a fictional AI tool for work — things like accuracy, response speed, integrations, price, and security certifications. 400 respondents each worked through 12 rounds of four items, picking the one that mattered most and the one that mattered least each round, on a design where every item appeared exactly four times per person. Every respondent also answered the must-have/nice-to-have/not-important question once per item.

The dataset is synthetic, built from scratch for this guide. A real MaxDiff study already sitting in Sprig doesn't need to be exported first: Sprig MCP reads the choice tasks and anchor responses directly out of the live study, so the data checks in step 1 below run against what's actually there right now — not a snapshot from whenever you happened to export it.

What does a plain MaxDiff prompt get wrong?

A plain prompt gets the relative ranking exactly right but misses one of two possible anchor-question mismatches — not from a calculation error, but because the summary statistic it defaults to (a correlation coefficient, or a top-five overlap) can only see disagreement in one direction.

The prompt we used

We gave Claude nothing but the data and one plain question:

"I ran a MaxDiff survey for an AI tool for work, plus a follow-up question asking whether each feature is a must-have, nice-to-have, or not important. The data's attached. Can you run the analysis and tell me which features come out on top, and whether that matches what people are calling essential?"

What came back looked entirely reasonable: a ranked list, a comparison against the must-have answers, and a single correlation number tying the two together. Nothing looked wrong on a skim.

Did the ranking hold up?

Yes. Scored the simplest way — how often each item won "best" minus how often it lost "worst," as a share of how often it was shown — the order matched a properly fitted choice model exactly, all 12 items, same sequence.

The comparison half is where a single run isn't enough to trust. Correlating the score against the must-have percentage gave 0.668 — a number that reads as "fairly strong, generally aligned" on its own. But that number sits on top of a 12-row table where:

The correlation doesn't surface either row. Whether a plain read catches them depends entirely on whether someone scans the table instead of trusting the one summary number.

Did a second, independent attempt confirm it?

Yes — and it revealed something more specific than "the correlation missed things." A second attempt used plain Python instead of pandas, scored items by best-picks alone (not best-minus-worst), averaged all three anchor categories on a 2/1/0 scale instead of isolating the must-have share, and checked alignment with a top-five overlap instead of a correlation.

The ranking replicated a third way: best-picks-alone produced the identical 12-item order.
The comparison check caught data privacy (it appears in the anchor top five, not the relative top five).
It never had a chance to catch mobile access, in either direction: a top-five overlap can only surface items unusually high on one measure and low on the other — never an item that's merely respectable on relative preference while unusually low on the anchor measure. That direction is invisible to the method by construction.

Bottom line: both attempts caught privacy's pattern; neither caught mobile's — for two unrelated reasons specific to each method, not one shared mistake.

What prompt should you use for a rigorous MaxDiff analysis?

A rigorous MaxDiff prompt needs four things a plain ask skips: an explicit reference item (not a tool's alphabetical default), a rank-based comparison instead of a z-score, a stated magnitude threshold instead of significance alone, and independent recomputation of every number through a genuinely different code path. Below is the reusable template, broken into steps.

Setup block (fill in your own study before sending this):

I have best-worst (MaxDiff) survey data on [PRODUCT/SERVICE/CATEGORY].

The items tested were: [LIST YOUR ITEMS]

Each respondent saw [K] items per screen, across [J] screens, with each item appearing [approximately how many] times total. [DESCRIBE HOW THE DESIGN WAS GENERATED.]

The data is in [DESCRIBE FORMAT]. Columns are: [LIST YOUR COLUMNS —

use actual header strings, not question labels like "Q7"].

‍

[IF ANCHORED: I also asked respondents to independently classify each item as "must-have," "nice-to-have," or "not important." Compare relative MaxDiff importance against this absolute measure, separating real differentiators (high on both) from hygiene factors (low relative rank, high absolute rate) and false differentiators (high relative rank, low absolute rate).]

‍

[IF SEGMENTING: I also want to compare results across [SEGMENT VARIABLE].]

Step 1–2: Validate and clean your data, then fit the choice model

Load the data defensively: don't let the default reader treat category labels like "None," "NA," or "NULL" as missing values, in case any item or response category happens to be named that way. Check the choice data is internally consistent — every task should show exactly [K] items with exactly one "best" and one "worst," and the two should never be the same item. Confirm every item appears the expected number of times. Stop and flag any failures rather than modeling around them.

Then fit a sequential best-worst logit: "best" as a conditional logit over the items shown, "worst" as a conditional logit over the remaining items with utilities negated, after removing the chosen "best." This uses the full information in each task, not just the best pick. State explicitly which item is the reference (utility fixed at 0) — don't let a tool's default alphabetical ordering pick it for you.

Sanity checks before moving on:

Report fitted utilities with standard errors; confirm none are degenerate
Don't trust a convergence warning at face value in either direction — check the gradient norm and confirm a different starting point lands in the same place

Step 3: Calculate preference share

Each item's preference share is exp(utility) ÷ sum of exp(utility) across all items, as a percentage (should sum to 100%). Confirm this is unchanged if you refit with a different reference item — if it changes, that's a coding error to fix before continuing. If anchored, also calculate each item's anchor-response rate. Do this in code, not by estimation, carrying full precision and rounding only for final display.

Step 4: Segment your results (optional)

Before splitting by your segment variable, check the sample size in each segment. Flag any segment under [N, e.g., 30] respondents — including zero — as too small to model separately (they still count in the overall model from Step 2). Refit Steps 2–3 for every segment that clears the bar.

Step 5–6: Bootstrap and check for divergence

Resample respondents with replacement at the respondent level and recompute Steps 2–3 (2–4 if segmenting) from scratch on each resample, at least [N, e.g., 1,000] times. Report a confidence interval for each item's preference share (and anchor rate, if applicable).

If anchored, use the same bootstrap replicates to check divergence:

In each replicate, rank items by preference share and separately by anchor rate.
Use rank position, not a z-score — preference share is more right-skewed than a plain rate, so z-scoring and subtracting compares magnitude of standing, not order.
An item's divergence score = preference-share rank minus anchor-rate rank, averaged across all replicates (not computed once from the point estimate, which is prone to exact ties).
Sort items by |mean divergence| and find the largest gap between consecutive values; set the threshold at its midpoint. Divide that gap by the median of the other gaps and require it to clear 3x (a reasonable default) — unless the median of other gaps is zero, in which case any nonzero largest gap clears automatically.
Flag an item only if both hold: its confidence interval excludes zero, and its divergence magnitude clears the threshold. A CI excluding zero alone isn't sufficient — at typical MaxDiff sample sizes, even a trivial one-rank difference becomes statistically detectable.
State the direction: worse preference-share rank + better anchor-rate rank = hygiene factor. The reverse = false differentiator.

Step 7: Independently verify every number

Before finalizing anything, recompute every number through a second, genuinely different implementation — not the same code path run twice. Judge matches using both an absolute and relative tolerance together (|a − b| ≤ atol + rtol × |b|), not either alone. If a recomputed number doesn't match, report both numbers, the likely cause, and which you trust — don't silently correct it.

For bootstrap distributions specifically: if the independent implementation is markedly slower per fit, don't re-run every resample through it. Verify a random subsample of at least 10 resamples instead, and say plainly that it's a subsample check, not a full re-verification.

Steps 8–10: Present results, chart, and summarize

Step 8 — Results table: item, preference share (% with CI), anchor rate (% with CI), divergence score (with CI), and any flag from Step 6.
Step 9 — Chart, only if it adds something the table doesn't: a horizontal bar chart, one row per item, sorted by preference share, with a second bar for anchor rate, and flagged items labeled directly. Build it only from the Step 8 table — don't recompute for the chart. Drop the top and right border lines so annotations and error bars near the edge aren't cut through by a frame.
Step 10 — Summarize in 2–3 sentences, explicit about what the data actually supports versus what's directional but not something to bet on.

Setup requirement: this prompt depends on Claude actually executing code for every number, including Step 7's independent recomputation. Turn on Code execution and file creation (on by default for Team/Enterprise; Free, Pro, and Max users enable it under Settings > Capabilities). If your survey data lives in Sprig, Sprig MCP connects live surveys, responses, and themes directly — no export step needed, and Sprig's own announcement names MaxDiff as a supported analysis type.

What did the engineered MaxDiff prompt actually find?

Running the full engineered prompt found the same 12-item ranking as the plain prompt, but additionally flagged two items the plain prompt's headline output missed one of: data privacy as a hygiene factor and mobile access as a false differentiator, each backed by a bootstrapped confidence interval and a magnitude check.

The headline numbers:

Accuracy: 24.6% preference share (highest)
Price: 19.1% (second highest)
These two leading isn't surprising for a work tool — it's a sanity check that the model is behaving, not the finding itself.

The two flagged items:

Data privacy ranks 9th of 12 on relative preference — it rarely wins against flashier items like accuracy or price — but has the highest must-have rate in the study, 79%, edging out accuracy itself. That's a hygiene factor: nothing anyone gets excited comparing side by side, but disqualifying if absent.
Mobile access shows the opposite pattern: a real, respectable preference share (enough to beat weaker items head-to-head fairly often) paired with the lowest must-have rate of any item in the study. People will take it if it's there; almost nobody is asking for it.

Ten of the remaining twelve items move together as expected — whatever wins more head-to-head also gets called essential more often, with only a rank position or two of noise.

What mistake does a z-score comparison make with MaxDiff data?

Standardizing preference share and must-have rate with a z-score and subtracting the two produces false positives, because the two measures don't share a distribution shape. In an early pass on this dataset, that method flagged six of twelve items as mismatched — including the two most-liked items in the study, which should never register as a mismatch since they're rated highly on both counts.

Why it failed: preference share is a skewed measure (a couple of standout items pull the whole distribution up), while a must-have rate is comparatively even. Z-scoring both and subtracting compares how extreme a number looks, not how it ranks.

The fix: switch to rank position, which doesn't depend on either distribution's shape. But rank position alone still isn't enough — at 400 respondents, even a real one-rank difference is usually statistically detectable, so the gap also has to clear a size threshold before it counts as a finding rather than a coincidence of sample size. Data privacy and mobile access both clear that bar by a wide margin; nothing else does.

Plain prompt vs. engineered prompt: which gets MaxDiff right?

Both get the ranking right; only the engineered prompt catches both anchor mismatches and backs every claim with a confidence interval.

| | Plain prompt | Engineered prompt | |---|:---:|:---:| | Relative ranking | Matches the verified order exactly, across 2 attempts and 3 scoring methods | Matches, confirmed through a second, independent implementation | | Hygiene factor (privacy) | Caught by both attempts | Caught, backed by a CI and magnitude check | | False differentiator (mobile) | Missed by both attempts' headline output, for two unrelated reasons | Caught, flagged by name with the same statistical backing | | Statistical backing | None, on either attempt | Every flagged item clears a bootstrapped interval and an explicit magnitude threshold | | Reproducibility | Same ranking, same comparison gap, both attempts | Same data, same result, every run |

The plain version isn't wrong — both attempts got the ranking right and caught the more visible anchor pattern. What neither attempt did was notice it had only checked for one of two symmetric things, because neither had a reason to look for the second. A correlation and a top-five overlap each answer "does this generally line up" and stop there — neither asks whether a specific item is quietly failing to line up in the direction its own method can't see.

What should you remember when prompting Claude for MaxDiff?

Pair MaxDiff with a simple absolute-anchor question if you can. MaxDiff alone ranks 12 mediocre features with the same apparent confidence as 12 genuinely important ones. The anchor is what tells the difference.
Use rank position, not a z-score, when comparing two measures that don't share a scale. A preference share from a choice model usually isn't shaped like a plain rate.
A confidence interval excluding zero isn't the same as a gap worth reporting. At a few hundred respondents, almost any consistent difference clears that bar — pair it with an actual magnitude check.
Don't take a solver's convergence flag at face value in either direction. Check the gradient at the reported solution and confirm a different starting point lands in the same place.
A summary statistic can look complete while structurally missing half the story. Check what your chosen statistic can and can't see, not just whether it returns a reasonable-looking number.
Run the plain version of your own prompt once, honestly, before deciding the engineered version was overkill.

FAQ

Why not just ask Claude to run this directly, the way most people would? Both plain attempts in this guide got the relative ranking right and caught the more visible anchor pattern, but neither caught the other, for two separate reasons unrelated to each other. A plain ask returns something in seconds either way — whether it's something you can act on depends on whether you know which half of the picture you're missing.

Is this a Claude-specific problem, or would other AI tools hit the same issues? We only tested Claude, so we can't say for certain about others. The actual gap — a correlation and a top-five overlap each checking for only one direction of disagreement — comes from which summary statistics get reached for by default, not anything specific to Claude's training. Any tool or analyst using an off-the-shelf summary statistic on two differently-shaped measures would plausibly hit the same structural blind spot.

Do I need the calibration/anchor question for this prompt to work? No. The anchor question only touches the setup paragraph and Step 6. Skip both, and the rest of the prompt — fitting the choice model, computing preference share, bootstrapping, and independent recomputation — runs as a complete, standalone MaxDiff analysis.

Can I use this prompt with segments, like job level or company size? Yes. Step 4 flags any segment under a set threshold (including zero) and only refits the model for segments that clear it. This guide's own example runs the full 400-person sample as one group, since the point was demonstrating relative-versus-absolute importance, not segment splits.

Why fit a full choice model instead of just counting best-minus-worst picks? In our own test, a raw count got the ranking right too — best-minus-worst and best-picks-only both matched the properly fitted model exactly. The reason to use a real choice model isn't the ranking; it's what a raw count can't give you: a share that's provably a probability summing to 100%, a standard error you can bootstrap into a confidence interval, and fitted parameters you can check for degenerate values. A count getting the right answer on one dataset isn't a method guaranteed to.

Should I trust my optimizer's convergence flag? Not by itself, in either direction. Check the gradient at the reported solution and confirm a different starting point or algorithm lands in the same place. A solver can flag a correct answer as unresolved just as easily as it can wave through a bad one.

Why use rank position instead of a z-score to compare the two measures? Z-scoring assumes both distributions are shaped the same way. A preference share from a choice model is typically more skewed than a plain rate or percentage, since a couple of standout items pull it up — see "What mistake does a z-score comparison make?" above for what that did to our own numbers before we caught it.

Why isn't a confidence interval that excludes zero enough to call something a real finding? At a few hundred respondents, almost any consistent difference clears that bar — which tells you the difference is real, not that it's big enough to matter. Pair statistical significance with an actual magnitude check.

Why does the magnitude threshold use 3x specifically? 3x is a default, not a law. The prompt derives the actual cutoff from your own data — the biggest gap between consecutive sorted divergence scores — and 3x is the bar for calling that gap meaningful rather than just the largest of several similar-sized ones. If your data has a less obvious break, try a couple of different multiples and check whether the flagged items actually change.

How do you know the plain version wasn't quietly set up to fail? We ran it twice, independently, genuinely changing the approach each time. The first attempt used pandas, best-minus-worst share, and a correlation coefficient. The second used plain Python, best-picks-only, a weighted 2/1/0 anchor scale, and a top-five overlap. Both landed on the identical ranking and caught the same one of the two anchor patterns — if this were rigged to fail, two genuinely different approaches wouldn't have agreed this much.

Does this mean a quick, unengineered Claude analysis is never worth trusting? No. Both plain attempts got the harder half of this analysis — the actual ranking — exactly right, and both surfaced one real anchor pattern without being told to look for it. For a fast read on which features are winning head-to-head, that's genuinely useful. The gap shows up only when you need to know whether you're seeing the whole picture or just the half that happened to be visible to whichever method you reached for.