Introduction
Run statistical analysis on survey data in four steps: fix the comparison before fielding, size the sample for the smallest difference that would change a decision, match the test to the question type, and report the difference with its interval. Sprig fields and exports the data, and its Analyze Agent is listed as coming soon for t-tests, ANOVA, and correlation, so it can’t run anything today.
Last updated October 2026.
Why Survey Results Need Statistical Testing
A survey result is an estimate from a sample, so every number you report carries error you can measure. When satisfaction moves from 61 percent to 54 percent between waves, the real question is whether that gap is larger than the noise two samples of that size would produce anyway.
Statistical analysis answers that question, and it answers it reliably only when you decide the question before you see the data. This guide walks through the full sequence: planning, sample size, test selection, calculation, interpretation, and reporting.
It also covers where the method misleads, including the debate over p-values that statisticians have been having since 2016. Every worked example uses illustrative numbers, labeled as such, so you can reproduce each figure yourself.
What Statistical Analysis of Survey Data Is
Statistical analysis of survey data is the use of inferential tests to judge whether a difference or relationship seen in a sample is likely to exist in the population the sample came from. Descriptive statistics summarize the responses you collected. Inferential statistics estimate what those responses say about everyone you didn’t ask.
Four terms carry most of the weight, and each one gets misread often enough to define here.
- A p-value measures how incompatible the data are with a stated statistical model, usually the model in which two groups don’t differ. It isn’t the probability that the hypothesis is true.
- A confidence interval is the range of values consistent with the data at a stated confidence level, usually 95 percent.
- An effect size is how big the difference is, in points of a percentage or a scale, independent of whether it’s statistically detectable.
- Power is the probability that a test detects a real difference of a stated size, given the sample.
The most common starting point is a cross-tab, the table that splits one question by another. A cross-tab analysis shows the numbers. The tests in this guide tell you which of the gaps in that table are worth believing.
How a Significance Test Works
Every test follows the same logic. It starts from a default assumption, called the null hypothesis, usually that the two groups don’t differ. It then asks how surprising your data would be if that assumption were true, given the sample sizes. The p-value is the answer: the probability of seeing a difference at least this large if the groups truly didn’t differ and the model’s assumptions held.
A small p-value says the data are hard to square with the default assumption. It doesn’t say the groups differ by a meaningful amount, and it doesn’t say your hypothesis is right. That distinction is the source of most misreadings later in this guide.
When to Use Statistical Testing
Use statistical testing whenever a number will drive a decision and the number comes from a sample. Five situations account for most survey work:
- Comparing two waves of the same study to see whether a score moved
- Comparing segments, such as plan tiers or regions, on one measure
- Choosing between variants in a product concept test or a message test
- Checking whether two measures move together, such as satisfaction and renewal intent
- Reporting a headline percentage with a margin that a skeptical reader will accept
Testing matters most when the stakes of acting on noise are high. Retiring a feature because one quarter’s score dipped is expensive. A product concept test that picks a winner by two points on a sample of 100 can easily pick the wrong one.
Testing matters least when the decision doesn’t depend on the answer. If you’ll ship the change either way, a significance test adds ceremony without adding information.
A Quick Check Before You Test
Ask three questions before running any test. Is there a decision that depends on the answer? Is the sample close enough to random that a test means something? Did you write the comparison down before you saw the data? If the answer to any of them is no, the test can still be useful, but the report needs to say which assumption is missing.
When Not to Use Statistical Testing
A significance test is the wrong instrument for five jobs that people routinely hand it. Each limitation below names the better instrument.
Cause: Use an Experiment
A significant gap between two groups doesn’t show what produced it. Customers who use a feature may be more satisfied because they were already more engaged. The better instrument is a randomized experiment, where chance decides who gets the change. Surveys can measure the outcome of an experiment, but they can’t create the random assignment.
Population: Use a Designed Sample With Weighting
Tests assume the sample is random. A survey answered by whoever clicked is a self-selected sample, and a p-value computed on it describes the people who answered, not the population. The better instrument is a designed sample, with weighting to correct known imbalances.
Importance: Use Effect Sizes and Decision Thresholds
With a large enough sample, a trivial difference becomes significant. Significance reports whether a difference is detectable, not whether it matters. The better instrument is an effect size with an interval, judged against a threshold you set before fielding.
Small Segments: Pool, Oversample, or Interview
A segment with a few dozen respondents produces intervals so wide that no test can separate it from its neighbors. The better instruments are pooling similar segments, oversampling the segment in the next wave, or following up with qualitative research. AI-moderated interviews belong to that qualitative category. They explain why a small group behaves as it does, and they aren’t a substitute for survey measurement.
Theme Shares: Validate the Codebook
When open-text answers are coded into themes, a test on theme shares inherits every coding error. The better instrument is a human-validated codebook, checked against a double-coded sample before any theme share enters a test.
Study Design: Confirmatory or Exploratory Analysis
The first design decision is whether you’re testing a question you wrote down before fielding or searching the data for patterns. The two modes need different rules, and mixing them is the most common way survey analysis produces findings that don’t hold up.
Confirmatory Analysis
Confirmatory analysis tests the few comparisons the decision depends on. You name each one before the survey launches, including the groups, the measure, the test, and the threshold. Because the comparisons are fixed in advance, each p-value means what it says.
Exploratory Analysis
Exploratory analysis scans many segments and measures to see what stands out. It’s a legitimate way to generate hypotheses. But every extra comparison is another chance to see a gap that’s only noise.
The arithmetic is unforgiving. If you run 20 independent tests at a 0.05 threshold and no real differences exist, the chance of at least one false positive is 1 minus 0.95 to the 20th power, or about 64 percent. With 10 tests it’s about 40 percent.
A Bonferroni correction divides the threshold by the number of tests, so 20 tests would need a p-value below 0.0025. The Benjamini–Hochberg procedure, published in 1995 in the Journal of the Royal Statistical Society, Series B, controls the false discovery rate instead, which is the expected share of false findings among the ones you report. It gives up less power than Bonferroni.
Which Mode to Choose
Choose confirmatory analysis when the result will drive a decision or appear in a report to leadership. Choose exploratory analysis when you’re deciding what to ask next. Label every exploratory finding as a hypothesis, and retest it on a fresh sample before acting.
A simple decision tree:
- Did you write the comparison down before fielding? If yes, run the test at your stated threshold and report it as confirmatory.
- If no, are you looking at more than one comparison? If yes, apply a multiple-comparison correction and report the result as exploratory.
- If you’re looking at one comparison you didn’t plan, report it as exploratory and say how many other comparisons you looked at before choosing it.
Recording the Plan
Store the plan where it can’t quietly change. A dated document with a version history works, and so does a note in your research repository. Record the date, the author, and any later edit with the reason for it. The point isn’t bureaucracy. A plan you can show a skeptical reader is what separates a confirmatory result from an exploratory one.
The Analysis Plan Sheet
Fill in this sheet before the survey launches. It takes little time and heads off most of the problems this guide describes.
| Field | What to write |
|---|
| Decision | The decision this analysis will inform |
| Primary comparison | Groups and measure, such as "Enterprise vs. Team, top-two-box on Q4" |
| Test | The test from the selection table later in this guide |
| Significance level | The threshold, set now (0.05 is a convention, not a law) |
| Smallest difference that matters | The gap that would change the decision, in points |
| Sample size target | From the sample size table, for that gap |
| Secondary comparisons | A short list, labeled exploratory |
| Segment cap | The most segments you'll examine, and the correction you'll apply |
| Exclusion rules | Which responses you'll drop, decided before you see outcomes |
Run this in Sprig The Analyze Agent is listed as able to recommend relevant analyses from a survey’s structure and study context, and to refine them through natural-language follow-ups. That speeds up the work of running tests. The limitation: Sprig’s pages don’t describe a multiple-comparison correction or a way to record which comparisons you planned in advance, so keep the analysis plan sheet outside the tool and log every comparison you run.
Writing Questions That Can Be Tested
A statistical test can only be as good as the question behind it. Four habits make survey questions testable.
Ask one thing per question. A double-barreled item such as "How satisfied are you with pricing and support?" produces a score nobody can interpret, so no test on it can be interpreted either.
Use the same wording, scale, and order every time you plan to compare. If you reword a question between waves, a change in the score could be a change in the question. A short note in the study file recording the wording version removes the ambiguity.
Define the group variable as carefully as the outcome. A comparison between plan tiers needs a reliable tier attribute on every respondent, whether it comes from your product data or from a screening question. Respondents who self-report their tier will misreport it at some rate.
Decide how you’ll score the answer before you collect it. For a rating scale, choose now whether the headline will be the mean, the share in the top two boxes, or the full distribution. Choosing after the data arrive invites picking whichever one clears the threshold.
Question Patterns and the Test Each One Feeds
| Question pattern | Typical wording | Result you report | Test family |
|---|
| Yes or no | "Did you use [FEATURE] in the last 30 days?" | Percentage answering yes | Proportions |
| Rating scale | "How satisfied are you with [EXPERIENCE]?" (1 to 5, all points labeled) | Top-two-box share, mean, distribution | Proportions or means |
| Single choice | "Which plan are you on?" | Share per option | Chi-square |
| Select all that apply | "Which of these do you use?" | Percentage of respondents per option | Proportions, one per option |
| Number | "How many times did you contact support?" | Mean or median | Means or rank tests |
Teams running a Net Promoter Score survey or a customer effort score survey already have fixed wording and scale, which makes the comparison design easier. The work is deciding in advance which scoring you’ll test.
Pretesting and Wording Checks
Pretest the instrument on a handful of people from the target audience before launch. Ask them to say in their own words what each question means, and listen for readings you didn’t intend. A question that two people interpret differently will produce a gap between groups that comes from interpretation, not from opinion.
Check the group variable the same way. If the comparison depends on a plan tier, run a few respondents through the screener and confirm that each one lands in the tier your records say they belong to.
Labels, Order, and Response Options
Label every point on a rating scale, not just the ends. Fully labeled points reduce the guesswork about what a 3 means, and they make the gap between adjacent points closer to equal, which matters if you plan to report a mean.
Keep response options exhaustive and mutually exclusive, and include a "don’t know" or "not applicable" option where a respondent could reasonably lack an opinion. Decide in advance how you’ll treat those answers. Most analysts exclude them from the base for a rating question and report their share separately.
Randomize the order of answer options that have no natural sequence, and keep scales in a fixed direction throughout. Flipping a scale partway through a survey is a reliable way to produce respondents who answer the wrong way by accident.
Scale and Scoring Choices
Rating scales produce ordered categories, and the choice of how to summarize them decides which test fits.
The mean treats the gap between a 3 and a 4 as equal to the gap between a 4 and a 5. That’s an assumption, and with fully labeled points it’s often a reasonable one. The top-two-box share, the percentage choosing the highest two points, avoids the assumption and reads the same way in a headline. The full distribution shows everything and tests nothing by itself.
Report at least two of the three. A mean that rises while the top-two-box share holds flat usually means movement in the middle of the scale, and a reader should see that.
Match the test to the summary. A difference in top-two-box shares is a difference between proportions, tested with a two-proportion test. A difference in means is tested with a t-test. If the distribution is badly skewed or the scale has only a few points, a rank-based test such as Mann–Whitney is better suited, though it answers a slightly different question than a difference in means.
Whether means on rating scales are legitimate at all is a live argument, covered in the critique section below. In short, one camp says no, and another shows that parametric tests rarely mislead on such data. Showing the distribution beside the mean sidesteps the fight.
For customer satisfaction surveys, where the five-point scale is standard, the top-two-box share is the usual headline for exactly this reason.
When the Mean and the Top Box Disagree
Two summaries of the same scale can point in different directions. Imagine one group where answers cluster at 4, and another where answers split between 5 and 1.
The two groups could share a similar mean while the second has a far larger share of very satisfied customers and a far larger share of very dissatisfied ones. A mean comparison would call the groups equal, and a top-two-box comparison might not.
When a result matters, look at the full distribution first and let it tell you which summary describes the situation. Then pick the test that matches the summary you report.
Testing Ordinal Data Directly
If you’d rather not treat the scale as numeric at all, a rank-based test compares the whole distributions. The Mann–Whitney test compares two independent groups, and the Kruskal–Wallis test compares three or more.
They answer a slightly different question, whether one group tends to give higher answers than the other, and they don’t report a difference in means. Use them when the distribution is badly skewed, the sample is small, or a reader will object to averaging an ordered scale.
Sample Size and Precision
Sample size decides two things you care about: how precisely you can estimate a single number, and how small a difference between two groups you can detect. They need different calculations.
Precision of a Single Estimate
For a percentage, the 95 percent margin of error is approximately 1.96 times the square root of p(1−p)/n, where p is the percentage and n the number of respondents. The worst case is p = 0.5. Solving for n gives n = 1.96² × 0.25 / E², where E is the margin you want.
| Margin of error (95 percent) | Respondents needed (p = 0.5) |
|---|
| ±10 points | 97 |
| ±7 points | 196 |
| ±5 points | 385 |
| ±3 points | 1,068 |
Derived, not sourced. Assumes a simple random sample, ignores finite-population correction, and ignores design effects from weighting or clustering.
Detecting a Difference Between Two Groups
Detecting a difference needs more respondents than estimating a number. For two independent proportions, the per-group sample size is:
n = (z_α/2 × √(2·p̄·q̄) + z_β × √(p₁q₁ + p₂q₂))² / (p₁ − p₂)²
Here p̄ is the average of the two proportions, q̄ is 1 minus p̄, z_α/2 is 1.96 for a two-sided 0.05 test, and z_β is 0.8416 for 80 percent power.
A worked example: baseline 50 percent, and you want to detect a 10-point difference to 60 percent. Then p̄ = 0.55, √(2 × 0.55 × 0.45) = 0.7036, and 1.96 × 0.7036 = 1.379. The second term is 0.8416 × √(0.25 + 0.24) = 0.5891. The sum is 1.968, squared is 3.873, and dividing by 0.10² gives 387.3, so 388 respondents per group.
| Smallest difference to detect | Per group, 80% power, baseline 50% | Per group, 90% power, baseline 50% | Per group, 80% power, baseline 30% |
|---|
| 10 points | 388 | 519 | 356 |
| 7 points | 796 | 1,065 | 713 |
| 5 points | 1,565 | 2,095 | 1,377 |
| 3 points | 4,356 | 5,831 | 3,763 |
Derived from the formula above. Two-sided test at 0.05, simple random sampling. The 0.05 level and 80 percent power are conventions, not requirements.
Read the table in two directions. Halving the difference you want to detect roughly quadruples the sample. And no table can rescue a design where the segment you care about is a small share of the audience, which is why segment sizes belong in the plan before fielding.
Adjusting for Design Effects
The tables assume every respondent counts as one independent observation. Two common features of real surveys break that assumption, and each one shrinks your effective sample.
Weighting is the first. A standard approximation, attributed to Kish, gives the design effect from weighting as n × Σw² / (Σw)². Take 1,000 respondents where 250 are enterprise accounts weighted at 0.4 and 750 are other accounts weighted at 1.2. The weights sum to 1,000 and the squared weights sum to 1,120, so the design effect is 1.12. Divide by it, and the sample behaves like 893 respondents.
Clustering is the second. When several respondents come from the same account, their answers resemble each other. The design effect from clustering is approximately 1 + (m − 1) × ρ, where m is the average number of respondents per cluster and ρ is the similarity among respondents in a cluster. With four respondents per account and a similarity of 0.2, the design effect is 1.6, and 800 responses behave like 500.
To plan for either effect, multiply the sample size from the table by the expected design effect.
Planning for Unequal Groups and Nonresponse
The table assumes two groups of equal size. When one group is much smaller, the smaller group sets the precision, and adding respondents to the larger group helps less than you’d expect. Plan the smaller group first.
Also plan for the people who start but don’t finish and the people who never respond. If you expect 40 percent of invitations to produce a usable response, you need to invite 2.5 times the sample you want. Treat that rate as a number from your own history, not a number from this guide.
What You Can Detect With the Sample You Have
Sometimes the sample is fixed, and the useful question is what it can detect. Rearranging the formula for a 50 percent baseline, the smallest detectable difference at 80 percent power is approximately (1.96 + 0.8416) × √(2 × 0.5 × 0.5 / n), where n is the per-group sample. With 200 respondents per group, that’s about 14.0 points. With 100 per group, it’s about 19.8 points.
Run this calculation before you promise a comparison. If the smallest difference your sample can detect is larger than any difference you’d plausibly see, the comparison isn’t worth running.
Why Don't These Tables Give a Minimum per Cell?
No rule such as "at least 30 per group" appears in this guide, because no source supports a single universal floor. The right minimum depends on the difference you need to detect. Use the table to set the target, and treat any cell far below it as too wide to test.
Real surveys usually need more than the table says. Weighting, clustering, and nonresponse all increase the sample needed for the same precision.
Run this in Sprig Sprig’s research panels can supply respondents when your own audience is too small to reach a target in the table. The cross-tab prompt in Sprig’s documentation flags cells below a minimum respondent count, and a cell with zero respondents counts as under the threshold. The limitation: the threshold is a value you set, with 10 used as an example in the documentation, not a statistical rule. Sprig doesn’t publish sample size guidance or a power calculator, so derive the target from the table above before fielding.
Audience, Targeting, and Weighting
Significance tests assume that each respondent had a known chance of being selected. Almost no business survey meets that standard, so the practical question is how far your sample departs from it and what you can do about the gap.
Three departures matter most. Coverage error occurs when part of your population can’t be reached, such as customers who never open the app. Self-selection occurs when the people who answer differ from the people who don’t. And imbalance occurs when your sample over- or under-represents a group, such as enterprise accounts that answer at a higher rate than small teams.
Weighting adjusts for imbalance you can measure. If enterprise accounts are 10 percent of your customers and 25 percent of your respondents, each enterprise response gets a weight below 1. Weighting can’t fix self-selection on traits you didn’t measure, and it increases the variance of your estimates, so weighted results need larger samples than the tables above suggest.
Plan segments the same way you plan the overall sample. If you’ll compare small segments, oversample them deliberately, then weight back to the population for any overall figure. A segment you didn’t plan for will arrive too small to test.
Delivery channel shapes who answers. In-product surveys reach active users in context and miss lapsed ones. Email surveys reach people who open email. Name the channel and its blind spot in your analysis plan, and don’t generalize past the people the channel could reach.
Weighting Arithmetic in Brief
A weight is the population share divided by the sample share. If enterprise accounts are 10 percent of customers and 25 percent of respondents, their weight is 0.10 / 0.25 = 0.4. The remaining accounts are 90 percent of customers and 75 percent of respondents, so their weight is 0.90 / 0.75 = 1.2.
Weighted results then count each enterprise respondent as 0.4 of a person and each other respondent as 1.2. Weighted tests need software that supports weights, because a plain test on a weighted file treats every row as one person and understates the uncertainty.
Run this in Sprig Sprig supports targeting by user attributes and events, so you can set up segment definitions that match the comparisons in your analysis plan. The limitation: the Analyze Agent page doesn’t list weighting among its methods, and Sprig doesn’t publish a weighting approach. Apply weights in your own analysis layer, and say in the report that the results are unweighted if you didn’t.
Fielding and Delivery
Fielding decisions change which tests are valid. Three of them deserve attention.
First, know whether your comparison is between independent groups or within the same respondents. Comparing enterprise and team customers uses independent groups. Comparing the same customers before and after a change uses paired data, and a paired test is the right one. Using an independent-groups test on paired data throws away information and loses power.
Second, don’t change the instrument mid-field. Reordering questions, rewording an item, or adding a screener part-way through splits your sample into two studies without telling you. Randomizing answer order is fine. It’s a design choice made before launch and applied to everyone.
Third, don’t stop fielding because the result looks significant. Checking a p-value repeatedly and stopping when it crosses the threshold inflates false positives. Set the target sample from the sample size table and stop there.
Adaptive probing deserves a note. When a survey uses AI follow-up questions that depend on each respondent’s answer, different respondents see different follow-ups. Treat follow-up answers as open text for theming, not as a measure you can compare across respondents.
Clustered Respondents
Business surveys often reach several people at the same company. Their answers aren’t independent, because they share a product, a contract, and a support history. A test that ignores the clustering treats 400 respondents from 100 accounts as 400 independent voices, and it reports more certainty than the data support.
Two fixes are common. Analyze at the account level, with one value per account, or use methods that account for clustering. Whichever you choose, record the account identifier in the data from the start.
Repeated Measures
When the same people answer in several waves, their responses are correlated across time. Paired tests use that correlation to detect smaller changes than independent tests could. But they need a respondent identifier that persists across waves and enough returning respondents to matter.
Timing and Cadence
Comparing waves is only valid when the waves are comparable. That means the same instrument, the same sampling frame, and a similar time of year when seasonality matters.
Set the interval between waves from the decisions it supports, not from a convention. A brand tracking survey that reports quarterly can’t detect a change that happens and reverses within the quarter. A weekly read on a small sample produces mostly noise.
When a series runs for a long time, test each new wave against the previous one for a signal of movement, and compare against a longer baseline for a signal of drift. A change that’s significant against last quarter but within the range of the previous four quarters may be ordinary variation.
Testing Waves: A Worked Rule
Combine two comparisons for each new wave. First, test the new wave against the previous one with a two-proportion or Welch test, as in the core calculations. Second, ask whether the change is unusual compared with your history.
Here’s an illustrative history. A score across eight waves reads 61, 63, 60, 62, 64, 61, 59, and 62. The seven wave-to-wave changes are +2, −3, +2, +2, −3, −2, and +3, with an average absolute change of about 2.4 points. A new wave that moves 2 points is ordinary movement for this series. A move of 8 points isn’t, even before you run a test.
Keep the history in a sheet with the wording version, the sample size, and the interval for each wave. Without it, every new result is judged in isolation.
Quality Control and Data Hygiene
Clean the data before you test it, and decide the rules before you see the outcomes. A rule chosen after seeing which exclusions help a result is a form of cherry-picking.
Check these in order:
- Partial responses: drop respondents who didn’t finish, or say why you kept them
- Duplicates: keep one response per respondent when a person can retake the survey
- Speeders and straight-liners: respondents who finish impossibly fast or give the same answer to every row
- Bots and fraudulent entries, especially on open links
- Impossible values, such as an age of 200
Report the number of responses before and after each exclusion. A reader who sees 1,000 collected and 840 analyzed will want to know where the 160 went.
Checking for Speeders and Straight-Liners
Speeders finish in a fraction of the median time, and straight-liners give the same answer to every row of a grid. Neither group is reading the questions. Flag them using a rule you set from the distribution of completion times in your own data, and record the rule in the analysis plan.
Then run the key comparison with and without the flagged respondents. If the conclusion changes, the exclusion rule is doing the work, and the report should say so.
Run this in Sprig The Analyze Agent is described as including built-in data cleaning before analysis. Sprig’s CSV export includes a nullable completedAt column, which identifies unfinished responses, and a visitorId column, which helps find repeat respondents. The captchaScore column appears only when bot detection is on for the account. Use these as the basis for your exclusion rules. The limitation: Sprig’s pages don’t describe the cleaning criteria, and the export documentation doesn’t address how retakes appear. Run your own checks on the export, log the counts, and don’t assume a cleaned dataset matches your exclusion rules.
Running the Tests: The Core Calculations
This chapter is the center of the guide. It covers which test to choose, how each calculation works, and what to report. The examples use illustrative numbers.
Choosing the Test
| Question pattern | Comparison | Test |
|---|
| Yes/no or top-two-box | Two independent groups | Two-proportion z-test |
| Yes/no | Same respondents, two moments | McNemar's test |
| Rating mean | Two independent groups | Welch's t-test |
| Rating mean | Same respondents, two moments | Paired t-test |
| Rating mean | Three or more groups | One-way ANOVA |
| Category by category | Two categorical variables | Chi-square test of independence |
| Two numeric variables | Strength of relationship | Pearson correlation |
| Ordinal or skewed | Two or more groups | Mann–Whitney or Kruskal–Wallis |
| Outcome and several predictors | Independent effects | Regression |
If you’re running a test your tool doesn’t offer, use your own statistics software or an AI client with code execution, as described later in this guide.
Comparing Two Proportions
Suppose a renewal-intent question in two waves. Illustrative numbers: wave 1 had 412 respondents and 252 chose a top-two-box answer, which is 61.2 percent. Wave 2 had 388 respondents and 209 did, which is 53.9 percent.
The pooled proportion is 461 / 800 = 0.576. The standard error is √(0.576 × 0.424 × (1/412 + 1/388)) = 0.0350. The difference is 7.3 points, so z = 0.073 / 0.0350 = 2.09, and the two-sided p-value is 0.037.
The 95 percent interval for the difference uses the simple unpooled standard error and runs from 0.5 to 14.1 points. For small groups, Newcombe’s Wilson-based interval is the better choice. The result is significant at 0.05, and the interval nearly touches zero. The honest report says renewal intent probably fell, by anywhere from almost nothing to about 14 points.
Now a smaller comparison. Illustrative numbers: segment A has 60 respondents with 27 in the top two boxes (45.0 percent), and segment B has 75 with 25 (33.3 percent). The gap is 11.7 points, which looks large in a dashboard. But z = 1.38, p = 0.166, and the interval runs from −4.8 to +28.2 points. The data can’t separate this gap from zero.
Comparing Two Means
For rating scales, Welch’s t-test compares two means without assuming the groups have equal variances. Delacre, Lakens, and Leys argued in 2017 in the International Review of Social Psychology that Welch’s test should be the default for independent groups, because it controls false positives better when variances differ and loses little power when they don’t.
Illustrative numbers: 152 respondents rated group A at a mean of 3.70 (standard deviation 1.13) and 154 rated group B at 3.39 (standard deviation 1.22). Welch’s test gives t = 2.31 on about 303 degrees of freedom, p = 0.022.
The same data, scored as top-two-box share, give 64.5 percent against 53.2 percent, an 11.2-point gap with z = 2.00 and p = 0.046. Both methods agree, but barely, and a report should give both with their intervals.
Comparing the Same Respondents Twice
When the same respondents answer at two moments, use a paired test. For a yes/no outcome, McNemar’s test looks only at the respondents who changed. Illustrative numbers: 200 respondents answered before and after a change. In 28 pairs the answer moved from yes to no, and in 46 it moved from no to yes.
The corrected statistic is (|28 − 46| − 1)² / (28 + 46) = 3.91, with p = 0.048. The pairs that didn’t change carry no information about the change, which is why they don’t enter the calculation.
For a rating, a paired t-test works on each respondent’s difference. Illustrative numbers: 120 respondents, an average change of +0.21 points, and a standard deviation of the changes of 1.0. The standard error is 1.0 / √120 = 0.091, so t = 2.30 on 119 degrees of freedom, p = 0.023.
Comparing Several Groups
When a rating is compared across three or more groups, ANOVA tests whether any group differs. It doesn’t say which. Follow it with pairwise comparisons, corrected for the number of pairs, as covered earlier.
For a categorical outcome across groups, use a chi-square test of independence. Illustrative numbers: 150 respondents on each of three plan tiers, with feature use of 84, 62, and 54 respondents, or 56.0, 41.3, and 36.0 percent. The chi-square statistic is 13.03 on 2 degrees of freedom, p = 0.0015. The result says the tiers differ somewhere, and the same follow-up logic applies.
Following Up an ANOVA
A significant ANOVA says that at least one group differs. To find which, run pairwise comparisons with a correction. Illustrative numbers: three groups of 90 respondents with mean ratings of 3.8, 3.5, and 3.3 and a common standard deviation of 1.1. The F statistic is 4.71 on 2 and 267 degrees of freedom, p = 0.010.
Tukey’s honestly significant difference for these groups is about 0.39 points. The 3.8 against 3.3 gap of 0.5 exceeds it, with an adjusted p-value of 0.007. The other two gaps, 0.3 and 0.2, don’t. So the evidence supports one difference, between the highest and lowest groups, and says nothing reliable about the middle group.
Measuring Relationships
Pearson correlation measures the strength and direction of a linear relationship between two numeric variables. For ordinal data or data with outliers, Spearman’s rank correlation is the safer choice. Correlation never establishes cause, and a significant correlation in a large sample can still be too weak to matter.
One-Sided Tests and Assumption Checks
Use a two-sided test unless you have a strong, pre-stated reason to care about a change in only one direction. A one-sided test makes it easier to reach significance, which is exactly why a reader will distrust one chosen after seeing the data.
Each test also assumes things. The proportion tests assume independent respondents and enough responses in each outcome that the normal approximation holds. The t-test assumes roughly bell-shaped differences, which matters less as samples grow. Check that no group has so few respondents that the calculation becomes unstable, and when one does, say so and use an exact method or skip the test.
Reporting Effect Size
Report the size of a difference in terms your reader uses. For percentages, the difference in points is the clearest, and the ratio helps when the base is small.
In the wave example, 61.2 percent against 53.9 percent is a 7.3-point difference and a ratio of about 1.14. For ratings, report the difference in scale points beside the standard deviation, so a reader can tell whether 0.3 points is large relative to how much respondents disagree.
Intervals for Percentages
Report an interval with every percentage. The textbook Wald interval behaves poorly, especially for small samples and percentages near 0 or 100.
Brown, Cai, and DasGupta showed this in a 2001 paper in Statistical Science and recommended the Wilson interval or the Jeffreys prior interval for small samples, and the Agresti–Coull interval for larger ones, though Wilson also performs well there. For segment A above, 27 of 60, the Wilson interval runs from 33.1 to 57.5 percent.
Run this in Sprig The Analyze Agent page lists crosstabs, t-tests, ANOVA, and correlation, and says calculations run on the pandas and SciPy packages, with the AI translating a question into the analysis. Against the test table above, that covers the t-tests, ANOVA, and correlation rows, and the crosstab starting point for the chi-square row. It doesn’t list proportion tests, McNemar’s test, rank-based tests, or regression. The limitation: the page labels the agent as coming soon and doesn’t say which t-test variant it runs, whether ANOVA includes follow-up comparisons, or which significance level and confidence level it uses. Confirm those settings before reporting any result from it.
Benchmarks and What a Good Result Looks Like
No source sets a cross-industry standard for how large a survey difference must be to justify a decision. The 0.05 line is a convention, and the American Statistical Association’s 2016 statement says a p-value alone shouldn’t decide one.
That leaves you building your own standard. Three habits do the work.
Set the smallest difference that matters before fielding, and write it in the analysis plan. A 2-point change in a satisfaction score may be irrelevant for a product with a stable customer base and decisive for one with a tight renewal margin. Only the decision can set the line.
Compare against your own history. A difference that’s large compared with how much the score moves between normal waves is more meaningful than one that sits inside that range. Keep a record of past waves on the same instrument and compute the typical wave-to-wave change.
Don’t borrow effect-size labels. Words such as small, medium, and large are conventions that vary by field, and a gap called small in one field can be decisive in another.
A good result, then, isn’t a p-value below a line. It’s an interval narrow enough to rule out the outcomes that would change your decision, around a difference large enough to matter.
A Worked Method for Your Own Threshold
Build the threshold in three steps. First, write down what the decision costs if you act on noise and what it costs if you miss a real change. Second, pick the smallest difference that would flip the decision. Third, use the sample size table to confirm the study can detect it.
Suppose a pricing page redesign has to lift a conversion-intent score by at least 5 points to justify the engineering cost. From the table, detecting 5 points from a 50 percent baseline needs about 1,565 respondents per group.
If you can only reach 400 per group, the study can’t answer the question, and the right move is to change the design or accept that the result will be a direction, not a verdict.
Interpreting and Acting on the Result
Four outcomes cover most results, and each calls for a different action.
| Outcome | What it means | What to do |
|---|
| Significant, and the difference exceeds your threshold | A real change large enough to matter | Act, and report the interval |
| Significant, but the difference is below your threshold | Detectable, but too small to change the decision | Report it, and don't act on it alone |
| Not significant, and the interval is wide | The data can't tell you | Collect more data before deciding |
| Not significant, and the whole interval sits inside a margin you set before fielding | Any difference is probably too small to matter | Treat the groups as equivalent for this decision |
The third row is the one teams misread most. "Not significant" doesn’t mean "no difference." It means the data didn’t resolve the question.
Three Worked Readings
Take the earlier examples through the table.
The wave comparison gave a 7.3-point drop with an interval from 0.5 to 14.1 points. It’s significant, and the interval is wide. If the decision threshold was 5 points, the data support a change but can’t say whether it crossed the line. The report should say that.
The small-segment comparison gave an 11.7-point gap with an interval from −4.8 to +28.2 points. It’s not significant, and the interval is wide. The data can’t tell you whether segment A leads, trails, or ties. Collect more respondents before drawing a conclusion.
The rating comparison gave a 0.31-point gap with p = 0.022. The result is significant, and the practical question is whether 0.31 points on a five-point scale matters to the decision. That’s a question about your threshold, which is why the plan sets one before fielding.
Communicating Uncertainty to Leaders
Leaders usually want a verdict, and a bare "not significant" gives the wrong one. Translate the result into plain statements about what you know.
For a significant result with a wide interval: "Renewal intent probably fell, by somewhere between about half a point and 14 points. We’d treat it as a real decline of uncertain size."
For a result that isn’t significant: "We can’t tell whether the two segments differ. The data are consistent with segment A leading by a lot, trailing slightly, or tying. A larger sample would settle it."
State the range first, then the plain-language reading, and keep the test name and p-value in an appendix for readers who want them.
Reporting Template
Use the same structure every time, so readers learn to find what they need:
- Comparison: the groups, the measure, and the wave or date
- Result: each group’s value and its n
- Difference: in points, with the 95 percent interval
- Test: the name of the test, the statistic, and the p-value
- Exclusions: how many responses were dropped, and why
- Status: confirmatory or exploratory, and how many comparisons were run
- Decision: what the result means for the decision, in one sentence
Put the interval and the decision in the first lines of any summary, and the p-value in the detail. Leaders who read only the top should come away with the size of the difference, not a verdict on significance.
A team doing experience measurement over time benefits most from this consistency. Each wave’s report becomes a row in a history that shows how much the score normally moves.
Running the Analysis With Claude or ChatGPT
An AI client with code execution can run every test in this guide on an exported file. The value is speed, and the risk is a number nobody can check. The prompts below are built to make the checking part of the work.
What the Analysis Produces
Each prompt returns a named table and a list of exclusions and disagreements. The first compares one measure between two groups. The second runs the same comparison across many segments and corrects for the number of comparisons. Cross-tab mechanics, such as building the table itself, belong to the cross-tab guide for Claude, so these prompts start from a clean export.
Prompt One: Compare Two Groups
What this prompt does: compares one survey measure between two groups and reports the matching significance test, the size of the difference, and its interval.
What it returns: a table named "Group comparison" and a short list of every exclusion and every disagreement.
Use the attached file [FILENAME, default: survey_export.csv]. The outcome column is [OUTCOME COLUMN, default: Q1]. The group column is [GROUP COLUMN, default: segment]. The groups are [GROUP A] and [GROUP B].
Use code to calculate this, not estimation.
Rules:
1. Exclude rows where the outcome or the group is blank, and rows where [COMPLETION COLUMN, default: completedAt] is blank. Report the count excluded at each rule.
2. If either group has fewer than [MIN GROUP, default: 10] respondents after exclusions, report "too small to test" instead of a result. A group with zero respondents also counts as under the threshold.
3. If the outcome is yes/no or a top-two-box share ([TOP BOX VALUES, default: 4 and 5]), run a two-proportion z-test. If the outcome is a rating mean, run Welch’s t-test. Say which test you ran.
4. Report each group’s n and result, the difference, its 95 percent interval, the test statistic, and the p-value.
5. Compare the p-value to [ALPHA, default: 0.05]. Don’t change the threshold after seeing the result.
6. Recompute the p-value by a different method: a permutation test that shuffles the group labels [PERMUTATIONS, default: 10000] times. Don’t reuse the first calculation.
7. If the two p-values differ by more than [TOLERANCE, default: 0.01], or any step couldn’t be done as written, report the mismatch and what you did about it. Don’t silently correct it.
8. Save the table as [OUTPUT FILE, default: group_comparison.csv].
The default for the minimum group size is a placeholder, not a statistical rule. Raise it using the sample size table.
Prompt Two: Compare Many Segments With a Correction
What this prompt does: runs the same comparison across every segment in one column and corrects for the number of comparisons.
What it returns: a table named "Segment comparisons" with raw and adjusted p-values, and a list of every exclusion and every disagreement.
Use the attached file [FILENAME, default: survey_export.csv]. The outcome column is [OUTCOME COLUMN, default: Q1], scored as the share answering [TOP BOX VALUES, default: 4 and 5]. The segment column is [SEGMENT COLUMN, default: segment]. Compare each segment with all other respondents combined.
Use code to calculate this, not estimation.
Rules:
1. Exclude rows where the outcome or the segment is blank. Report the count excluded.
2. If a segment, or the rest of the sample, has fewer than [MIN GROUP, default: 10] respondents, report "too small to test" for that segment. A segment with zero respondents also counts as under the threshold. Count only the segments you tested as the number of comparisons.
3. For each tested segment, run a two-proportion z-test and report both shares, the difference, its 95 percent interval, and the raw p-value.
4. Adjust the p-values with the Benjamini-Hochberg procedure at [FDR LEVEL, default: 0.05], and separately with a Bonferroni correction at [ALPHA, default: 0.05]. Report both adjusted columns.
5. Recompute the Benjamini-Hochberg adjusted p-values by hand in plain Python, with no statistics library, and compare them with the first result.
6. If the two routes differ by more than [TOLERANCE, default: 0.001] on any value, or any step couldn’t be done as written, report the mismatch and what you did about it. Don’t silently correct it.
7. Label every segment that survives adjustment as "exploratory" unless I listed it as planned: [PLANNED SEGMENTS, default: none].
8. Save the table as [OUTPUT FILE, default: segment_comparisons.csv].
What a Good Output Looks Like
A trustworthy result from the first prompt reads like a short report, with each number traceable to a step. A useful output includes a table with one row per group showing n, the share or mean, and the exclusions applied, followed by a line for the difference, its 95 percent interval, the test named, and both p-values from the two methods.
Below the table sits a list that states how many rows were dropped for each rule and whether the two methods agreed.
If the output skips the exclusion counts, shows a single p-value, or doesn’t name the test, send it back. The prompt asked for those items on purpose.
Segment-versus-rest comparisons overlap, because each respondent sits in one segment and in every other segment's comparison group. Treat the adjusted p-values as approximate, and say so in the report.
Setup and Getting Your Data in
Export responses as a CSV from your survey tool and attach the file. Code execution needs to be enabled in the AI client, so confirm that before you start. Look for evidence in the chat that code ran. A response with no code, from a prompt that says to use code, is an estimate.
Put the instructions after the data, and keep each file to the columns the analysis needs. Long files can bury instructions, and Liu and colleagues documented in 2024 in Transactions of the Association for Computational Linguistics that models use information in the middle of long inputs less reliably than information at the start or end.
Adapting the Prompts
Both prompts work on a flat export with one row per respondent. If your file has one row per answer, reshape it first so each respondent has one row and each question has one column. For select-all-that-apply questions, test each option as its own yes/no outcome and say so, because the options within one question aren’t independent of each other.
When you change a placeholder, change it in the analysis plan too. A threshold that differs between the plan and the prompt is how a report ends up using a different rule from the one you committed to.
Reading the Output
Read the exclusions first. Then check each group’s n against your analysis plan. Then read the interval before the p-value. If the output reports a p-value and no interval, ask for the interval.
What to Verify Before Reporting
Run the same prompt twice in fresh chats and confirm the numbers match. Recompute one result in a spreadsheet or your statistics tool. Check that the test named in the output matches the test in your plan. And check that the number of comparisons reported matches the number you chose to run.
Pitfalls
Models don’t give identical answers to identical prompts. Thinking Machines Lab reported in September 2025 that 1,000 temperature-zero completions from one open model produced 80 unique outputs. Hosted assistants can vary in similar ways, so produce any reportable number twice.
Models can also agree with you too readily. Sharma and colleagues documented sycophancy in assistants trained on human preference data in a 2024 paper at ICLR. Never ask a model to confirm the difference you hope to find. Ask it to test the comparison you planned and to report what it finds.
A model asked to double-check often reruns the same calculation and agrees with itself, which is why both prompts require a different route for the recompute.
Run this in Sprig Sprig’s cross-tab documentation says its example prompt was tested and validated with Claude and requires code execution to be on. It describes code execution as on by default for Team and Enterprise accounts, with Free, Pro, and Max users turning it on under Settings and Capabilities.
The same page uses 10 as an example of a minimum respondent count you set yourself. Sprig’s open-text analysis returns themes linked to their source responses and exports them alongside survey responses, so theme assignments can sit in the same file as your outcome columns. The limitation: Sprig’s documentation says themes regenerate as new responses arrive, and the tested example covers cross-tabs, not the significance tests above. Freeze a version of the codebook before you test theme shares, and see the guide on analyzing survey data with AI without losing rigor for validating the codes.
The Critique You Should Know
Two debates shape how careful analysts use the tests above. Both have a critique and a counter-critique, and both end in practical rules.
The P-Value Critique and Its Counter
In 2016 the American Statistical Association published a statement, written by Ronald Wasserstein and Nicole Lazar in The American Statistician, with six principles on p-values. Principle 5 says a p-value doesn’t measure the size of an effect or the importance of a result. Principle 3 says scientific conclusions and business or policy decisions shouldn’t be based only on whether a p-value passes a specific threshold.
Critics of the 0.05 threshold include a group led by Daniel Benjamin, who proposed in Nature Human Behavior in 2018 that the threshold for claiming new discoveries move from 0.05 to 0.005. Their argument keeps significance testing and tightens the bar.
The two positions disagree about the cure but agree on the disease: a bare threshold crossing is weak evidence. For survey work, the usable synthesis is to set your threshold in advance, report the interval and the size of the difference, and treat a result near the threshold as inconclusive.
One caution on scope. The Benjamin proposal covers new discoveries, and a business comparison between this quarter and last isn’t one. The principle that carries over is that a threshold crossing alone isn’t evidence.
The Rating-Scale Critique and Its Counter
Susan Jamieson argued in Medical Education in 2004 that Likert data are ordinal, so means, standard deviations, and parametric tests such as ANOVA are inappropriate. She cited the point that the average of "fair" and "good" isn’t "fair-and-a-half."
Geoff Norman argued the opposite in a 2010 paper in Advances in Health Sciences Education: that parametric tests can be used with Likert-type data and rarely lead to wrong conclusions. His case rests on how parametric tests behave in practice rather than on the measurement level.
The sensible response is practical. Report the full distribution and the top-two-box share beside any mean, and use a rank-based test when the distribution is badly skewed. If every presentation of the data agrees, the argument doesn’t matter. If they disagree, the disagreement is the finding.
What Both Debates Share
Both debates are about misplaced confidence. In one, a threshold gets treated as truth. In the other, a number gets treated as having more meaning than the scale supports. The remedies are the same: state your choices before you see the results, show more than one view of the data, and let readers see how much the conclusion depends on those choices.
The same logic applies when AI codes open-text answers, which the guide on analyzing survey data with AI without losing rigor covers.
Common Mistakes
Eight mistakes account for most weak survey statistics.
Testing Everything and Reporting the Winners
Running 40 comparisons and reporting the three that cleared 0.05 is the exploratory trap. Report how many comparisons you ran, and correct for them.
Reading "Not Significant" as "No Difference"
A wide interval means the data are inconclusive. Report the interval, and decide whether more data are worth collecting.
Reporting Significance Without Size
A p-value without a difference and an interval invites either overreaction or dismissal. Always give the size.
Testing Segments Too Small to Separate
Small cells produce wide intervals, and a large-looking gap inside one is mostly noise. Check the sample size table before you promise a segment comparison.
Changing the Threshold or the Scoring After the Fact
Switching from the mean to top-two-box, or from 0.05 to 0.10, once you’ve seen the result is how a null result becomes a finding. Write the choice in the plan.
Treating a Self-Selected Sample as a Random One
The test is arithmetic and the sample is the evidence. A perfect test on a biased sample describes the bias. This is the mistake most often made in a product concept test, where enthusiastic respondents are the ones who answer.
Ignoring Dependence Between Respondents
Several people from one account, or the same person across waves, aren’t independent observations. Treating them as independent overstates certainty. Record account and respondent identifiers, and analyze accordingly.
Forgetting That Weighting Changes the Math
A weighted dataset needs weighted calculations. Running an ordinary test on a weighted file counts every row as one person and understates the uncertainty. Use software that handles weights, and scale the sample size up by the design effect.
Synthetic Respondents and Statistical Testing
A synthetic respondent is a model-generated answer standing in for a person. For statistics, the problem is variance. A test needs the spread that real people produce. Bisbee and colleagues found in Political Analysis in 2024 that ChatGPT’s averages tracked a real national survey closely, but the responses showed less variation than the real ones, and small wording changes shifted the distributions.
Less variation narrows intervals and inflates apparent significance. A significant result from synthetic data tells you about the model, not about your customers. Use synthetic respondents, at most, to pretest question wording. Don’t test comparisons on them.
How Sprig Supports Statistical Analysis
Sprig is an enterprise survey platform powered by AI agents, and it covers the stages around the statistics. It builds and fields the study, reaches the audience, exports the data, and, once available, runs part of the analysis.
The Design Agent turns a research goal into a structured study, which helps with the question-writing stage. The Synthesize Agent groups open-text answers into themes linked to the responses behind them, which produces categories you can test once the codes are validated.
Panels, email, and in-product delivery set who can respond. The Analyze Agent is listed as coming soon, with crosstabs, t-tests, ANOVA, and correlation named as its methods.
Three gaps matter for this method. Sprig doesn’t publish a weighting approach, sample size guidance, or a power calculator, and it doesn’t document random assignment of respondents to study arms, which an experiment needs.
The Analyze Agent’s pages also don’t state a significance level, a confidence level, a multiple-comparison correction, effect sizes, or assumption checks. Until those are documented, treat any figure from the tool as one to confirm against your own calculation, using the prompts above or a statistics package.
Alternatives and Adjacent Methods
Statistical testing answers one kind of question, and other methods answer the rest.
When the question is which attributes drive preference, conjoint analysis estimates it directly, and MaxDiff ranks a long list of features or messages. When the question is price, the Gabor-Granger analysis guide covers a revenue-optimizing price check.
When the question is why a small group behaves as it does, AI-moderated interviews fit, and they’re a separate qualitative category rather than a survey method. When the question is cause, run a randomized experiment.
Choosing among them comes down to the decision. Use statistical testing when you need to know whether a difference in your own sample is real. Use a modeling method such as conjoint or MaxDiff when you need to know what people value, and an experiment when you need to know what causes a change. The three combine well.
A conjoint study can narrow the options, a test can confirm that a segment difference is real, and an experiment can confirm that a change produces the result you want.
Plan the sequence before you commit budget. A team that begins with qualitative interviews, sizes the themes with a survey, tests the segment differences that matter, and then runs an experiment on the change has used each method for what it does best.
Frequently Asked Questions
How Do You Know if Survey Results Are Statistically Significant?
Choose the test that matches the question, run it at a significance level you set before fielding, and compare the p-value with that level. A difference is statistically significant when the p-value falls below it.
Report the size of the difference and its 95 percent interval too, because significance alone doesn’t show whether the difference matters. The cross-tab guide for Claude shows how to build the table first.
What Sample Size Do You Need for a Survey?
It depends on the margin of error or the difference you need to detect. For a single percentage at 95 percent confidence, ±5 points needs about 385 respondents and ±3 points about 1,068. To detect a 10-point difference between two groups from a 50 percent baseline at 80 percent power, you need about 388 per group. These figures assume simple random sampling.
Which Statistical Test Should You Use for Survey Data?
Use a two-proportion test for yes/no or top-two-box shares, Welch’s t-test for rating means between two groups, ANOVA for means across three or more groups, a chi-square test for category-by-category tables, and correlation for two numeric measures. For badly skewed ratings, use a rank-based test such as Mann–Whitney.
Can You Use a T-Test on Likert Scale Data?
You can, with care. Critics such as Jamieson argue that Likert data are ordinal and shouldn’t get means or parametric tests, and Norman argues that parametric tests rarely mislead on such data. Report the distribution and top-two-box share beside any mean, and use a rank-based test when the distribution is badly skewed.
What Is the Difference Between Statistical and Practical Significance?
Statistical significance means a difference is unlikely to be noise. Practical significance means the difference is big enough to change a decision. With a large sample, a trivial gap can be statistically significant, so set the smallest difference that matters before fielding and judge the result against it.
Does a Non-Significant Result Mean There's No Difference?
No. It means the data didn’t resolve the question. A wide interval means you can’t rule out a large difference in either direction. A narrow interval around zero suggests any difference is too small to matter.
When Should You Not Use Statistical Testing?
Skip it when the sample is a convenience sample you can’t defend, when the groups are too small to separate, when no decision depends on the answer, when you’re exploring without a plan, or when the question is why people answer as they do. In each case, report descriptive results and say what they can’t support.
What Minimum Group Size Do You Need to Compare Segments?
There’s no universal minimum. The size depends on the difference you need to detect, so use the sample size table: a 10-point gap from a 50 percent baseline needs about 388 per group at 80 percent power. Smaller groups can still be described, but a test on them will usually be inconclusive.
Do You Need to Correct for Multiple Comparisons?
Yes, whenever you test many comparisons and report the ones that stand out. With 20 independent tests at 0.05 and no real differences, the chance of at least one false positive is about 64 percent. Use a Bonferroni or Benjamini–Hochberg correction, and label uncorrected findings as exploratory.
Can Claude or ChatGPT Run Statistical Tests on Survey Data?
They can run the calculations when code execution is on, and the result is only as reliable as the prompt and your checks. Require code, require a recompute by a different method, and produce any reportable number twice. The prompts in this guide are built that way.
What Is the Difference Between a Confidence Interval and a Margin of Error?
A margin of error is half the width of a confidence interval around a single estimate. A 95 percent interval of 54 to 66 percent has a margin of error of ±6 points. The interval is the fuller report, because it also covers differences and intervals that aren’t symmetric, such as the Wilson interval for small samples.
Should You Use a One-Sided or Two-Sided Test?
Use a two-sided test by default. A one-sided test is defensible only when you stated, before collecting data, that a change in the other direction would lead to the same action. Choosing the direction after seeing the data inflates false positives.
How Do You Test Whether Two Waves Differ?
Treat the two waves as independent groups if different people answered each one, and run a two-proportion test or Welch’s t-test. If the same people answered both, use a paired test. Then compare the change with how much your score normally moves between waves.
What Should You Report Alongside a P-Value?
Report each group’s size, the difference in points, its 95 percent interval, the test used, any exclusions, and whether the comparison was planned or exploratory. The American Statistical Association’s 2016 statement says full reporting and transparency are required for proper inference.
