Non-response bias is the error that creeps into a poll when the people who agree to take it hold different views from the people who don’t. A low response rate alone doesn’t cause it. A poll can reach 2% of the people it contacts and still be accurate if that 2% looks like everyone else. The trouble starts when willingness to answer is tied to how people vote.
That’s the central problem of modern polling, and it’s behind most of the big misses of the last decade. Here’s how it works, how pollsters try to correct it, and how to spot it when a new poll lands.
What is non-response bias? #
The math is simple. The bias in any estimate is roughly:
non-response rate × the difference between respondents and non-respondents
Both parts matter, but the second matters more. If 98% of people hang up but the 2% who stay on the line vote exactly like the 98%, the bias is zero. If the two groups differ even a little, a high non-response rate magnifies it.
A worked example #
Imagine a state where Candidate A has 52% support and Candidate B has 48%. Now suppose A’s supporters are slightly more willing to answer: 2 in 100 complete the survey, versus 1.5 in 100 of B’s supporters.
The raw sample would show A at about 59% and B at about 41%. A four-point race looks like an 18-point blowout, because of a half-point difference in willingness to pick up the phone. Weighting can fix a lot of this, but only if the pollster weights on something that tracks the difference.
Why don’t people answer polls? #
Non-response is rarely random. It follows patterns that often line up with politics:
- Technology. Spam filters, call screening and carrier warnings keep many calls from ringing at all. Our piece on why polling response rates are falling covers this in depth.
- Engagement. People who follow politics closely are more likely to spend 15 minutes on a survey. Less engaged voters hang up, and they may vote differently.
- Trust. People who distrust the media, universities or institutions in general are more likely to refuse a survey from any of them.
- Schedules. Shift workers and people juggling jobs are harder to reach than retirees.
When any of these traits correlates with vote choice, the sample tilts.
Did non-response bias cause the polling misses of 2016, 2020 and 2024? #
It played a large part in all three, in different ways.
2016. The American Association for Public Opinion Research’s review found two main problems in state polls. Many didn’t weight by education, so college graduates, who answer surveys at higher rates and leaned toward Hillary Clinton, were overrepresented. And late-deciding voters broke toward Donald Trump. The review found little evidence that people lied to pollsters.
2020. Most pollsters had added education weighting, and the polls missed anyway. AAPOR’s task force called it the largest national polling error in 40 years. It couldn’t pin down one cause, but the leading explanation was non-response among Trump supporters that education and other demographics didn’t capture. Within the same age, race and education group, the people who answered surveys were more Democratic than the people who didn’t.
2024. The misses were smaller, but most swing-state averages again understated Trump’s margin. Pollsters who weighted by respondents’ recalled 2020 vote generally did better, which suggests they were partly correcting for the same problem.
How do pollsters correct for non-response bias? #
No pollster can force a 100% response rate, so they adjust the sample after collecting it. Our guide to how pollsters weight survey data goes deeper; here are the main tools.
Demographic weighting #
If census data says 15% of a state’s adults are men under 30 without a degree, but only 5% of respondents are, each of those respondents is counted about three times. This is usually done with a method called raking, which adjusts for several traits at once: age, sex, race, education and region.
Weighting by past vote or party #
Some pollsters ask how respondents voted last time and adjust the sample to match the real result. This targets political non-response directly. The catch is that people misremember or misreport past votes, often drifting toward the winner, which introduces its own error.
Better sampling and mixed modes #
Pollsters now combine phone calls with text-to-web invitations, mailed invitations and online panels. Pew Research Center, for example, recruits its main panel by mail from a random sample of addresses rather than by phone. Reaching people through more than one channel narrows the gap between who answers and who doesn’t. Mode affects answers too, which we explain in what a polling mode effect is.
Why weighting can’t fix everything #
Weighting only works on traits the pollster can measure and knows the true distribution of. It can make a sample look right on age, race and education and still be wrong if, within each group, the people who answer are more trusting, more engaged or more politically active than those who don’t. Pollsters can’t weight to a census figure for “trust in institutions.”
That’s why a poll can be carefully weighted and still miss, and why misses tend to point the same way across many pollsters in the same year.
How to spot non-response bias in a poll #
You can’t see a poll’s non-respondents, but you can look for warning signs.
- Check who was surveyed. Adults, registered voters and likely voters are different populations, and a poll of adults will often differ from one of likely voters.
- Read the methodology. Look for the weighting variables. A poll that doesn’t weight by education is a red flag after 2016.
- Look at the party mix. If the sample’s party identification is far off from recent results in that state, ask why.
- Compare it to the average. A poll far from every other recent poll isn’t necessarily wrong, but it deserves a closer look at its sample.
- Note the mode. Live-phone-only polls, online opt-in panels and mixed-mode polls each reach different people.
Election Tracker puts most of that checklist on the screen. Every poll card shows the pollster, sample size, population (likely voters, registered voters or adults) and field dates, and pollster releases open in an in-app browser so you can read the weighting details. The app’s 30-day average collapses repeated polls from the same pollster into one, at its strongest sample, so a single firm with a skewed sample can’t dominate a race’s number.
Frequently asked questions #
What’s the difference between response rate and non-response bias? #
The response rate is the share of people contacted who complete the survey. Non-response bias is the error that results if those people differ from the ones who didn’t respond. A low response rate makes bias more likely but doesn’t guarantee it, and a high rate doesn’t rule it out.
Can a poll with a 1% response rate be accurate? #
Yes. Pew Research Center’s studies have found that response rates are an unreliable predictor of accuracy, and many polls with single-digit response rates have been close to the result. What matters is whether the people who respond resemble the people who don’t, after weighting.
Does weighting eliminate non-response bias? #
No. Weighting corrects for measurable traits like age, race, sex and education. It can’t fully correct for traits pollsters can’t benchmark, such as social trust or political interest, when those traits predict vote choice.
Are online polls immune to non-response bias? #
No. Online panels have their own version: the people who join a panel and keep answering surveys differ from those who don’t. Opt-in panels add self-selection on top. We compare the approaches in whether online opt-in polls are reliable.
Is non-response bias worse in some races? #
Usually in lower-profile, lower-budget polling, such as House districts or local races, where pollsters have smaller samples and fewer resources to reach hard-to-reach groups. Midterms add another layer, because pollsters must also guess which respondents will actually vote.