Pollsters weight survey data by giving each respondent’s answers more or less influence so the sample matches the population on traits like age, sex, race, education and region. If a poll ends up with too many college graduates, each graduate’s answer counts a little less and everyone else’s a little more. Most pollsters adjust for several traits at once using a method called raking, and many now also weight by party or how people say they voted last time.
Nearly every poll you see is weighted. Knowing how it’s done helps explain why polls taken the same week can disagree.
Why do raw polls need weighting? #
Because the people who answer surveys aren’t a random slice of the public. Response rates are in the single digits for most phone polls, and online panels are made up of people who chose to join. Some groups reliably respond more than others. Older people, women and college graduates tend to be overrepresented; younger people, men and people without a degree tend to be underrepresented.
If a pollster reported the raw numbers, the poll would lean toward the views of whoever answers most. Our explainer on non-response bias covers why that happens.
How is a weight calculated? #
For one trait, the math is simple:
Weight = share of the population ÷ share of the sample
Say a poll of 1,000 adults has 600 women and 400 men, but adults are split about 50-50:
| Group | Sample share | Population share | Weight |
|---|---|---|---|
| Women | 60% | 50% | 50 ÷ 60 = 0.83 |
| Men | 40% | 50% | 50 ÷ 40 = 1.25 |
Each woman’s answers now count as 0.83 of a response and each man’s as 1.25, so the weighted sample is half women and half men.
What is raking? #
Real polls weight on five or more traits at once. Fixing one trait can knock another out of line: correcting for education might throw off the age balance. Raking, formally called iterative proportional fitting, solves that by adjusting one trait at a time, then the next, and cycling through them again and again. Each pass changes the weights a little less, until the sample matches every target at the same time within a small tolerance.
The targets usually come from government data, such as the Census Bureau’s Current Population Survey or American Community Survey for adults, and state voter files for registered voters.
Which traits do pollsters weight for? #
Standard demographics: age, sex, race and ethnicity, education and region or state. Some also use urban-rural location, income or household size.
Education deserves special mention. After the 2016 election, the American Association for Public Opinion Research found that many state polls hadn’t weighted by education, even though college graduates were much more likely to respond and, that year, voted very differently from people without degrees. Those polls overstated support for Hillary Clinton in several Midwestern states. Education weighting is now standard among reputable pollsters.
Should polls weight by party or past vote? #
This is where pollsters disagree:
- Party identification. Some pollsters weight to a target share of Democrats, Republicans and independents. Critics point out that party ID is partly an attitude that can shift with events, so locking it in can hide real changes.
- Recalled vote. Many pollsters now weight to how respondents say they voted in the last presidential election, matched to the actual result. It helps correct for samples that tilt toward one party’s voters. The drawback is that people misremember or misreport their past votes, and some say they voted for the winner when they didn’t.
Neither choice is clearly right, and they can move a poll by a couple of points. That’s one source of the consistent lean between pollsters known as a house effect.
What does weighting cost? #
Weighting fixes bias but adds uncertainty. When a few respondents stand in for a large group, the poll becomes more sensitive to who those few people are. Statisticians measure this with the design effect.
A handy approximation: if a poll of 1,000 people has a design effect of 1.5, it has the precision of a random sample of about 667. The margin of error grows from about ±3.1 points to about ±3.8. A poll that reports only the simple margin of error for 1,000 people understates its uncertainty. Our guide to the margin of error explains the basic calculation.
Extreme cases are worse. If a poll reaches only a handful of young men without degrees and has to count each of them as 10 or 15 people, one unusual respondent can move the whole result. Good pollsters cap or trim weights to limit how much any one person counts, and they work hard to get a balanced raw sample before weighting.
What about likely voters? #
Weighting makes the sample look like the population; a likely voter model decides which part of it to count. Pollsters estimate who will vote using stated intention, past turnout from voter files and interest in the election. Two pollsters can weight the same raw data identically and still report different results because their likely voter models differ. See registered vs. likely voters for how that works.
What can’t weighting fix? #
- Traits you don’t measure. If people who answer polls differ from non-responders in ways that aren’t in the weighting, such as trust in institutions, weighting won’t catch it.
- Bad questions. Leading or confusing wording produces bad data no matter how it’s weighted.
- Groups the poll can’t reach. An online-only poll can’t represent people who aren’t online just by weighting the few it reached. We cover this in are online opt-in polls reliable?
- Fake respondents. Weighting a bogus answer just gives it more influence.
How do you judge a weighted poll? #
- Check the weighting variables. A reputable pollster lists them. Education should be on the list.
- Check the targets’ source. Census data and voter files are standard.
- Look for the unweighted sample size and, ideally, a design effect or a margin of error that accounts for weighting.
- Compare with other polls. An average smooths out differences in weighting choices across pollsters.
Election Tracker doesn’t reweight anyone’s data. It groups every published poll for a race with its pollster, sample size, population and field dates, and builds a 30-day average in which each pollster counts once per window, at its largest sample, with newer polls weighted more. That way one pollster’s weighting choices can’t dominate. It’s free on iPhone.
Frequently asked questions #
Why is weighting by education so important? #
College graduates are more likely to answer surveys, and in recent elections they’ve voted differently from people without degrees. A poll that doesn’t weight by education will lean toward graduates’ views, which is what happened to many state polls in 2016.
Can weighting fix a bad poll? #
No. It can correct a sample’s demographic balance, but it can’t fix poor questions, missing groups or fake respondents. Heavy weighting also makes a poll less precise.
What is raking in polling? #
It’s a method for weighting on several traits at once. The pollster adjusts the weights for one trait, then the next, and repeats the cycle until the sample matches all the targets simultaneously.
Do all pollsters weight by party? #
No. Some weight by party identification, many weight by recalled presidential vote, and some use neither. It’s one of the main methodological differences between pollsters.
What is a design effect? #
It’s a measure of how much weighting reduces a poll’s precision. A design effect of 1.5 means the poll is as precise as a random sample two-thirds its size, so its true margin of error is larger than the simple formula suggests.