A polling mode effect is a difference in results caused by how a survey was conducted rather than by who was asked. Put the same question to similar people by live phone, automated phone, text or an online form, and the answers can differ by several points. People tend to give more flattering or socially acceptable answers to a live interviewer, and they’re blunter on a screen.
It matters because today’s polls use many different modes. When two polls of the same race disagree, the method is often part of the reason.
How are polls conducted today? #
- Live telephone interviews. A person calls landlines and cell phones and reads the questions. This was long the standard, but it’s slow and expensive, and response rates are now in the low single digits. Our guide to how telephone polling works covers the details.
- Interactive voice response (IVR). Automated “robopolls” where respondents press keys. Federal law generally bars autodialed calls to cell phones without consent, so IVR-only polls lean heavily on landlines and skew older unless supplemented.
- Online panels. Some are opt-in, where people sign up to take surveys for small rewards. Others are probability-based: members are recruited from a random sample of addresses and given internet access if they need it.
- Text-to-web. A text message with a link to an online survey, which reaches cell phones without a live caller.
- Mixed mode. A combination, such as mail invitations to go online, followed by phone calls to people who don’t respond.
What causes a mode effect? #
Social desirability #
With a live interviewer on the line, people want to come across well. They’re more likely to say they’ll vote, report being satisfied with their lives and soften negative opinions. On a screen, with no one listening, those pressures fade.
Who each mode reaches #
Each mode reaches some people more easily than others. Younger adults rarely pick up calls from unknown numbers, while some older adults are underrepresented in online panels. Weighting corrects for a lot of this, but not all of it, and whatever’s left shows up as a mode difference. Our guide to non-response bias explains the underlying problem.
Hearing vs. seeing the question #
On the phone, people hear the choices read aloud and lean slightly toward the last option they heard. On a screen, they see all the choices at once and lean slightly toward the first. Online surveys that display “not sure” as an option also tend to get more undecideds than phone polls, where interviewers usually ask undecided respondents which way they lean.
How big are mode effects? #
The best-known study is Pew Research Center’s 2015 experiment, which randomly assigned people from the same panel to answer by phone or online. Across 60 questions, the average difference between modes was 5.5 points and the median was 5, with some gaps as large as 18 points.
The largest gaps involved political figures and personal satisfaction:
| Question | Phone | Web |
|---|---|---|
| “Very unfavorable” view of Hillary Clinton, all respondents | 19% | 27% |
| “Very unfavorable” view of Clinton, Republicans | 36% | 53% |
| Very satisfied with family life | 62% | 44% |
| Very satisfied with social life | 43% | 29% |
The pattern is consistent: people answering online were more negative about politicians and more candid about their own lives. That’s the direction you’d expect if some people soften answers for a live interviewer.
For approval ratings, the practical takeaway is that a pollster’s mode can shift its numbers by a few points in either direction. That’s one reason a single pollster’s approval number shouldn’t be compared directly with a different pollster’s.
Is the “shy voter” idea a mode effect? #
Partly, and it’s often overstated. After 2016 and 2020, some analysts argued that people who supported Donald Trump were reluctant to say so to live interviewers. Post-election studies found little evidence of large-scale lying to pollsters. The bigger problem was that some groups of voters were less likely to take surveys at all, which is non-response, not a mode effect. The idea echoes an older debate about the Bradley effect, which has similarly thin support in recent elections.
How do pollsters deal with mode effects? #
Most high-quality pollsters now use mixed-mode designs. A common approach starts with mail sent to a random sample of addresses inviting people to answer online, adds text or phone follow-ups for people who don’t respond, and uses live interviewers to reach groups that are still short. That combines broad coverage with the candor of self-completed surveys.
Mixing modes creates its own work. Pollsters weight the combined sample to match the population on age, education, race, region and often past vote. Some also model how answers differ by mode. Aggregators, meanwhile, watch for consistent differences between pollsters, which is how mode effects feed into what analysts call a house effect.
How should you read polls with mode effects in mind? #
- Compare like with like. To see whether a race is moving, compare a pollster’s new poll with its own previous poll, since the mode is the same.
- Find the method. Reputable pollsters say whether they used live phone, IVR, text, online panels or a mix, usually in the release’s methodology section.
- Expect more “not sure” online. A web poll with 12% undecided and a phone poll with 5% can describe the same electorate.
- Lean on the average. An average across pollsters with different methods washes out part of each mode’s quirks.
Election Tracker lists every published poll in a race with its pollster, sample size, population and field dates, next to a 30-day average. The cards don’t label the mode, but pollster releases open in an in-app browser, so you can check the methodology of any poll that looks out of line with the others.
Frequently asked questions #
What’s the difference between a mode effect and a house effect? #
A mode effect is caused specifically by how people were contacted and answered. A house effect is any consistent difference between one pollster and the others, which can come from mode but also from weighting, likely-voter screens and question wording.
Are online polls less accurate than phone polls? #
Not as a rule. Well-designed probability-based online panels and mixed-mode polls have strong records, and live phone polling has struggled with falling response rates. Opt-in online panels vary more in quality. Our guide to opt-in online polls covers the differences.
Why do phone polls show fewer undecided voters? #
Live interviewers are trained to follow up when someone says they’re undecided, usually by asking which way they lean. Online surveys that show “not sure” as a choice let people pick it and move on.
Do aggregators adjust for mode? #
Some model-based averages estimate each pollster’s consistent lean, which captures mode along with other method choices. Simpler averages don’t adjust, and rely on including many pollsters so the quirks partly cancel out.