05 November 2020

2020 Election Polling Had A Democratic Bias

UPDATED November 6, 2020.


The polling in the 2020 election was badly biased against Republicans and Trump, and in favor of Democrats and Biden, in particular. 

The pro-Democrat bias in the U.S. Senate polling was even worse (an average of 6.99 percentage points) than in the Presidential race (an average of 4.42 percentage points).

Of 51 state polls in the Presidential race, the national Presidential rate polling and polling in 35 U.S. Senate races, a total of 87 polling averages, 79 (91%) erred in favor of Democrats. 

Of the 87 polling averages, 54 (63%) were off by more than 4 percentage points (easily more than one standard deviation), with 98% of these large errors in favor of Democrats and the one large error in favor of a Republican coming in a race without a Democrat running. 

If the polls were not biased, you would expect a mean error of close to zero for 87 polling averages, about equal numbers of polls biased in favor of Democrats and of Republicans both in terms of any errors, and in terms of large errors (about 43-44 errors for each party), and 28 or fewer large errors (about 14 errors for each party).

It isn't obvious why this bias occurred in both 2016 and 2020 polling. 

The most plausible theory is that Trump supporters and Republicans disproportionately refuse to answer political polls, thereby omitting their support even after adjustments for demographic balance. This would imply a reduced response rate of about ten to twenty percent by Republicans relative to non-Republicans in the same demographic categories.

The Presidential Race

The final 538 blog 2020 election forecast was primarily a weighted average of lots and lots of good quality polls. These predictions were way off.

In the 50 U.S. state and District of Columbia predicting the outcome of the Presidential race, there was an average bias in favor of Biden relative to the actual results of the election was 4.42 percentage points. There were 45 errors in favor of Biden and 6 in favor of Trump.

Twenty-seven of the fifty-one polls were off by 4 or more percentage points, outside the one standard deviation margin of error. Of those outliers, all twenty-seven were in Biden's direction.

The popular vote prediction was an 8.0 percentage point lead for Biden. The actual outcome was a 2.8 percentage point lead for Biden. This may improve from a 5.2 percentage point error to as little as a 3 percentage point error as more votes are added to tallies from slow counting safe states like California.

All of the errors in favor of Trump were in states that were either as safely Republican as Utah, or as safely Democratic as Colorado. Every single battleground state had an error in favor of Biden. There were swing state errors in favor of Biden of 4 percent or more in Illinois (5.6), Michigan (5.2), Nevada (4.5), Wisconsin (7.7), Pennsylvania (4.5), Florida (4.5), Texas (4.4), Ohio (7.5), Iowa (6.7) and South Carolina (4.5). Pennsylvania may improve somewhat as vote counting continues there, but it will almost surely be off by more than 4 percentage points in favor of Biden when all of the votes are counted. The pro-Biden polling error was smaller in Arizona (0.2), Georgia (0.9), and North Carolina (3.1).

The U.S. Senate race

The 538 blog also made 35 polling average driven predictions for the 35 U.S. Senate races. The average Democratic bias in the polling was 6.99 percentage points. There were 33 errors in favor of Democrats and 2 in favor of Republicans.

Twenty-seven of the thirty-five polls were off by 4 or more percentage points, outside the one standard deviation margin of error. Of those outliers, one was in a Republican's favor (in a safe race with no Democrat against a third party candidate), and twenty-six were in the Democratic candidate's direction.

There were less than safe state polling errors in favor of Democrats of 4 percent or more in U.S. Senate races in the Georgia Special Election (6.0), New Mexico (7.0), Minnesota (7.0), Michigan (5.4), North Carolina (5.0), Iowa (5.2), Maine (9.5), Montana (6.8), Kansas (5.6) and South Carolina (8.3). 

The only U.S. Senate races reasonably in play which the polling got close to right on were Colorado (1.0 bias in favor of GOP), Arizona (1.0 bias in favor of the Democrat),  Georgia regular election (1.7 bias in favor of the Democrat), Texas (2.5 bias in favor of the Democrat), 


This is surprising. The factors usually associated with major polling errors weren't present in this case. 

The poll numbers were robust to errors limited to particular polling operations. They were weighted by methodology quality past partisan bias, by sample size, and by recency. In many cases there were eight or more recent, high quality polls that went into the averages.

There was not a strong third-party candidate or a lot of undecided voters except for one U.S. Senate race (Arkansas) which was off by a larger margin than any other Presidential or U.S. Senate race this cycle except the U.S. Senate races in Alaska and West Virginia.  None of those races were remotely close.

Polling was stable leading up to the election, not volatile as it had been in 2016, and there were trend lines in favor of Democrats and Biden in particular leading up to the election (except for Cunningham, the Democratic Senate candidate in North Carolina whose affair was revealed in an October surprise).

Turnout was higher than predicted (the highest since the year 1900 in any national general election), which should have favored a pro-Democratic adjustment in polling models.

Pollsters adjusted their methodologies after the major errors of 2016 in an effort to remedy the problems then, but those problems returned this year in many of the same places.

The explanation in bold below seems to be the most likely one.

The 2020 polling error “matches the pattern of the 2016 error really well, so there really does seem to be something wrong here,” explained G. Elliott Morris, a data journalist who runs the Economist’s election forecast, during a Wednesday postmortem on the “Science of Politics” podcast. “It’s not just two random polling errors.” . . . 

Researchers have largely ruled out the idea of “shy Trump voters” who lie to pollsters and say they’re undecided or that they favor someone else when they really favor Trump. But it’s possible, Grossman and Morris speculated, that pro-Trump, non-college-educated whites are simply less inclined to pick up the phone or participate in polls than pro-Biden, non-college-educated whites.

Because the pro-Trump cohort also tends to have less “social trust” — i.e., less “trust in other people or institutions,” as Morris put it. Spurred by Trump’s “fake news” mantra, participating in polls may have itself become politicized. When overall response rates are as low as 4 percent, this could skew the results against Trump in places like the Rust Belt or Texas. 

A similar dynamic may have also made it seem like more Republicans were flipping from Trump to Biden than ultimately did — again, because pro-Trump Republicans may be less inclined than pro-Biden Republicans to answer a pollster’s call or participate in an online survey. 

Other potential reasons for 2020’s big miss may have been beyond anyone’s control. It’s unlikely that late-breaking voters who decided within the last week made the difference, even though they told exit pollsters they favored Trump over Biden by 14 percentage points. There simply weren’t enough of them this year — just 4 to 5 percent of the overall electorate vs. about 14 percent in 2016 — to explain Trump’s overperformance on Election Day. 

A more plausible scenario, Morris said, is that a significant number of pandemic-induced mail ballots are either arriving late, or being rejected, or not being returned at all. If tons of people tell pollsters they’ve voted by mail and then, for whatever reason, some those “likely votes” don’t actually count on Election Day, it could widen the gap between the polls and the results. 

Political scientists and pollsters will debate these problems for years to come, and they’ll probably devise new approaches to deal with them. But after being told that Joe Biden could win in a landslide — and then watching as Trump beat his polls by even more than 2016 in state after state — the broader public might be more inclined to dismiss political surveys in the future. 

There are “systematic problems that they haven’t solved since 2016, and in fact seem to be worse this time,” Morris said. “That’s pretty troubling if you’re a pollster — especially if you’ve spent the last four years trying to reckon with the fact that polls were missing Trump supporters. So they have a big reckoning ahead of them.”

From Yahoo News. 

It may well be days, if not weeks, before the winner of the 2020 presidential race is decided, but one clear lesson from Tuesday night’s election results is that pollsters were wrong again. It’s a “giant mystery” why, says Nick Beauchamp, assistant professor of political science at Northeastern.

There are a number of possible explanations, Beauchamp says. One is that specific polls undercounted the extent to which certain demographics—such as Hispanic voters in specific states—shifted toward President Trump.

Another is that, just as in 2016, polls undercounted hard-to-reach voters who tend to be less educated and more conservative. Beauchamp is less convinced that “shy” Trump voters deliberately misrepresented their intentions to pollsters.

“Whatever the cause, it has huge implications not just for polling, but for the outcome of the presidential and Senate elections,” Beauchamp says. “If the polls have been this wrong for months, since they have been fairly stable for months, that means that campaign resources may have also been misallocated.” . . . 

This year’s polling errors, Beauchamp said, were “enormous” even compared to 2016, when polls failed to predict Trump’s defeat of Democratic nominee Hillary Clinton.

Indeed, just days before the presidential contest, FiveThirtyEight founder Nate Silver predicted that Biden was slightly favored to win Florida and its 29 Electoral College votes. The race was called on election night with Trump comfortably ahead. . . .

“I think Nate Silver and the other pollsters are saying ‘Well that’s just within the bounds of systematic polling,’ but it seems awfully large to me for just that,” Beauchamp says.

Now, election watchers and media leaders are questioning the value of polling overall. Washington Post media columnist Margaret Sullivan wrote that “we should never again put as much stock in public opinion polls, and those who interpret them, as we’ve grown accustomed to doing.”

“Polling,” she wrote, “seems to be irrevocably broken, or at least our understanding of how seriously to take it is.”

Polling misses aren’t unique to the United States, Beauchamp points out. Polls also failed to predict the 2015 elections in the United Kingdom, as well as the UK’s 2016 “Brexit” vote to exit the European Union.

In those cases, as with the 2016 US election, Beauchamp said, pollsters “made these mistakes, which are relatively small, but in the same direction and with fairly significant effects.”

To avoid a repeat of 2016 and 2020 in the United States, Beauchamp says, pollsters should shift their tactics—and perhaps attach different weights to factors they’re trying to measure, such as social distrust or propensity to be a non-voter.

“Hopefully they’re going to start modeling all of that information as a way to better capture these issues in voters,” Beauchamp says.
From here.

The bottom line is that distrust of the media has translated into distrust of pollsters, and that has made the polling work product  highly unreliable.

Pollsters take lots of classes in statistics and psychology (driven disproportionately by studies based on "WEIRD" samples), and not nearly so many in anthropology. 

But one of the core lessons in advanced anthropology fieldwork is that you need to win the trust of your informants from another subculture or culture to get accurate information from them. 

Pollsters haven't even begun to scratch the surface to figure out how to do that and until then, polls pertinent to our national culture war are going to be seriously flawed.


Morris said...

Balanced view but maybe missing the elephant in the room. First rule of statistical analysis is sufficient sample size.To think that 5% sample is enough is on the face of it wrong if not deceitful. Who and why should one spend a nanosecond on answering a poll. If the answer (a list + probabilities) is not obvious to you then no amount of discussion will convince you to consider the issue seriously.
My list's most probably is that pollsters do polls b/c they are paid and responders are disproportionnaly wishful thinkers/busibodies.
Maybe if you phrased the question as "what dicision will you make on a basis of a poll?" it might be easier but then you would have to want to be rational.

andrew said...


You are completely and absolutely wrong about how polling works. You are deeply wrong headed about this issue. Your inaccurate belief isn't uncommon but it lacks any and all merit and should be completely disregarded. If you had any meaningful understanding of probability and statistics you would realize how deeply screwed up your thinking is on this issue.

The relationship between random sample size and margin of error is established with 100% precision from pure mathematical analysis. The relationship between sample size and uncertainty due to random sampling error is non-linear. Increasing your sample size by a factor of ten, for example, increases the precision of your result by much less than a factor of ten.

The sample sizes in these polls are all large enough to have a very modest amount of random sampling error, particularly given that these are all averages of multiple polls, each independently with sufficient sample sizes, and are weighted by upon factors including sample size.

This is a case of sampling bias, not sample size. Sample bias is just as much a problem if your sample is 30% of the voting age population of a state as it is if your sample if a few hundred people. The problem is that the respondents are not selected in a manner that is truly random or a reasonable approximation of random, and instead, are biased in their response rates in a manner that undermines the goal of the polls. This is an intractable one but has nothing to do with an insufficiently large sample size.

I was a math major with a focus on applied math problems like these. I know what I'm talking about. You clearly do not.

Morris said...

OK, I mispoke the sample needs to be represntative which in turn depends on sufficient size to satisfy that. If you are testing ball bearing quality you will need to experiment with sample size first to calibrate. The math comes after that, applied knowlege then.
I do not respond to polls and busy people I know do the same. Activists that I know are engaged even for trivial things.
Thanks for replying but maybe you could have made your point with softer language. Probably I pissed you off on other topics. I have to admit to a strong bias/dislike: smart people claiming irrational certainty.

Morris said...

From pure curiousity do you think that poll accuracy failings support the idea that the sample size is accurate?
"The relationship between random sample size and margin of error is established with 100% precision from pure mathematical analysis" Do you think my ball bearing sampling statement is incorrect? Not looking for controversy just curious.

andrew said...

The 2020 polling data does not support a hypothesis that the sample size was inadequate. If small sample size was the problem there would have been much more variance between different polls of the same states than was observed. Instead, we saw the same systemic bias in all of the polls which were consistent with each other at a level consistent with the sample sizes used.

"If you are testing ball bearing quality you will need to experiment with sample size first to calibrate. The math comes after that, applied knowlege then."

This is incorrect. You can determine the sample size needed a priori based upon the level of quality control you desire. Incidentally, the industrial quality control context you reference, and in particular, the industrial quality control efforts of the Guinness Brewing Company in making beer, is where, historically, most of the statistical tools used in connection with polling statistics were developed.

"Thanks for replying but maybe you could have made your point with softer language. Probably I pissed you off on other topics. I have to admit to a strong bias/dislike: smart people claiming irrational certainty."

Yes, I could have been a bit softer in my language. I didn't notice that you'd ever commented before on any other topic. But, I have a strong bias/dislike of people who don't understand probability and statistics making inaccurate statements about it that are misleading and can cause others to go astray.