How to improve polling? Ask voters who will win.
© Getty Images

A new national CNN/ORC poll made headlines Tuesday morning. This poll gives GOP nominee Donald TrumpDonald John TrumpHouse Democrat slams Donald Trump Jr. for ‘serious case of amnesia’ after testimony Skier Lindsey Vonn: I don’t want to represent Trump at Olympics Poll: 4 in 10 Republicans think senior Trump advisers had improper dealings with Russia MORE a marginal 1-point lead over Democratic nominee Hillary ClintonHillary Diane Rodham ClintonGrassley blasts Democrats over unwillingness to probe Clinton GOP lawmakers cite new allegations of political bias in FBI Top intel Dem: Trump Jr. refused to answer questions about Trump Tower discussions with father MORE among likely voters, and 2 points in a four-way contest including Libertarian Party nominee Gary JohnsonGary JohnsonGary Johnson: I don’t want to be president anymore ‘because of Trump’ Gary Johnson ruling out 2020 bid: 'It does boil down to two political parties' Court: Excluding outside parties from presidential debates does not violate First Amendment MORE and Green Party nominee Jill Stein. The results are thus in stark contrast to the organization's previous survey conducted in late July, right after the conventions, which had Clinton up by 9 points.

ADVERTISEMENT
Major swings in polling numbers are a well-known phenomenon in presidential campaigns. The question is whether such variations reflect true changes in people's vote intention and thus provide an accurate assessment of what is going to happen on Election Day. Research suggests that they don't.

In comparing the results of both polls, Philip Bump at The Washington Post's "The Fix" argues that the increasing support for Trump is due to a combination of three factors. First, Trump has consolidated support among Republicans and, second, he has improved his standing among independents. Third, ORC changed the composition of their sample: While the previous survey polled registered voters, the new survey included likely voters.

This methodological change may in fact be the most important reason for the discrepancy in the results of both polls. Recent research by Andrew Gelman at Columbia University suggests that apparent swings in vote intention represent mostly changes in sample composition.

In analyzing polls conducted during the 2012 campaign, the study shows that then-GOP nominee Mitt Romney's increased support after his first debate with President Obama was not a result of people changing their vote intention. Rather, Republicans were simply more likely to participate in the survey than Democrats, most likely as a result of Romney's good performance in that debate.

This might be what is going on with the CNN/ORC polls. The post-convention poll was conducted during Trump's feud with the Khan family, which may have reduced the likelihood of many Republican voters to participate in a survey and express support for their candidate.

Similarly, the latest survey overlapped with the FBI's release of documents about Clinton's use of a private email server while secretary of State, which may have decreased Democrats' willingness to respond.

The latest CNN/ORC survey does provide some support for this rationale: while 58 percent of Trump supporters said that they are extremely or very enthusiastic about voting, only 46 percent of Clinton supporters felt that way. Unfortunately, this question was not asked in the post-convention survey, so we do not know whether voters' enthusiasm for their candidates was different in late July.

Yet, there is another way to assess whether the recent CNN/ORC numbers reflect a true swing in favor of Trump. That is, we can look at people's answers to the vote expectation question.

Interestingly, when asked who will win the election, 59 percent of respondents said Clinton, compared to only 34 percent who thought that Trump would win. This is striking, given that the very same survey has Trump ahead in the vote intention question.

Pollsters rarely include the expectation question in their surveys. This is unfortunate, given that people's aggregated answers are among the most accurate methods available for predicting election outcomes. I analyzed the relative accuracy of the expectation question across the seven U.S. presidential elections from 1988 to 2012 by comparing their results to predictions from four established methods for forecasting elections — namely traditional intention polls, prediction markets, expert judgment, and "fundamentals-based" models developed by political scientists.

I found that citizens' expectations provided more accurate forecasts of election winners and the final vote shares than each benchmark method. Gains in accuracy were particularly large compared to intention polls: on average, expectations cut the error of intentions in half.

The figure below shows the results of intention polls and expectation-based two-party vote share forecasts (i.e., excluding third-party candidates) that were published during the last 100 days prior to the 2012 election.

The vertical axis shows Obama’s predicted lead in the two-party popular vote. The solid black line shows the final election outcome: Obama achieved nearly 52 percent of the popular vote and thus won by about 4 percentage points. The results from intention polls varied heavily and ranged from predicting a 3-point lead for Romney to an 18-point lead for Obama.

In particular, as the election came closer, a considerable number of intention polls saw Romney in the lead. In comparison, the expectation forecasts were much more stable, less extreme, and closer to the election result than individual polls. Expectation forecasts ranged from a 1-point to a 6-point lead for Obama and thus never predicted Romney to win.

In other words, even during times when many intention polls showed Romney in the lead, people still expected Obama to finally win the election.

Intentions and expectations prior to the 2012 U.S. presidential election

The results may surprise but there are good reasons why expectations should provide more accurate forecasts than intentions. Expectations incorporate more information than intentions.

At the very least, each expectation captures the respondent's own vote intention. More likely, however, the respondent's expectation also captures information about other people's intentions. For example, the respondent might have seen polling results and likely has some idea about how her family members, friends and colleagues will vote.

Furthermore, in contrast to the intention question, the expectation question captures responses from those who do not vote, who do not reveal how they’re voting, and who are still undecided.

Many of these people may have accurate expectations about the election outcome, since they are less influenced by partisanship and thus less subject to wishful thinking. Wishful thinking describes people's tendency to predict their preferred candidate to win, a behavior that harms individual forecast accuracy.

So, given that the expectation question provides such accurate forecasts, why do journalists widely ignore it when covering election campaigns?

One possible reason is that journalists are more interested in newsworthiness than accuracy. In order to satisfy the needs of the news cycle, journalists constantly look for interesting stories.

The large variance in vote intention polls makes it easy to generate news by simply focusing on who is ahead in the polls, emphasizing outliers, and linking the latest poll results to campaign events, a behavior that harms the quality of campaign coverage.

In contrast, expectations are more robust and less extreme than intentions, and they rarely change. If journalists would pay more attention to expectations, they might overcome the horserace mentality of campaign coverage and be able to focus on providing explanations for the relative performance of candidates, such as their positions on the issues and proposed policies.

Thus, focusing more on expectation-based forecasts has the potential to increase the quality of campaign coverage.

As a general rule, election observers should not pay much attention to a single poll, particularly if the poll is an outlier. I would even go one step further. Do not put too much faith in polls in general, even a polling average.

While aggregating polls does increase accuracy, polls tend to be among the least accurate methods to forecast elections until shortly before Election Day. Rather, forecasting research suggests combining forecasts from many different methods that use different information.

This is the approach we take in the PollyVote, which has provided highly accurate forecasts of U.S. presidential elections since 2004.

As of today, the PollyVote has Clinton ahead by more than 5 points and predicts her to win 347 electoral votes, compared to 191 for Trump.

Graefe is the leader of the PollyVote.com project on evidence-based election forecasting. He is currently a research fellow at the Tow Center for Digital Journalism at Columbia University's School of Journalism and at LMU Munich's Department of Communication Studies and Media Research. He also holds the endowed Sky Professorship at Macromedia University in Munich, Germany.


The views expressed by contributors are their own and not the views of The Hill.