In the wake of the surprising Brexit vote in the UK, headlines around the globe have declared potential lessons for the US. presidential election—populist movements across the world taking back their countries, an angry working class rejecting the status quo, and Donald TrumpDonald TrumpJudd Gregg: Trump gets his sea legs Week ahead: US raises pressure on WikiLeaks Trump's Hollywood Walk of Fame star defaced MORE at the center of the same sentiments here in the US.
Time will tell whether any of these prognostications actually prove true, but at least one real lesson for the US elections is already clear—many people (including the media) are caught off guard because they misinterpret polls. While the “leave” vote might have been a bit of a surprise, a careful review of the polls actually suggests that it was nearly always within the realm of possibility.
Several high-profile failures of British polling in recent elections created an environment in which pollsters were pressed hard to give an answer. But predicting a referendum vote can be difficult—particularly when you are trying to model the voting turnout and behavior of more than 30 million voters in an unprecedented election. Past voting history may provide a guide, but this will be nowhere near as definitive as in other elections. When viewed objectively, however, the “leave” and “remain” percentages were frequently within the margin of error—indicating a statistical tie.
Differences in results from telephone-administered polls versus internet-only polls created conflicting results. The telephone polls consistently tilted towards REMAIN, and the internet polls usually tilted towards “leave”. One key difference—most people answering telephone polls answered leave or remain, whereas the internet polls generally included an explicit third option (undecided). In many polls where the third option was available, the number of undecided was larger than the gap between “leave” and “remain”.
Pollsters frequently tried to allocate the undecideds. One poll actually assigned three out of every four votes to “remain,” introducing an obvious bias and skewing the results (although it’s true that many undecided voters choose the status quo). Similarly, when polls asked questions about the certainty of people’s positions, the possibility of changing minds was very real. For example, in the final IPSOS poll in the UK, 12% of respondents reported they could potentially change their mind—this being just days before the vote.
Voter turnout made a difference. A final look at the polls compared to the actual voter turnout shows that those who were most likely to vote “remain” included the young, single, and college-educated—all of which is consistent with the pre-Brexit polling. But these groups did not turn out in the same numbers as the pro-Brexit voters, who tended to be older, married with kids, and non-college-educated. Turnout is almost always key in elections—and the British pollsters’ turnout models were not quite right in terms of predicting which voters would actually show up at the polls.
So, what are the lessons for the US from the Brexit polls?
First, remember that how questions are asked will affect the results. For example, in the US, the treatment of third-party candidates in the polling questions could create similar issues as the “undecided” vote on Brexit did for British pollsters. It is important to pay attention to how results change when only Clinton and Trump appear in the polling, versus the inclusion of either a generic “other” candidate or, perhaps more importantly, a named third-party candidate.
Second, always guard against the tendency to overinterpret support for a candidate without considering the margin of error. It is easy to latch on to statements like “Hillary ClintonHillary Rodham ClintonWeek ahead: US raises pressure on WikiLeaks Poll: 85 percent of Clinton supporters would vote for her again OMB director: Government shutdown not a 'desired end' MORE leads by 10 points,” but without considering the margin of error (the degree of certainty we have in a statistical result) that is somewhat meaningless.
Third, unlike a referendum vote in the UK, the US Presidential election is based on the Electoral College. The Brexit vote was a popular vote across the UK’s government regions and Northern Ireland. Whichever side received the most total votes won. In the US, however, we have the Electoral College, which means there are essentially 50 state elections (51 with DC). Turnout of key demographics by state (and especially in swing states) is what will really matter. How a given poll models state-by-state turnout can really shape the results, and it’s worthwhile to look closely.
Polls are useful tools in trying to gauge the state of the electorate, but they have to be interpreted thoughtfully and carefully to give meaningful information. Brexit may have been a surprise, but perhaps a more objective view of polling would have shown it wasn’t quite the surprise that people thought based on the polling. We can avoid similar mistakes in gauging the US election by learning the lessons from the UK experience.
John H. Johnson, PhD is the CEO of Edgeworth Economics and the co-author of EVERYDATA: The Misinformation Hidden in the Little Data You Consume Every Day.