Home | Opinion | Columnists | Mark Mellman

When polls go wrong

Last week, my friend and distinguished sparring partner in these pages, David Hill, commented on the American Association of Public Opinion Research’s (AAPOR) report on what went wrong with the polling in New Hampshire’s presidential primary. This compendium of potential polling problems is so important that I am going to add my perspective as well.

Full disclosure: I was the Democratic pollster on the committee charged with producing the report, which was composed mainly of distinguished academics, including Census Director-designate Bob Groves. Professor Michael Traugott of the University of Michigan was the intellectual powerhouse, as well as the workhorse, behind the report.

Perhaps its paramount contribution is to remind us that producing an accurate poll is not as easy as it looks — lots can go awry. When things do go terribly wrong — as they did during New Hampshire’s Democratic primary, where not one late poll (including those of the candidates themselves) showed Hillary Clinton winning and where the public polls underestimated Clinton’s support by an average of nine points — we tend to look for a cause equal to the consequence in magnitude. One of the report’s implicit conclusions was that no one big factor by itself explains the debacle. Like most disasters, it likely resulted from an accretion of small causes.

ADVERTISEMENT
One of those was a race-of-interviewer effect. Combining data from three surveys, the analysis found that respondents polled by a white telephone interviewer gave Obama just a one-point lead, while those interviewed by African-Americans gave him a much larger 14-point advantage — an effect that survives controls for a variety of other respondent demographics.

Of course, even if that race-of-interviewer effect were eliminated, the polls would still have picked the wrong winner and would have underestimated the Clinton vote by four points instead of nine.

Much-accursed (by me) likely voter models were another culprit. Gallup’s likely voter model turned a five-point Obama margin in the full sample into a 13-point advantage among “likely voters.” This is not to say that such models are uniformly problematic, but rather that getting this wrong can have a powerful effect on accuracy.

Dry, technical aspects of the craft, like the number of callbacks to each household, are also implicated. Clinton voters were harder to reach, so polls that were in the field a short time and did not make a sufficient effort to reach respondents underestimated her vote to a greater extent, though correcting this too would not have yielded an accurate result, just a better one.

My own favorite explanation is not discussed in the report because it is un-testable. I call it the “Young/Stevens effect.”

Iowa polls underestimated Obama’s vote while New Hampshire polls nailed it almost precisely, undermining the so-called “Bradley effect.” However, in the days following the Iowa caucuses, Obama got gobs of positive publicity, while Hillary Clinton was the object of a torrent of hostile press. Voters may well have been reluctant to tell interviewers they were voting for the subject of such vilification.

While the circumstances were much different, the Senate and House races in Alaska exhibited a similar pattern. There, voters seemed disinclined to admit support for suspected felons. One late poll had Democrat Mark Begich ahead by 22 points in a race he won by less than one, and three of the last polls showed his margin over seven points. In the House, every single poll for months had Rep. Don Young (R) behind, by an average of over six points at the end, in a race he won by five.

One prominent consultant recently declared, “We are all pollsters now.” AAPOR’s catalog of potential pitfalls is testament to the fact that such statements betray more ignorance than competence.



Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the majority leaders of both the House and Senate.

More in Mark Mellman

Mellman: What really happened in Maryland

Read more »