By Mark Mellman - 04/07/09 05:57 PM EDT
Full disclosure: I was the Democratic pollster on the committee charged with producing the report, which was composed mainly of distinguished academics, including Census Director-designate Bob Groves. Professor Michael Traugott of the University of Michigan was the intellectual powerhouse, as well as the workhorse, behind the report.
Perhaps its paramount contribution is to remind us that producing an accurate poll is not as easy as it looks — lots can go awry. When things do go terribly wrong — as they did during New Hampshire’s Democratic primary, where not one late poll (including those of the candidates themselves) showed Hillary Clinton winning and where the public polls underestimated Clinton’s support by an average of nine points — we tend to look for a cause equal to the consequence in magnitude. One of the report’s implicit conclusions was that no one big factor by itself explains the debacle. Like most disasters, it likely resulted from an accretion of small causes.
Of course, even if that race-of-interviewer effect were eliminated, the polls would still have picked the wrong winner and would have underestimated the Clinton vote by four points instead of nine.
Much-accursed (by me) likely voter models were another culprit. Gallup’s likely voter model turned a five-point Obama margin in the full sample into a 13-point advantage among “likely voters.” This is not to say that such models are uniformly problematic, but rather that getting this wrong can have a powerful effect on accuracy.
Dry, technical aspects of the craft, like the number of callbacks to each household, are also implicated. Clinton voters were harder to reach, so polls that were in the field a short time and did not make a sufficient effort to reach respondents underestimated her vote to a greater extent, though correcting this too would not have yielded an accurate result, just a better one.
My own favorite explanation is not discussed in the report because it is un-testable. I call it the “Young/Stevens effect.”
Iowa polls underestimated Obama’s vote while New Hampshire polls nailed it almost precisely, undermining the so-called “Bradley effect.” However, in the days following the Iowa caucuses, Obama got gobs of positive publicity, while Hillary Clinton was the object of a torrent of hostile press. Voters may well have been reluctant to tell interviewers they were voting for the subject of such vilification.
While the circumstances were much different, the Senate and House races in Alaska exhibited a similar pattern. There, voters seemed disinclined to admit support for suspected felons. One late poll had Democrat Mark Begich ahead by 22 points in a race he won by less than one, and three of the last polls showed his margin over seven points. In the House, every single poll for months had Rep. Don Young (R) behind, by an average of over six points at the end, in a race he won by five.
One prominent consultant recently declared, “We are all pollsters now.” AAPOR’s catalog of potential pitfalls is testament to the fact that such statements betray more ignorance than competence.
Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the majority leaders of both the House and Senate.