By Mark Mellman - 11/30/05 12:00 AM EST
After weeks of substance in this space, let’s switch to a nerdy discussion of technique.
The rash of inaccurate public polls before last month’s election provides reason to focus on an oft-cited culprit: declining response rates.
Fewer people respond to polls, the argument goes, and therefore polls grow less accurate. It’s one of those widely accepted pieces of received wisdom that has almost no basis in fact.
There is little doubt that response rates are declining. Even the very careful studies done by agencies like the Centers for Disease Control and Prevention saw response rates decline by over 20 points between 1993 and 2000. People are busier, more suspicious, tired of telemarketers, protected by caller ID, answering machines and screening capabilities. Though there is dispute about the precise numbers, between 1.5 and 4 percent of the public (not voters) only have cell phones and can’t be polled. A relatively small problem now, this one is likely to grow as 15 percent claim they will abandon their landlines over the next five years.
In analyzing the impact on political polls of what we call in the trade “non-response,” the key question is not how many people don’t take surveys but how different politically those voters we can contact are from those we cannot. To date, the evidence suggests not very. Surprising as it may seem, there is no scientific study that indicates inaccurate election polls result from non-response.
In fact, the evidence suggests no relationship at all. Say what you will about the exit poll (and much of what has been said is foolish), but exhaustive analysis indicates no meaningful relationship between the magnitude of error in precincts and voters’ willingness to participate.
Others have examined what happens when heroic efforts are made to increase response rates. A few years ago, analysts compared two otherwise identical surveys, one with a 36 percent response rate, the other with a 61 percent response rate. Most of the significant differences between the two were on demographics. Responses to attitudinal questions were nearly identical. Across all 91 items, including demographics, the average difference was about two points, well within the margin of error.
While complaining about low response rates, political commentators often ridicule the steps most likely to increase them. How many times have you read derogatory references to surveys in the field 10 or 20 days? In the context of fast-moving political campaigns, a poll done over the course of weeks seems ridiculous. But long field times are the single most effective way to increase response rates. Increasing the response rate in the survey above from 36 percent to 61 percent required keeping it in the field for eight weeks instead of five days.
Another approach to increasing response rates involves sending advance letters to respondents alerting them to expect a survey call from a reputable researcher. This method can increase response rates by three-13 percentage points. But a recent study by Yale’s Christopher Mann, working with The Washington Post and Quinnipiac polls in 2002, found that while the letters increased response rates somewhat, there was no evidence that the higher response rates produced more accurate forecasts of the election outcomes.
Indeed, extraordinary efforts to increase cooperation make surveys less accurate by bringing into the sample normally uninterested people who are much less likely to vote.
None of this should be read to suggest that methodology is unimportant or that we should forget about increasing cooperation. Even if noncooperation isn’t degrading results now, it may as the problem worsens. But the data do suggest that the conventional wisdom about non-response is wrong, for now. We need to look at a host of other factors, from faulty sampling to poor analysis, to find the villain.
In the end, all pollsters share the same respondents, but many of us regularly get it right.
Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982, including Sen. John Kerry (D-Mass.) last year.