|
Panels need warning labels |
|
By David Hill
|
|
Posted: 10/06/04 12:00 AM [ET] |
Public polling reached a new low last week when several influential research organizations passed off some dubious panel-back surveys as legitimate arbiters of who won the first presidential debate.
Panel-back surveys were conducted immediately after the debate by Gallup, the Los Angeles Times and ABC News with voters whom these pollsters had interviewed in the days before the debates. This pseudo-scientific perversion of polling is quackery at best and outright misleading at worst.
Polling has always been a mixture of science and art. And some pollsters, suffering from low self-esteem because their work often seems to lack methodological rigor, are drawn to the black art known in the trade as “quasi-experimental” research.
Lacking test tubes, plutonium or even white lab coats, these Mr. Wizard wannabes cook up ersatz experiments to reveal how opinions are formed.
Like most junk science, these studies don’t come with enough warning labels. While all of the experimenters last week disclosed vague details of their methodologies, none really told readers the complete truth. Even Gallup, the organization that was most forthcoming in issuing an advisory warning to readers, only advised its readers that there are problems with surveys conducted in a single night.
That’s like a pharmaceutical company warning about side effects of a potential drug, like vomiting or diarrhea, when the drug maker in fact knows the preparation doesn’t really cure whatever ailment it’s supposed to treat. The side-effects warning is appreciated, but hardly forthcoming. That describes Gallup’s understated cautionary note that “polls conducted entirely in one day, such as this one, are subject to additional error or bias not found in polls conducted over several days.”
What none of the pollsters acknowledged was that their pre-interviews potentially biased panel members, rendering them wholly unrepresentative of almost any real-life voters. The supposed allure of this sort of polling is that it reveals something about the effects on the whole electorate of exposure to an event such as a debate. But studies of voters who have been previously interviewed are worthless.
Considerable scholarly research demonstrates that simply being interviewed renders an otherwise normal voter abnormal. After being polled, voters are much more likely to seek out political information through the media, discuss politics with others and eventually to vote. The known effects are so great that in the earliest days of polling, voters would be screened at the outset of an interview to ascertain if they had ever been interviewed before.
Researchers felt it was appropriate to speak only with “virgin” respondents who had not been debauched by a prior survey instrument. I recall even reading a concerned musing by George Gallup about eventually running out of survey virgins. Because of potential pre-interviewing biases, it is routine even today for focus-group moderators to screen out voters who have recently been in another group.
The effects of the pre-debate interviews are only one threat to the validity of last week’s panels. Another problem was timing. In the East, the debate wasn’t even over until 10:30 p.m. Normally, professional practice forbids interviewing past 9 p.m. or 9:30 p.m. local time. Who agreed to wait up for an interview? Probably not people who had to wake up early the next day and go to work or get kids off to school.
Gallup was limited to night owls who might or might not have been representative of the whole East Coast population. Even in time zones where the post-debate interviews were conducted at a more reasonable hour, only those with a lot of time on their hands would have been able to take time for a poll after already spending 90 minutes watching the debate.
There are other problems, too, including self-selection to the panel (the Times acknowledged that Kerry voters were more likely to watch) and the pollsters’ inability to validate voter claims of having watched.
Hill is director of Hill Research Consultants, a Texas-based firm that has polled for GOP candidates and causes since 1988.
|
|
