About $1.6 billion worth of political ads will grace our TV screens this year.

In many cases, the sponsors will have little idea how effective those ads will be before they air. 

Campaigns pay pollsters modest amounts to help develop compelling messages and then spend millions to air broadcast versions. A lot can get lost in the translation. The message may not be communicated in a compelling way; it may not convey real emotion or even be understood. Ad testing can ensure you spend media dollars effectively.

But using faulty methodologies can be worse than no testing at all. Focus groups are about the worst technique ever devised for this purpose. My aversion to focus groups for ad testing (and my concern about their over use in general) is not some theological issue, but derives from fundamental weaknesses in the method.

Focus groups make two fundamental assumptions: first that that the considered responses participants provide in the discussion, after some time for thinking, represent their “real” views and second, that people can accurately describe why they feel the way they do. Unfortunately, 40 years of research in psychology demonstrates the frailty, if not outright falsity, of those assumptions.

The very process of deliberating about their responses, as participants do when they think about what to say, can change people’s answers. In a cousin of Heisenberg’s uncertainty principle, the very act of putting thoughts into words can change the thought.

Moreover, people are very poor reporters of their own decisionmaking process. We can ask them how and why they react the way they do, but those answers often bear little resemblance to reality. One model experiment encapsulates a vast scientific literature.  Some subjects were hypnotized to feel disgusted at the mention of the word “take,” while others were not.  When those who had been hypnotized heard about a congressman who “takes” bribes they condemned him more vehemently than those who had not been hypnotized.

The kicker though, is that they were also more likely to find a student who “takes” interesting courses, morally suspect. Here an emotion affects judgments through a mechanism that is completely unknowable to the subject. Participants could not have told the moderator the true reason they felt they way they did. That did not stop them though. Some suggested that students who “take interesting courses” must be up to something nefarious. Of course, the un-hypnotized subjects did not find the student reprehensible, nor did they agree that he was up to something.

In short, these folks rendered opinions without knowing why and then offered a completely inaccurate explanation of their statements.

People often behave similarly in focus groups, reaching judgments without knowing why, then making up explanations after the fact. Yet, because we hear the explanations coming from sincere participants we take them as fact. In truth, during focus groups people are often telling us much more than they actually know.

People’s well-documented tendency to conform to group pressure also threatens the validity of focus group results. In famous experiments during the 50’s, Solomon Asch demonstrated that large numbers of subjects said an obviously shorter line was the longest on a page, just because everyone else in the group did. While some admitted that they did not want to seem peculiar, others revealed no awareness of the peer pressure that produced their wrong answers, attributing it to poor eyesight even when theirs was in fact perfect.

We see the impact of conformity in focus groups all the time. The problem is we have no way of knowing when we see it.

Test ads before they air, but use a better method than focus groups.

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982, including Sen. John Kerry (D-Mass.) in 2004.