The Fox News GOP debate, and errors real and imagined

There are few errors more persistent than journalists’ and pundits’ use of the phrase “margin of error.” Even though scientific polling has been an important part of national election campaigns for decades and the importance of Big Data should have made us more empirically literate by now, mainstream media figures continue to misuse the term.  Recent examples of such misinformation are evident in commentary over which GOP candidates will make the ten-person cutoff for Fox News’ important first debate. 

Both Jon Stewart and Rachel Maddow, for example, mocked Fox’s criteria last week as being what the latter called “totally arbitrary,” based on what is ironically their “totally” mistaken understanding of margin of error.  Or more precisely “margins” of error, because that is essentially what exists – many different calculations of sampling error when there are several different options for respondents to choose from. So when George Pataki sits at less than 1 percent in the polls it makes no sense to say that with an overall survey margin of error of 3 percent, he may actually be as high as 4 percent. Trust me, he isn’t that high.

{mosads}Yet this is precisely the case made by Rachel Maddow on a recent segment of her show, saying of Pataki’s .4 percent: “If you factor in the best case scenario with the margin of error, maybe he’s at something more like 3.4 percent…That would put him above John Kasich and it would put him on the stage.” Sorry George but no stage for you.  Maddow went on to proclaim “I can do the math!”  Unfortunately she didn’t do the right math, so let’s do that now.

In a poll with several options like the GOP primary polls the overall margin of error is not very informative.  What really matters is the margin of error of the difference between the candidates, and this is not the same thing.  So to determine whether Pataki may actually be ahead of Kasich, we have to figure out the error around the difference between their polling numbers.  In the polls chosen by Maddow, Kasich has 2.6 percent and Pataki has .4 percent, so the difference is 2.2 percent.  The margin of error for the difference of their proportions (assuming a sample size of 450) turns out to be 1.6 percent, meaning that Kasich’s lead could be as much as 3.8 percent (2.2 percent + 1.6 percent) or as little as .6 percent (2.2 percent-1.6 percent).  The point is that under both extremes he still leads Pataki.  Maddow’s guest, Marist Polling Director Lee Miringoff, alluded to this distinction between margins of error near the end of their conversation, but unfortunately his comments probably did more to confuse than to enlighten.  

On “The Daily Show” Jon Stewart made a similar error to Maddow when he chastised Fox News by mockingly musing “what do you do when…I don’t know…the last four qualifying spots for your debate are locked in a statistical tie with the margin of error?”  It wasn’t clear which polls Stewart was using, but for the sake of demonstration, let’s look at the current Real Clear Politics average (as of 7/31).  Though Stewart’s statement was a bit confusing, I assume he meant that those candidates currently in positions 7-10 could just as easily be somewhere in between positions 11-16 and vice versa.  This would mean, for example, that Rand Paul in 7th place with 5.5 percent is actually trailing Rick Perry in 11th place at 2.2 percent.  But if we ditch the overall margin of error for the correct margin of error of the difference of proportions, we get a more accurate picture.  Treating the RCP average as one poll with a sample size of 450 (the average of the 6 polls), Paul’s lead of 3.3 percent is as high as 5.8 percent or as low as .8 percent, but in either extreme he still leads Perry and we can logically assume he leads the remaining field as well. 

But maybe Perry belongs as high as 8th place, currently occupied by Ted Cruz at 5.2 percent.  Nope, Cruz also holds a statistically significant lead.  Perry could be as high as 9th, but Kasich is almost certainly leading the rest of the candidates.  We can safely say that at least eight and probably nine of the top ten candidates have a legitimate claim to the stage based on the RCP average poll (note: this is a conservative test, as an average of several polls should yield even smaller margins of error) . 

All of this is not to say that polling is an exact science, and that none of the excluded GOP candidates will have a legitimate gripe.  This article only addresses sampling error, and there are several kinds of polling errors which could affect the outcome.  It is also perfectly reasonable for pundits to object to using such early polls as a meaningful criteria for inclusion.  But when prominent political pundits misinterpret sampling margin of error to misinform the public (and in mocking tones to boot) it is worth making a correction. 

Maddow began her segment with the GOP ranking based on her own particular choice of polls.  Not too surprisingly it included the exact same top ten as the RCP average (in slightly different positions).  Much could change in a few days, but when Fox News makes its decision we can probably expect there will be a statistically significant difference between most of the top ten and everyone else. What will almost certainly not be true is Maddow and Stewart’s belief that according to the polls, everyone will have earned a spot on the stage.  That will very likely be an error.

Liebertz has a Ph.D. in Political Science from Florida State and will be an assistant professor at the University of South Alabama this fall.

Tags Rand Paul Ted Cruz
See all Hill.TV See all Video