Tuesday, November 13, 2012

So Gallup is Happy With it's 2012 Performance?

     Gallup thinks the knocks on its polling during this cycle are off base as, after all, it's final poll of likely voters was within the likely margin for error.

      "The Gallup organization on Tuesday defended its polling of the 2012 presidential election in a memo posted on its website. Frank Newport, editor-in-chief of Gallup and author of the memo, argued that the organization's final estimate of the national popular vote showing Mitt Romney with 50 percent to President Barack Obama's 49 percent was "well within the statistical margin of error."

       If Gallup is happy with its performance they're about the only ones who are. I guess it was welcome by the Romney campaign before they realized that their optimism and hope for not just winning but maybe even a landslide was totally misplaced.

      According to Fordham, they had one worst performance of the major polls this cycle:

       "Gallup's polling came under sharp scrutiny during the final stretch of the campaign for showing Romney with much wider national leads than other outlets. A study from Fordham University last week ranked Gallup as one of the least accurate pollsters in the 2012 cycle."

        Gallup, however, thinks being within the margin of error is something to write home about:

       "As our tradition has been in presidential election years, Gallup's focus this year was on producing an estimate of the national popular vote. We don’t “predict” the election, nor do we make estimates of the Electoral College. In the end, Gallup's national popular vote estimate was that the popular vote was too close to call, a statistical tie -- 50% for Mitt Romney, 49% for Barack Obama. When the dust settled, Romney got 48% of the popular vote and Obama received 50%, meaning that Gallup’s percentage-point estimate was within two percentage points for Romney and within one point for Obama. The “gap” difference was three points. All of these are well within the statistical margin of error and underscore the accuracy of random sampling today, even with all of the challenges provided by changing forms of communication (i.e., cellphones), changing demographics, lowered response rates, identifying likely voters, and a wide variety of other factors."

       Here I can't help but think of Keynes here who complained about neoclassical economists who "set themselves to easy and useless a task." If the point of a good poll is just having a poll that was within the margin for error, it's a pretty wide net. A good poll is not hard to find and so it's not clear why Gallup has the standing it has-at least until now. As Nate Silver suggests this may chance thanks to a performance like that:

       "Perhaps it won’t be long before Google, not Gallup, is the most trusted name in polling."

        First of all, Gallup is kind of cherry-picking by only looking at it's last poll. Indeed, as Nate tells us, often the last polls don't give us so much information as many of the outlier polls deliberately bring themselves back in within consensus. In the previous week and a half before the election-Gallup actually shut down the last week during Hurricane Sandy-it showed Romney with leads as large as 7-much wider than the other polls, even Rasmussen. When you factor them in, their miss is considerably larger.

        Secondly, while their last poll may have been within the margin for error, the result also had the loser in the lead.

        Then again, they don't make predictions but we have others who make very accurate ones-like Nate as well as others who predicted the electoral college-Nate was 49 of 50 in 2008 and a perfect 50 for 50 this time.

         Even if you buy that being in the margin for error in your very last poll-where you've probably deliberately adjusted it win with the consensus that you were previously missing widely-it's not clear based on this what makes Gallup special, much less deserving the special reputation it currently enjoys.





  1. Also when all ballots are finally accounted, Obama is going to have well over 51% and a larger than 3% margin.

    Prediction: -1%
    Actual: 3%

    Well outside the margin of error...

  2. True as well. At this point how does Gallup justify its pedigree among polls?