The Silver Standard
Poblan Unico jamás será vencido
Extremely regular readers of AngryBear will have noticed that I became a guest contributor a while ago. Then I vanished. The reason is that I have become obsessed with Presidential Polling and don’t have much original to say about it. A dialog from last night between me and my 11 year old daughter
11) dad what are you thinking about
11) Obama right ?
11) why don’t you ever think about anything else
Now about polling. My favorite site (by far) if www.fivethirtyeight.com. Very very impressive. I call it the silver standard, because, Nate Silver (aka Poblano) being a Democrat, would never support crucifying mankind on a cross of gold.
fivethirtyeight simulates elections and calculates a probability distribution for electoral votes won. He (they?) consider(s) both state level and nation wide disturbances. They note that polls tend to narrow over time. They estimate pollster reliability with data from actual voting (mostly primaries I think).
As of recently he also estimates pollster fixed effects or house effects “the tendency of certain polling firms’ numbers to tend to lean in the direction of one or another candidate”. This happens to be very important largely because the most prolific pollster — Rasmussen — has, relative to other pollsters a tendency to lean pro-McCain. They also had excellent performance in the primaries and count extra.
My one concern about fivethirtyeight.com is that the calculations are very complicated and not at all transparent. I would like to see some reporting on a larger set of simulations done with different assumptions. For example, the house effects correction seems to me to be conservative and I would like to see simulations with a more aggressive correction.
3) The house effect adjustment is enacted only in cases where we are at least 90% certain that there is a house effect. Even in these cases, we hedge our bets a little bit, by subtracting 166% of the standard error from the house effect coefficient.
I would rather see with/without house effects and the with house effects estimates with just the point estimates no setting to zero if not significant and no subtracting 166% of a standard error.
Silver links to and praises an article on house effects in national polls written by
Charles Franklin at pollster.com (my second favorite site) . This is completely separate evidence of house effects, since fivethirtyeight’s raw data are state level polls. Rasmussen polls are significantly better for McCain than average polls.
much more after the jump.
OK here I just let go.
1. My problem with pollster.com is that their trend calculations are waaaay too complicated. They use a Loess trend estimate (trend value at t estimated with weights depending on how long before/after t the poll was taken then report the fitted value for time = t). This means that new data shifts past values of the trend line which freaks me out. It also means that they say Obama is ahead in Ohio because it is about tied now and he used to be behind so they are convinced there is a significant trend. Also the initial estimates downweight outliers (not explained exactly how). I do not agree with doing this (see below). The calculations might be optimal but they are much too complicated to understand. I’d rather a point estimate based on averaging (weighted regression on a constant and no trend) and an estimate of the recent slope with a standard error reported as a number). Still, since I can get the recent simple average from realclearpolitics.com I have no serious complaint (just one more url www.realclearpolitics.com to type I do *not* hotlist pollsters)
2. What happened to the Gallup anomaly ? For years I have been reporting on the Gallup likely voter anomaly — Gallup polls are better for Republicans than other polls. In the Pollster analysis, Gallup is much less pro-McCain than is Rasmussen and is 4th best for McCain. The reason is simple. The Gallup anomaly is an aspect of the Gallup likely voter filter and the vast majority of Gallup polls reported so far are from the Gallup tracking poll of registered voters.
As Gallup explains every 4 years, their likely voter filter is not reliable long before the election. Gallup has been forced by the competition to report likely voter results earlier than they used to (I remember way back when). There was a very large huge Gallup likely voter anomaly in the poll conducted July 25-27 (click and search for USA Today in which Obama lead among registered voters and McCain lead 49 to 45 among likely voters. The likely voter pool was strongly biased against the young compared to actual votes in past presidential elections.
Gallup has an excellent record predicting elections. What this means is that the last Gallup polls before the vote are very accurate. That is, the likely voter filter works in late October. This does not mean that it works in August.
What is going on ? It is simple. Admirably, Gallup has stuck with the same method they used long long ago. This is transparent and honest (they aren’t using their success in the past with one method to justify their claims based on a new method). It is almost comprehensible how they decide who is a likely voter. The filter is based on answers to simple questions. from the Gallup FAQ
“These questions include asking whether or not the individual knows the location of his or her voting place, whether or not the individual voted in the past election, how closely the person is following the election, and so forth.”
Now obviously knowledge of the location of the voting place in August and in October will differ — some people learn where it is between August and October. It is not surprising that someone who claims he or she will certainly vote but doesn’t know where to go to vote in late October is not likely to vote. In August, the answer has, I would suspect, much more to do with how long the voter has been registered to vote at his or her current address than the probability that he or she will vote. The use of that question creates a selection against younger voters (and people who moved recently) stronger than the correlation of age and not moving and voting.
Even the assumption that eligible adults who are not currently registered will definitely not vote is dubious in August. There is still plenty of time to register.
In any case the evidence that the Gallup filter works in October tells us little about whether it works in August (as Gallup insists whenever asked and often when not asked).
There will be Gallup likely voter polls. There will be complaints about the Gallup anomaly. Democrats will be alarmed at Gallups excellent record (based on late October polls). It is all very simple and right there on the FAQ.
So what’s with Rasmussen ? Here a key feature is that they assume that partisan inclination (Dem Rep Independent) changes slowly so that differences from poll to poll in partisan inclination are mostly noise. They reweight so that the partisan inclination matches the average over the past 3 months.
Like all polling firms, Rasmussen Reports weights its data to reflect the population at large. Among other targets, Rasmussen Reports weights data by political party affiliation using a dynamic weighting process. Our baseline targets are established based upon survey interviews with a sample of adults nationwide completed during the preceding three months (a total of 45,000 interviews). For the month of August, the targets are 40.6% Democrat, 31.6% Republican, and 27.8% unaffiliated. For July, the targets were 41.4% Democrat, 31.5% Republican, and 27.1% unaffiliated (see party trends and analysis).
Now there is no need to smooth that much given the sample size (especially because Rasmussen could use data from other pollsters on party affiliation). This also shows the difference between a report which is optimal for the pollster and one which is optimal raw material for meta-analysis (here just fancy talk for averaging across pollsters). Weighting to make party affiliation fit a target reduces noise (increases precision) at the cost of introducing possible bias (if true support for the parties has shifted over time). For one poll the optimum balance may be to weight. However if one averages many polls (many Rasmussen polls or many polls across pollsters who do the same thing) the noise averages out and the bias doesn’t.
Now Rasmussen polls could be corrected by taking a more recent average (also across pollsters) of party affiliation and then using the internals (really simple like 90% of Dems for Obama and 90% of Republicans for McCain and easily available) to calculated a desmoothed Rasmussen based number.
The fact that they are making a very strong, very dubious assumption and have results which are strongly significantly different from the average pollster should give Rasmussen pause.
Finally note how quiet times are good for McCain. In quiet times the averages across pollsters are dominated by the Rasmussen and Gallup tracking polls. They both are more favorable to McCain than the average poll. Some of the alarm (among Democrats hope among Republicans) about Obama’s vacation, McCain’s celebrity campaign etc is based on this (I don’t know how much).