I clicked on this story at Daily Kos to have a laugh about how unskewing isn’t just for conservatives any more and am very alarmed to find myself almost partly convinced. I confess that I have fallen into at least the near occasion of unskewing.
Can We Really Trust YouGov?
by Tyler Yeargain
If you have been following the midterm elections closely, you will know that YouGov has, three times now, dumped a bunch of surveys on every single Senate and gubernatorial election taking place this year. These surveys paint a mostly rosy picture for Republicans, and display some numbers that have not been picked up by other, more reputable pollsters. As someone who has taken all three of their midterm elections surveys, I feel as though I am in a good position to evaluate what exactly they are doing, and how effective it is.
Here’s how their latest survey (their third) began:
As you may have read, a group of Islamic militants referred to as ISIS, has taken control of parts of Iraq and Syria. Do you favor or oppose U.S. airstrikes against ISIS?
From what you’ve seen or heard, which political party comes closest to your view of what should be done about ISIS?
How important will the situation with ISIS and the Middle East be for your vote in the November midterm elections?
What ?!? YouGov asks issue questions before they ask horse race questions ? I thought that was an absolute no no.
In particular, it is very likely that thinking about ISIS causes some people to decide to vote for Republicans. First the shift towards more support for Republicans followed the ISIS on camera beheadings of James James Foley and Steven Sotloff. Second, reproducible psychological experiments show that people’s political views become more conservative when they are frightened.
update: I think that the fear-> conservative hypothesis is quite a big thing in the Psych literature. After a bit of googling I found this link to an abstract
It would be very important if a large number of polls prompted respondents to think of ISIS before declaring voting intentions.
I am going to redo some Huffington Post Pollster moving averages with and without yougov
Basically there isn’t a huge average difference (My sloppy unscientific estimate is a YouGov house effect +0.68% Republican.
However the impression of the probability of a Republican takeover is totally different. Without YouGov Alaska, Arkansas, and Louisiana move from at least lean Republican to toss up. Kansas isn’t close without YouGov, so it looks like a very likely Republican to independent shift.
Now to encourage you to waste a little of your time with my calculations, I note that I suspect a (Post beheading)(YouGov) effect. I am assuming that they are asking people outside of Florida about ISIS before asking about voting and that the distortion is due to a combination of the news and putting news before the horserace question. This means that a YouGov distortion might not appear as a YouGov house effect, since even if they were leading respondents before, they weren’t leading them to think of ISIS.
OK numbers after the jump
Alaska +2.8% Republican with YouGov +0.5% R without YouGov diff if drop +2.3% Dem
Arkansas +4.0% Republican with YouGov +1.2% R without YouGov diff if drop +2.8% Dem
Colorado +0.3% Republican with YouGov +0.9% R without YouGov diff if drop -0.6% Dem
Georgia +3.9% Republican with YouGov +2.5% R without YouGov diff if drop +1.4% Dem
Illinois -13.8% Republican with YouGov -11% R without YouGov diff id drop -2.8 Dem
Iowa +1.2% Republican with YouGov +1.7% R without YouGov diff if drop -0.5% Dem
Kansas -3.7% Republican with YouGov -7.4% R without YouGov diff if drop +3.7% Indep
Kentucky +5.2% Republican with YouGov +5.8% R without YouGov diff if drop -0.6% Dem
Louisiana +4.9% Republican with YouGov 1.9% R without YouGov diff if drop +3.0% Dem
Michigan -9.6% Republican with YouGov -8.2% R without YouGov diff if drop -1.4% Dem
N Carolina -3.6% Republica with YouGov -4.1% R without YouGov diff if drop +0.5% Dem
Hampshire -3.8% Republican with YouGov -4.2% R without YouGov diff if drop +0.4% Dem
Average effect of dropping hurts Republicans 8.2/12 = 0.68%
That’s not huge but I’m pretty sure it is enough to reject the null that the effect is just sampling error. Of course I have been wasting my time and yours. Someone serious has estimated the YouGov house effect. I got bored (or came to my senses) and stopped diddling with Huffpost Pollster.