CNN/ORC debate instant poll

To err is human to persevere is diabolical.  ORC and CNN have done it again.  Delighted by the success of their instant poll on the Presidential debate, they did an identical poll of the Vice Presidential debate. They report results for a sample of 381 (is that the smallest sample ever for a national poll which gets massive attention).  They used the same sampling strategy which yielded the unrepresentative sample of their presidential poll.  Respondents had to agree to respond twice and then actually respond a second time.

Survey respondents were first interviewed as part of a random
national sample on October 8-10, 2012. In those interviews,
respondents indicated they planned to watch tonight’s debate
and were willing to be re-interviewed after the debate.

Panel data are very useful, but the second wave of a panel with panel attrition over 50% is not a good cross section.  The interesting and worthwhile purpose of the ORC approach is to see how opinions changed.

But this polling memo is quite different from the one which caused CNN and ORC so much trouble.  The earlier polling memo showed results for some sub-samples, people in the South, people over 50.  The problem is that it didn’t show any results for the complementary sub-samples, because there were too few respondents for ORC to be convinced that the numbers were meaningful.  This made it clear that the second wave sample was, as one would guess given the reported huge rate of attrition, unrepresentative.

Learning from that mistake, this time they didn’t report how many people were interviewed in the first wave (so attrition can’t be calculated) and did not report any results for sub samples.

Last time I thought that there was interesting innovative polling, then a failure of communication leading to the mistaken presentation of frequencies from an unrepresentative sample.   This time, such a charitable interpretation is impossible.  Last time,  they presented data which showed that they had done something silly.  This time they decided to do the exact same thing and suppress the data which would show that it was still silly.  Deciding data are irrelevant because they convince readers that the poll is invalid is not at all OK: