The Stanford / Santa Clara county study
A bit of a post-mortem on my last post. Andrew Gelman discusses the Santa Clara county study here. He focuses on sample imbalances, selection bias, and especially the way the authors deal with the specificity of the test. See his post for the gory details, but he is quite critical of the study, to the point of asking for an apology from the authors for wasting our time:
I think the authors of the above-linked paper owe us all an apology. We wasted time and effort discussing this paper whose main selling point was some numbers that were essentially the product of a statistical error.
I’m serious about the apology. Everyone makes mistakes. I don’t think they authors need to apologize just because they screwed up. I think they need to apologize because these were avoidable screw-ups. They’re the kind of screw-ups that happen if you want to leap out with an exciting finding and you don’t look too carefully at what you might have done wrong.
I guess policy makers will have to rely on all those higher quality studies using more reliable statistical techniques and more accurate tests. There must have been dozens of them with at least a few done right.
I get the impression that this is like Romer’s measurement of the speed of light. It’s not very accurate, but it’s what we’ve got.