I grab a rare opportunity to criticize Paul Krugman. He is, again, discussing the roll out of Nate Silver’s new fivethirtyeight.com with concern bordering on dismay. Krugman’s concern is that he fears that Silver thinks that data are enough just by themselves without any theory (even the humble kind that calls itself a model). I actually share his concerns, but think the post is no good.
data never tell a story on their own. They need to be viewed through the lens of some kind of model, and it’s very important to do your best to get a good model. And that usually means turning to experts in whatever field you’re addressing.
Unfortunately, Silver seems to have taken the wrong lesson from his election-forecasting success. In that case, he pitted his statistical approach against campaign-narrative pundits, who turned out to know approximately nothing. What he seems to have concluded is that there are no experts anywhere, that a smart data analyst can and should ignore all that.
This is a critique of a straw blogger. The sloppy unstable over hasty data freak whom I will call quick-Silver. Quick-Silver has no familiarity with alleged experts other than “campaign-narrative pundits,” believes that all one needs are data (and presumably some universally valid statistical methods), and thinks the only possible mistake is to be a hedgehog not a fox (I think I am being fair to Krugman and not criticizing the made up strawman Krudeman).
update 2: Here is the Silver post which I didn’t find before. It is a devastating critique of the efforts by academic political scientists to forecast presidential elections using data on fundamentals. Krugman is right that Silver shows little respect for the expertise of academic political scientists. My claim is that Silver has justified that view with rock solid meta analysis (which, in my humble opinion, should be published in the American Political Science Review, not that they listen to me).
Krugman goes on to argue that there are valuable experts of whom quick-Silver is unaware
But not all fields are like that — in fact, even political analysis isn’t like that, if you talk to political scientists instead of political reporters. So, for example, before glancing at some correlation and asserting causation, you really should talk to the researchers.
Similarly, climate science has been developed by many careful researchers who are every bit as good at data analysis as Silver, and know the physics too, so ignoring them and hiring a known irresponsible skeptic to cover the field is a very good way to discredit your enterprise. Economists work hard on the data; on the whole you’re going to do better by tracking their research than by trying to roll your own, and you should be very wary if your analysis runs counter to what a lot of professionals say
Krugman clearly implies that Silver is confused, because he has had little contact with academic political scientists, climatologists and economists. Some data. Nate Silver has a BA in economics from the University of Chicago (roughly when the economics department included a total of 15 professors). Now I know undergraduates don’t get as much face time with professors as they should, but he does have some contact with prominent academic economics. I pass on climatology. Checking Wikipedia for the Chicago economics connection I read
Silver was born in East Lansing, Michigan, the son of Sally (née Thrun), a community activist, and Brian David Silver, a former chair of the political science department at Michigan State University.
I think Silver has met his dad.
Much more to the point, Silver is very familiar with the work of quantitative empirical political scientists. IIRC he tested the out of sample performance of all fundamentals based election forecasting models in the literature (including again IIRC ones presented only in working papers). He found it to be horrible. I think he has made animportant contrution to the literature which which Krugman seems to think heis unfamiliar.
I write IIRC because google isn’t sending me to the post I remember. Bleg how to I find posts which link back to the older post which I found, in which Silver finds that Hibbs’ model works poorly out of sample (using data on disposable income growth older than those Hibbs used).
Google also sent me to one criticizing a blogger, including “The issue with this model, and some others like it, is what’s known in the statistical business as overfitting. This occurs when the number of variables is large relative to the sample size: in this case … six variables, …used to explain only 15 cases.” Silver understands that correlation does not necessarily imply reliable forecasts and that approaching data with no a-priori structure is impossible and imposing very little structure is unwise.
I quote from a review of Silver’s book “The Signal and the Noise” (which I haven’t read)
While the hedgehogs’ mistakes were owed to overly encompassing theories, there is a related error, known as “overfitting,” which Silver calls “the most important scientific problem you’ve never heard of.” Working from historical data, forecasters may develop an elaborate model that seems to account for every wiggle in a curve. But that model may be, as Silver says, an “overly specific solution to a general problem,” with little or no value in making predictions.
Krugman’s concern is that Silver is aware of the hedgehogs’ mistake but has no idea that just messing around with data can lead to over fitting and poor forecasts. In fact, Silver discusses both problems.
I find Silver’s critique of academic political scientists extremely impressive. I agree with Krugman that climatologists clearly have useful expertise. But what about economists ? Does Krugman really think that Silver has too little respect for the judgment of the people who taught him at U Chicago ? Krugman warns “Economists work hard on the data; on the whole you’re going to do better by tracking their research than by trying to roll your own, and you should be very wary if your analysis runs counter to what a lot of professionals say.” Krugman should be very wary (as should all economists) because his analysis runs counter to what a lot of professionals say. There are few yes or no questions in economics about which the general public is divided on which you can’t find a lot of professional economists on each side. There are even questions in which the vast majority of people (including Paul Krugman) agree which are controversial among professional economists. I think the lot of professionals he has in mind are Paul Krugman and other economists who generally agree with Paul Krugman. Then I agree. When one disagrees with Paul Krugman, one is rarely right. But I think disagreeing with his clearly stated belief that Silver has no familiarity with such self identified experts as the professors who taught him in college and his dad is one of those cases.