Krugman Vs Silver
I grab a rare opportunity to criticize Paul Krugman. He is, again, discussing the roll out of Nate Silver’s new fivethirtyeight.com with concern bordering on dismay. Krugman’s concern is that he fears that Silver thinks that data are enough just by themselves without any theory (even the humble kind that calls itself a model). I actually share his concerns, but think the post is no good.
data never tell a story on their own. They need to be viewed through the lens of some kind of model, and it’s very important to do your best to get a good model. And that usually means turning to experts in whatever field you’re addressing.
Unfortunately, Silver seems to have taken the wrong lesson from his election-forecasting success. In that case, he pitted his statistical approach against campaign-narrative pundits, who turned out to know approximately nothing. What he seems to have concluded is that there are no experts anywhere, that a smart data analyst can and should ignore all that.
This is a critique of a straw blogger. The sloppy unstable over hasty data freak whom I will call quick-Silver. Quick-Silver has no familiarity with alleged experts other than “campaign-narrative pundits,” believes that all one needs are data (and presumably some universally valid statistical methods), and thinks the only possible mistake is to be a hedgehog not a fox (I think I am being fair to Krugman and not criticizing the made up strawman Krudeman).
update 2: Here is the Silver post which I didn’t find before. It is a devastating critique of the efforts by academic political scientists to forecast presidential elections using data on fundamentals. Krugman is right that Silver shows little respect for the expertise of academic political scientists. My claim is that Silver has justified that view with rock solid meta analysis (which, in my humble opinion, should be published in the American Political Science Review, not that they listen to me).
Krugman goes on to argue that there are valuable experts of whom quick-Silver is unaware
But not all fields are like that — in fact, even political analysis isn’t like that, if you talk to political scientists instead of political reporters. So, for example, before glancing at some correlation and asserting causation, you really should talk to the researchers.
Similarly, climate science has been developed by many careful researchers who are every bit as good at data analysis as Silver, and know the physics too, so ignoring them and hiring a known irresponsible skeptic to cover the field is a very good way to discredit your enterprise. Economists work hard on the data; on the whole you’re going to do better by tracking their research than by trying to roll your own, and you should be very wary if your analysis runs counter to what a lot of professionals say
Krugman clearly implies that Silver is confused, because he has had little contact with academic political scientists, climatologists and economists. Some data. Nate Silver has a BA in economics from the University of Chicago (roughly when the economics department included a total of 15 professors). Now I know undergraduates don’t get as much face time with professors as they should, but he does have some contact with prominent academic economics. I pass on climatology. Checking Wikipedia for the Chicago economics connection I read
Silver was born in East Lansing, Michigan, the son of Sally (née Thrun), a community activist, and Brian David Silver, a former chair of the political science department at Michigan State University.
I think Silver has met his dad.
Much more to the point, Silver is very familiar with the work of quantitative empirical political scientists. IIRC he tested the out of sample performance of all fundamentals based election forecasting models in the literature (including again IIRC ones presented only in working papers). He found it to be horrible. I think he has made animportant contrution to the literature which which Krugman seems to think heis unfamiliar.
I write IIRC because google isn’t sending me to the post I remember. Bleg how to I find posts which link back to the older post which I found, in which Silver finds that Hibbs’ model works poorly out of sample (using data on disposable income growth older than those Hibbs used).
Google also sent me to one criticizing a blogger, including “The issue with this model, and some others like it, is what’s known in the statistical business as overfitting. This occurs when the number of variables is large relative to the sample size: in this case … six variables, …used to explain only 15 cases.” Silver understands that correlation does not necessarily imply reliable forecasts and that approaching data with no a-priori structure is impossible and imposing very little structure is unwise.
I quote from a review of Silver’s book “The Signal and the Noise” (which I haven’t read)
While the hedgehogs’ mistakes were owed to overly encompassing theories, there is a related error, known as “overfitting,” which Silver calls “the most important scientific problem you’ve never heard of.” Working from historical data, forecasters may develop an elaborate model that seems to account for every wiggle in a curve. But that model may be, as Silver says, an “overly specific solution to a general problem,” with little or no value in making predictions.
Krugman’s concern is that Silver is aware of the hedgehogs’ mistake but has no idea that just messing around with data can lead to over fitting and poor forecasts. In fact, Silver discusses both problems.
I find Silver’s critique of academic political scientists extremely impressive. I agree with Krugman that climatologists clearly have useful expertise. But what about economists ? Does Krugman really think that Silver has too little respect for the judgment of the people who taught him at U Chicago ? Krugman warns “Economists work hard on the data; on the whole you’re going to do better by tracking their research than by trying to roll your own, and you should be very wary if your analysis runs counter to what a lot of professionals say.” Krugman should be very wary (as should all economists) because his analysis runs counter to what a lot of professionals say. There are few yes or no questions in economics about which the general public is divided on which you can’t find a lot of professional economists on each side. There are even questions in which the vast majority of people (including Paul Krugman) agree which are controversial among professional economists. I think the lot of professionals he has in mind are Paul Krugman and other economists who generally agree with Paul Krugman. Then I agree. When one disagrees with Paul Krugman, one is rarely right. But I think disagreeing with his clearly stated belief that Silver has no familiarity with such self identified experts as the professors who taught him in college and his dad is one of those cases.
I think you have a valid point regarding political scientists and perhaps even economists. I would however defer to Michael Mann regarding Climate Science and I am troubled in that regard. I guess that time will tell.
Robert:
Data always points a direction; but, it does not mean it is correct. You just have to look and confirm it. Very much the same as I would do LSS analysis in manufacturing.
Bill
“Freakonomics” left a bad taste in a lot of people’s mouths. I think the authors were on a par with or ranked above Mr. Silverman in Economics and statistical analysis (or at least in Economics). The ability of climate deniers to waste their educations misusing statistics and physics is also well known. And these sorts of people tend to make a lot of money – more than if they were in their right minds. So I hope Mr. Silverman will take heed of Dr. Krugman’s advice, even if such criticism is not warranted at this point.
Interesting that this post does not get into specifics, like Silver’s claim that SS and Medicare are making the public distrust the government. And going for “no comment” on Silver’s quasi climate denialism is avoiding about half of Krugman’s argument… if you’re going to say that Krugman has no basis for his judgment then you might want to respond to his argument directly instead of bringing up Silver’s father’s CV.
If I were “going to say that Krugman has no basis for his judgment”, then I wouldn’t write “I actually share his concerns.” You’re comment consists entirely of the argument that I didn’t demonstrate a conclusion which I very explicitly didn’t draw. It demonstrates an alarming failure of elementary reading comprehension.
I have to agree with Krugman on this. Silver’s main contribution was to improve measurement. He moved in this direction in the middle of 2008, changing his model to enhance it’s sensitivity to current polling results and making it much less reliant on more static variables drawn from political theory. The change led to more dynamic, and therefore more interesting, blog postings about the horse race (and to a gig at the New York Times?). Overall it was a great improvement over punditry and other efforts to translate polling numbers into electoral college votes, but it also was a clear move away from political science.
Well I think that Silver has presented a very impressive critique of the academic peer reviewed political science literature on forecasting elections based on fundamentals. I blegged for the link (the one I found is a much less impressive post).
I’m not an expert in the field, but the researcher in the field for whom Ihave the most respect is Nate Silver. He has also moved away from the economics he was taught at U Chicago. I do not think this shows insufficient respect for experts (I mean read what Krugman writes about some of the profs there).
Silver certainly has low respect for academic experts. But I think his low respect for academic political scientists is based on his doing meta analysis of their work (which one of them should have done). I think he is just better at quantitative analysis of fundamentals and election outcomes than, say, Doug Hibbs.
Litaratures can be strange. A valid critique of a whole literature does not necessarily affect it as it is dismissed, because the results of accepting it are too painful to authors, editors and referees. I am sure this is true of my own field and quite confident that it is true of the academic political science models which Silver decided to de-emphasize.
On Silver’s forecasts, he predicted much more than who would win. He estimated a cumulative density of winning margins separately for each state. This makes a ruthless pitiless evaluation of his model possible by looking at his estimates of the probability that Obama’s share would be less than or equal to his actual share. The distribution would be uniform if Silver’s model were exactly accurate. The empirical distribution over the 51 cases is indistinguishable from a uniform distribution. This is outrageously impressive.
He also made pretty good forecasts months out. That is very hard. What Silver does is fairly simple, but it worked.
This is a separate question from the alarm I feel about new fivethirtyeight which has a lot to do with the climate slatepitcher hire. Basically I think climate scientists are genuine experts whose opinions on climate are worthy of respect. I don’t think that of economists in general. I don’t know about political scientists, but I find [oops found I can’t find the post] Silver’s work on the topic absurdly impressive.