The Top Two Criteria for Expert Judgment: Curiosity and . . . Curiosity
First a recap:
Philip Tetlock’s Expert Political Judgment was a groundbreaking look at whether political experts really are expert, as judged by their success at making predictions. His overall conclusion: they aren’t. But (lifted from a previous post):
…among the experts, ”foxes” — those who in Nicholas Kristof’s words are “are more cautious, more centrist, more likely to adjust their views, more pragmatic, more prone to self-doubt, more inclined to see complexity and nuance” — resoundingly beat out the “hedgehogs” — those who “have a focused worldview, an ideological leaning, strong convictions.”
This even while hedgehogs end up getting the biggest megaphones for their incorrect predictions.
But as Bryan Caplan pointed out quite cogently, there were two key flaws in Tetlock’s methodology (my words here, again from my previous post):
1. He only examines questions that are highly controversial among experts. (If 50% believe each way, 50% will inevitably be wrong.) Tetlock explicitly ignores the “dumb” questions that seem to the experts to have obvious answers, but which everyday folks might consider controversial.
2. He doesn’t compare the the experts to the average person on the street. The only such comparison in the book is between experts and Berkeley undergrads — who are darned high on the elite/expert spectrum, in absolute terms. And even in that comparison, the experts win in a landslide. The undergrads aren’t even as good as chimps or dartboards.
Back to the present: Tetlock’s latest initiative, the Good Judgment Project, looks to address those shortcomings. The first-round results are in — reported in an email to Tyler Cowen – and they’re eye-opening. Their predictors:
collectively blew the lid off the performance expectations that IARPA had for the first year. Their original hope was that in Year 1 the best forecasting submissions might be able to outperform the unweighted average forecasts of the control group by 20%. When we created weighted-averaging algorithms that gave more weight to our most insightful and engaged forecasters, these algorithms beat that baseline by roughly 60% (exceeding IARPA’s expectations for Year 4).
And what, you may ask, are the characteristics of their successful expert predictors?
(1) an intense curiosity about the workings of the political-economic world; (2) an intense curiosity about the workings of the human mind; (3) cognitive crunching power (“fluid intelligence” and a capacity for “timely self correction”).
Again: the foxes kick the hedgehogs’ butts.
I always agreed with the commentary about George W and his ilk — that they have no real curiosity about how the world works, they just seek confirmation of their existing (and often simplistic) beliefs — but I never considered it much of a knock-down argument. These results — once we see them explained (the email is pretty thin stuff) — may change my beliefs about that.
Caveat: these “curiosity” criteria are uncannily good at describing Philip Tetlock, Bryan Caplan, Tyler Cowen, and me. I tend to look askance at findings that are self-congratulatory.
Justin Fox reports and ruminates on his experience as a fairly mediocre forecaster in the project (emphasis mine):
So what distinguishes a bad forecaster? In my case, two things: (1) a discomfort with expressing my level of confidence with the size of my bets — this is a real flaw, perhaps traceable to the fact that I had never played a game of poker until two weeks ago; and (2) an almost complete lack of interest in the events being forecast. I think I’m pretty curious about the workings of the political-economic world. I just wasn’t interested in whether the IMF would officially announce before 1 April 2012 that an agreement had been reached to lend Hungary an additional 15+ Billion Euros.
… Hedgehogs who are obsessively focused on a particular theory of how the world works aren’t very good at forecasting. But foxes who don’t care aren’t very good at it either. The best forecasters would appear to be foxes who really really want to win the game of forecasting. To quote Saffo again, the key is to “hold strong opinions weakly.” Don’t be stuck in your views; be willing to revise them quickly when new information comes in. But have bold views, or don’t bother trying to make forecasts.
Cross-posted at Asymptosis.
The desire to win is a strong indicator of success in general.
It’s possible that foxes are much more likely to have a strong desire to win (i.e. those who like to win don’t care how they win, and therefore are unlikely to hold to a losing strategy).
We can modify that a bit… in some places it appears, it makes sense to hold big bets… one make more money when the bets go well, and to be bailed if the bets go sour.
That’s more saviness in assessing the actual level of risk though. The “market determined” risk is a lie for some people/institutions.
That’s more saviness in assessing the actual level of risk though. The “market determined” risk is a lie for some people/institutions, for instance Warren Buffett, who can do no wrong because he can do no wrong. When he marches off a cliff, he always has a pile of bodies to climb back up because so many people followed him.
Curiosity!?! That’s for terr’ist coddling pot smoking hippie s-o-c-i-a-l-i-s-t-s! We True Uhmerikans get everything we need from 3 sources: the Bible, Fox News and our Gut. If it don’t come from one a them, then it just ain’t true –simple as that!
You namby pamby limp wristed libuhrals with all your fancy books and your science, what good are they? You worry and whine about so called “evidence” and “facts”. We conservatives don’t need that bullsh*t! We live in a world where “facts” are whatever we WANT them to be. And you know what, our side almost always wins and your side loses!
We’re #1, we’re #1, we’re #1…. wooohooo!!!
my one encounter with Tyler Cowan told me he was an idiot. He thinks global warming would be a good thing economically.
which makes me think that it ain’t curiosity. the pro’s are smarter because they know more. the people you call “hedgehogs” don’t care about making “correct” predictions. they make “predictions” to influence the outcome if they can,or to cheerlead their devoted followers, or to please the ear of their sponsors.
Yes. One of my all time favorite quotes:
“It is difficult to get a man to understand something when his salary depends on him NOT understanding it.” –Upton Sinclair