The Simpsons’ Predictive Powers Demonstrate the Limits of Forecasting
by Cactusman (Dan here…not so heavy post)
The Simpsons’ Predictive Powers Demonstrate the Limits of Forecasting
The long-running TV show The Simpsons is attracting much attention for some of its uncannily accurate predictions. The United States’ curling team’s defeat of Sweden in the Olympics – something The Simpsons predicted in 2010 – being the most recent example. And, of course, there is the notorious episode from 2000, which referenced a Trump presidency.
Given these examples and other predictions that have come true, I’m left wondering whether The Simpsons’ writers are good folks to ask for stock tips. Theories about the show’s prescience abound. They include those who believe its creator, Matt Groening, is a Freemason with supernatural occult powers, as well as more prosaic explanations, such as its many Harvard-educated writers have imbued the show with above-average intelligence.
Understanding The Simpsons’ touted predictive power requires exploring the nature of forecasting and the limits of forming accurate predictions.
One of the most sophisticated projects on humans’ ability to make accurate predictions, The Good Judgment Project (GJP), was launched in 2011 as a program within the US government’s Intelligence Advanced Research Projects Activity. (Full disclosure: I was a participant in this project for a year soon after it launched).
The GJP recruited participants to function as forecasters in a two-year tournament that involved submitting probability estimates for a variety of geopolitical events. Each participant was required to have at least a bachelor’s degree and had to pass a number of psychological and political-awareness tests. Because the GJP was interested in finding the best drivers of predictive accuracy, they tested the impact of several forecasting methods: training, teaming, and tracking.
Training was provided through several learning modules to a subset of participants. It included scenario training, which teaches how to overcome biases, or probability training, which teaches how to formulate and assign meaningful probabilities to events. Teaming involved splitting participants into subgroups of independent actors, teams, or a crowd-belief group, in which individuals made their own decisions but had access to information on what others had chosen. Tracking involved taking the top 60 forecasters from the first year of the tournament to create five new teams of “superforecasters” for Year 2, who would compete against the general population.
Some of the results from the tournament were in line with expectations (dare I say, predictable?): those with training outperformed those without, and crowd-belief forecasters and those working in teams outperformed individual forecasters. But the most interesting outcome of the experiment was the margin by which the superforecasters outperformed the rest of the field, which is summarized in the chart below (accuracy was measured with a metric known as the Brier score; a lower score indicates higher accuracy and consistency):
Not only did the superforecasters crush the competition, but their accuracy improved remarkably during the second year of the competition. The researchers summarized these findings, a bit drily, as:
“These superforecasters outperformed all other groups by a wide margin. There was no evidence of Year 2 regression to the mean; political forecasting appeared to be at least somewhat skill based, and the acquisition of skill accelerated when top performers worked together.”
Naturally, the emergence of an elite cast of superforecasters led to a fair amount of media hype, and one of the tournament’s founders, Philip Tetlock, wrote a book. In his writings and interviews, he notes that characteristics of superforecasters include high levels of fluid intelligence, open-mindedness, being self-critical, analytical, and focused on constantly updating their forecasts as new information becomes available.
Superforecasters are real. But when a team of superforecasters isn’t available, prediction markets such as PredictIt and Almanis, which allow participants to gamble on the probability of a future event happening based on the value of shares that fluctuate with each transaction, are another often-discussed mode of divining future events by harnessing the wisdom of crowds. These markets have grown with the spread of the internet, and one study showed they outperformed polls 74% of the time in predicting the outcome of presidential elections. But prediction markets have their limits, too, and overwhelmingly failed to predict the outcome of the 2016 election.
So are the writers of The Simpsons a select group of superforecasters or, perhaps, using a secret prediction market to generate their storylines? It’s hard to say, and they’ve certainly gotten plenty of things wrong over the years, too. I can recall one episode where dolphins leapt from the seas to successfully challenge mankind for dominion of the land. (Although, at the rate we continue to pollute the oceans, it’s best not to rule that one out yet.)
Nonetheless, the more likely explanation for The Simpsons’ success is that the writers are just really good at capturing – and mocking – the deeply irrational underpinnings of our cultural and political trends. Matt Groening admitted as much in an interview with The Guardian in 2016 when he said, “We predicted that he would be president back in 2000 – but [Trump] was of course the most absurd placeholder joke name that we could think of at the time, and that’s still true. It’s beyond satire.”
The work of e GJP and prediction markets show that some folks are better than others at forecasting the future within an environment focused on choosing between near-term, plausible events. But venture outside the confines of these experimental designs, and the reality of human behavior erodes the predictive power of these methods.
While The Simpsons’ writers might appear to possess uncanny forecasting abilities, as Groening’s quote demonstrates, they aren’t really making predictions at all. Instead they’re using seemingly implausible storylines to poke fun at American society. That some of these stories happen to come true just shows some things really are too absurd to predict.
Here is a superforecast from more than 40 years ago.
“Instead, Trumpism has been metastasizing in plain sight ever since — gravitating from the Democrats to the Republicans but in truth a remarkably consistent force no matter what its followers’ party affiliation (if any). When Wallace’s national political trajectory was accelerating in 1971, Donald Warren, a sociologist at Oakland University, began interviewing midwestern voters in depth to come to grips with the phenomenon. He coined the term “Middle American Radicals” to capture their paradoxical politics: They sided “with the traditional left in opposition to the privileges and power of the rich corporations,” but with the right in their fear “of the growing power of the poor and minority groups in our society.” In his 1976 book The Radical Center, Warren correctly surmised that Middle American Radicals “will be a significant force irrespective of their inclusion as part of the established parties,” and cautioned that their dissatisfaction, if left unaddressed, could pose a “great danger” and “affect the political and social fabric with possibly drastic consequences.” His research would be eerily echoed in 2016 when Amanda Taub reported in Vox on recent academic studies finding that Americans with authoritarian traits (a need for order, fear of outsiders, the desire for a strong man to preserve the status quo) are “a surprisingly large electoral group” that aligns with “right-wing populism.” Indeed, an authoritarian mind-set was the “single best predictor” of who would vote for Trump, just ahead of having only a high-school education. Taub sounded a warning that picked up where Warren had left off with his Radical Middle Americans of four-plus decades earlier: This “exists independently of Trump and will persist as a force in American politics.”
well, i don’t think it’s so much a “prediction” as it is a fairly accurate description of the effects of “policy” or, more accurately, “politics” manipulating the well known “facts” about human psychology.
as for the “having only a high school education” the sad fact is that “college” is pretty much a propaganda field for one set of politicians preferred beliefs (not monolithic…biologists are, after all, not economists)
and there was a time when grade school was a pretty good place to teach community values and the belief in democracy. now, I think, they only teach computer aps.
as for the intelligence of Harvard educated writers: from what I have seen in both the subject I studied in college, and a subject i think i know something about, Harvard educated means “I am so smart I don’t have to actually think about what I am writing about.” superficial, glib, with sprinklings of clever word play and references to great books.”intelligent” appears to mean “gets high scores on tests designed by Harvard “experts.”
It’s all propaganda from the 2+2=4 crowd. Sad.
no. i think it was
here come ol’ flat top
He roller coaster
He got early warning
He got Muddy Water
He one Mojo filter
He say. “One and one and one is three”
one thing i can tell you is you got to be free
Come together right now over me
btw. i would’t take that intelligence thing too seriously, even Harvard intelligence. the whip smart Democrats can’t figure out why the uneducated white men they despise won’t vote for them. i mean, it has to be because they are ignorant wife beater racists, right?
(i think the only reason “crowd sourcing” might beat out the expert predictors, is the crowd sees things the experts don’t bother to look at…. or is that obvious? I think Jefferson might have been on to that.)