The science behind human irrationality just passed a huge test

If you won $100 tomorrow, you’d be pretty happy. And if you lost $100, you’d be less than thrilled. But those two feelings wouldn’t be the same in magnitude: the loss would probably sting far more than the gain would delight.

People don’t approach things like loss and risk as purely rational agents. We weigh losses more heavily than gains. We feel like the difference between 1 percent and 2 percent is bigger than the difference between 50 percent and 51 percent. This observation of our irrationality is one of the most influential concepts in behavioral science: skyscrapers of research have been built on Daniel Kahneman and Amos Tversky’s foundational 1979 paper that first described the paradoxes of how people make decisions when faced with uncertainty.

So when researchers raised questions about the foundations of those skyscrapers, it caused alarm. A large team of researchers set out to check whether the results of Kahneman and Tversky’s crucial paper would replicate if the same experiment were conducted now.

Behavioral scientists can heave a sigh of relief: the original results held up, and robustly. With more than 4,000 participants in 19 countries, nearly every question in the original paper was answered the same way by people today as they were by their 1970s counterparts.

Back to the beginning

Kahneman and Tversky’s earthquake observations about human decision-making, called “prospect theory,” have found their way into policy, business, and health care. It’s prospect theory that earned Daniel Kahneman his Nobel Prize and prospect theory that underlies famous concepts like loss aversion—the tendency people have to weigh losses more heavily than gains.

“It would have been shocking, earth-shaking shocking, if [the results] had failed to replicate,” says Brian Nosek, a psychologist who focuses on replications and transparency in science and who wasn’t involved in this replication.

Despite the centrality of prospect theory in behavioral science, that original paper from 1979 had never been directly replicated. That doesn’t mean it was the only evidence for the idea: huge amounts of research had replicated it conceptually, by doing other experiments that provided evidence for the theory. And the original idea spawned children, and grandchildren, with their own bodies of research.

But as researchers poke at the foundations of much of behavioral science and find alarming weaknesses, some psychologists have argued that conceptual replication might paint an unconvincingly reassuring picture of an entire sub-field of research. The problems that weaken those foundations can also strike conceptual replications: small experiments produce erratic results, and it’s far too easy to see spurious patterns in those erratic results. Then the positive patterns get published, and the negative patterns never see the light of day, skewing the published picture.

And if an experiment departs from the original study that inspired it, it might seem like no big deal that it didn’t work—it’s probably just the difference in methods, not the whole theory that’s wrong, right? Conceptual replications that don’t work out might just get buried, while the ones that do work out get published. Over time, the result could be a house of cards that looks like a skyscraper until you poke it.

Could prospect theory be a house of cards? When a controversial paper came out arguing that the evidence for loss aversion was weaker than it seemed, social psychologist Kai Ruggeri assembled a large international team of researchers to check its foundations by replicating the 1979 study as closely as possible.

Gain most, lose some

The team tested more than 4,000 participants in 19 countries. All the participants had to answer questions about money and risk, like whether they’d prefer an 80 percent chance of getting $4,000 or a guaranteed $3,000.

The questions were the same as those used in Kahneman and Tversky’s original paper, although the amounts were updated for 2019 and adapted for different countries’ incomes. The original paper found that there were distinct patterns in how people answered these questions—like most people choosing the guaranteed $3,000. The replication found the same tendencies on 16 out of 17 questions.

The original paper also found that people made very different choices about losses and gains. For instance, given the choice between an 80 percent chance of losing $4,000 and a guarantee of losing $3,000, the majority of people pick the chance of losing $4,000—the opposite result to the same question about gain. These contrasts between choices also replicated.

The paper is “impressive in its rigor and transparency,” Nosek told Ars: all of its materials, data, and analyses are freely available online. And it’s nice, he says, to have the demonstration that “there are solid foundations on which areas of social-behavioral research are building.” Nosek’s own work has found that much of behavioral science doesn’t replicate—but also, importantly, that some does.

Many of the results in the replication are more moderate than in the original paper. That’s a tendency that has been found in other replications and is probably best explained by the small samples in the original research. Getting accurate results (which often means less extreme results) needs big samples to get a proper read on how people in general behave. Smaller sample sizes were typical of the work at the time, and even today, it’s often hard to justify the effort of starting work on a new question with a huge sample size.

Differences by country

There were some differences between the results from different countries—while all of the original results replicated in Australia and Hong Kong, only 77 percent replicated in Bulgaria and Chile. But this could be for the same boring reason, says Nosek: the “least successful” countries had the smallest sample sizes, introducing the same problem of getting an accurate read on a population at large. It’s not enough evidence to assume that there are meaningful international differences, he says.

But the differences between how different people respond are meaningful in how policymakers should think about the results, says Ruggeri, who led the study. The original paper and the replication find tendencies, not universal rules. “If you find somebody who doesn’t fit the pattern, it doesn’t make them wrong, it doesn’t make them strange,” he said. So we should expect to find that different people—and by extension, different randomly chosen groups of people—have different responses, and build that expectation of difference into any real-world use of prospect theory.

Ruggeri’s team is careful to point out that the successful replication doesn’t mean that prospect theory is unassailable. Other critiques might still land. But fundamentally, that original finding holds up. The foundations are solid.