Duke scientist questions his own research with new study faulting task fMRI

It all started with a rejected grant proposal. Ahmad Hariri, a neuroscientist at Duke University, was interested in using so-called “task fMRI”—in which subjects perform specially designed cognitive tasks while having their brains scanned—combined with genetic testing and psychological evaluations. The goal was to identify specific biomarkers for differences in how people process thoughts and emotions that might determine whether a given subject would be more or less likely to experience depression, anxiety, or age-related cognitive decline like dementia in the future.

“The idea was to collect this data once, then collect it again and again and again and be able to track changes in an individual’s brain over time to help us understand what changes over the course of their lives,” Hariri told Ars. So he submitted a funding proposal outlining his plans for a longitudinal study along those lines. The proposal hypothesized that an individual’s history of trauma, for instance, would map onto how their amygdala reacted to threat-related stimuli. And that would, in turn, enable the researchers to say something about the future mental well-being of the individual.

Hariri and his team designed four core, task-related measures to that end: one targeting the amygdala’s threat response, one targeting the hippocampus and memory, one targeting the striatum and reward, and the fourth targeting the prefrontal cortex and executive control. He thought he was on solid scientific ground. So he was shocked when the proposal wasn’t even scored by reviewers, based on skepticism regarding the reliability of fMRI to collect that kind of data.

“That was the real kick in the pants that I needed to think more seriously about the reliability of task fMRI,” Hariri said. Those concerns led him to undertake an extensive review of published studies claiming it is possible to predict a person’s patterns of thoughts or feelings using task fMRI. He looked specifically at what’s known as “test-retest reliability:” how much correlation there is when a person takes, and then retakes, the same cognitive test while being scanned. The results, described in a recent paper in Psychological Science, overwhelmingly showed that task fMRI was not a reliable indicator: the correlation between one scan and a later scan for the same person was only fair to poor.

The findings instigated a bit of a professional crisis for Hariri. “This is more relevant to my work than just about anyone else’s,” he told Duke Today with remarkable frankness. “This is my fault. I’m going to throw myself under the bus. This whole sub-branch of fMRI could go extinct if we can’t address this critical limitation.”

Granted, he’s not saying it’s impossible to measure brain activation function reliably. “You just can’t do it the way we’ve been doing it, using the tasks we’ve been using,” he told Ars.

“It’s not as if we haven’t known these issues of reliability, but this paper brings them together more sharply,” Russell Poldrack told Duke Today. Poldrack is a psychologist at Stanford University who was not involved in the review study, although one of his fMRI papers from 15 years ago was included in the analysis. “This is a good wakeup call, and it’s a marker of Ahmad’s integrity that he’s taking this on,” he said.

A bit of background

fMRI is one of the most popular brain-imaging techniques in use today, in part because it produces stunning full-color images—striking visualizations of statistical data—showing bright spots of brain activity in response to different tasks. Conventional medical MRI produces a static image of the brain, similar to an X-ray, but functional MRI (fMRI) monitors increases in blood flow produced by groups of neurons firing together in response to a given stimulus. Specifically, it detects slight increases in blood oxygenation levels, known as the BOLD response.

The imaging process produces a lot of raw data—as many as 50,000 data points per scan. So neuroscientists rely on computer algorithms to sift through it all, averaging out the results from the scans of many different study participants all engaged in the same tasks (typically one control task, and one designed to measure a specific target). The larger the difference between the control task and directed task, the stronger the BOLD response. Only those signals that exceed a certain statistical threshold are considered as demonstrating a correlation between the directed task and any affected brain regions.

There are inevitably false positives (the same area “lighting up” in two difference scans by random chance), but neuroscientists work very hard to factor potential false positives into their statistical analyses. The importance of this was famously illustrated in a 2010 paper that found a measurable BOLD response from an fMRI scan of a dead salmon. Neuroscientist Craig Bennett of the University of California, Santa Barbara, was one of the co-authors and a then-grad student at Dartmouth. He was in charge of calibrating the MRI machine, which is typically done by scanning a balloon filled with mineral oil. He and his lab partner decided to have some fun and tried scanning a Cornish game hen, a pumpkin, and finally, the infamous salmon.

Bennett and his lab partner placed the salmon inside the head coil and then ran the calibration test, which involved “presenting” the fish with pictures of human faces and “asking” it to determine the emotions on display in each image. Lo and behold, a signal appeared in the data when he analyzed it—even though there was no way the dead salmon would have shown any brain activity at all. Bennett et al. won the 2012 Ig Nobel Prize for Neuroscience for their illuminating work.

The point is not that fMRI is an unreliable technique. On the contrary, it has proven to be quite robust for studies of groups of participants performing the same task, since this produces a broad, general sample that enables scientists to pinpoint commonalities across populations. Things get a bit stickier when we’re talking about studies trying to measure a BOLD response in just one person—say, to determine if the subject is lying, their belief in god, or their level of empathy. For example, if you put 100 people in a scanner and tried to figure out which of them were lying, the best you could say is that one subgroup will likely lie more often than another subgroup. You have gained a statistically significant snapshot of the group as a whole, but that is not the same as definitively determining that a given person within that group is lying.

That’s why fMRI studies of individuals typically have the subject participate in multiple scanning sessions to compensate for the small sample size (N=1) and reach the required statistical threshold. But it is much more difficult to tease out strong correlations from the data, and it’s easy to convince yourself that you are seeing patterns and correlations in the data that aren’t really there.

A question of reliability

This distinction is where Hariri went wrong with his original proposal. He had developed a task years ago in which subjects look at images of faces while in the scanner. This reliably produces a giant signal from the amygdala when data is collected and analyzed across a group of test subjects. “The error that certainly I made was to assume that these very strong readouts that we saw when we averaged the information across people, were suitable to look at the individual readouts as well,” he said. “We made a leap of faith from the experimental model where we were talking about, how does the typical human brain do this, to the correlational model, asking how do the differences in the way different brains do [a task] matter.”

For their review, Hariri et al. re-examined 56 task fMRI papers across 90 experiments, as well as the test/retest results (four months apart) of 45 subjects participating in the Human Connectome Project. They also analyzed data they had collected through the Dunedin Multidisciplinary Health and Development Study in New Zealand, in which they conduced two scans for each of 20 subjects, two or three months apart. They found only fair to poor correlations from one test to the next in all three datasets.

“It wasn’t an accurate readout of that individual’s brain. It was only a readout of that person’s brain at that moment in time.”

 

In short, “we had no confidence that the readout is telling us anything related to the subject’s prior history of trauma or stress,” said Hariri. “And it made no sense that the readout would tell us something about the future because it wasn’t really an accurate readout of that individual’s brain. It was only a readout of that person’s brain at that moment in time.”

Naturally, this has enormous implications. Hariri has stopped all research on task fMRI in his lab. “I can’t keep doing something that I know is fundamentally flawed,” he said. He is now focusing on measuring brain structure instead, which is highly reliable in comparison, although brain function is not necessarily reflected in that structure. Still, “if you measure it at one time point, and again at another time point, you get almost exactly the same readout,” he said.

Hariri admits his graduate students were initially upset by the change in focus; many had opted to work in his lab because they wanted to do task fMRI research with individuals. “I said, look, I’m an old tenured professor, and sure, I have egg on my face, and some of my colleagues are going to get pissed,” he said, “but your futures are on the line here. I can’t in good conscience let you do this work, and then let you go out there and get torn apart.”

A hunt for solutions

So what can be done about it? “There are three things you can do,” said Poldrack. “You can just up and quit, you can stick your head in the sand (and act as if nothing has changed), or you can dig in and try to solve the problems.”

One approach might be to conduct much longer scans for individual subjects to collect significantly more data per subject. Several years ago, Poldrack had a project called MyConnectome, where he scanned himself for an hour or more every day for 18 months, simply lying still, eyes open and staring into space. He found that for that length of time, he could reliably measure the intrinsic pattern of his brain function day after day. Washington University in St Louis boasts the so-called “Midnight Scan Club,” in which graduate students scan themselves for hours at a time, week after week, to better understand the inner workings of individual people’s brains by compiling hours and hours of data for each subject.

https://youtu.be/7I662Se4qWU

However, that approach isn’t very practical for the kind of longitudinal studies Hariri wants to conduct. “There’s no way I’m going to take 1,000 people in Dunedin and bring them back every five to seven years and make them sit in a scanner for two hours,” he said. “No one would do it.” For one thing, it’s difficult to keep still in the scanner for 10-15 minutes even for adults, never mind young children. It’s even harder when the subjects are required to engage in repetitive cognitive tasks that have been typical of such studies to date, for an hour or more.

But Hariri and one of his graduate students, Max Elliott (a co-author on the review study), found that they could take the data from their task-oriented scans and combine them with the data from everyone’s eight-minute resting scans. This effectively created more data (about 35 minutes’ worth for each subject). “We typically don’t do that because we think of them as being very different in what they’re measuring about the brain,” said Hariri. “But by stringing them together, you can look at the intrinsic patterns of connectivity across the whole brain, from any type of functional data.”

This kind of “connectivity mapping” is not measuring the amygdala or the dorsal lateral prefrontal cortex doing X, Y or Z, but the intrinsic processing of information, or function of the brain, over time to get a reliable connectivity measure. It looks at how different areas of the brain are connected during a specific task, rather than which areas are most active. But Hariri thinks it has potential to transform existing individual datasets into a suitable format for the kind of biomarker studies he’s interested in conducting.

Hariri also would like to see more neuroscientists publish their null findings. “We should give them proper weighting in the grand scheme of science, to make sure they are included in the literature,” he said. That way, when a scientist searches the literature for prior work in their area, they won’t just see 20 papers all reporting similar findings; they will also see just as many, if not more, papers reporting null results.

As always, the first step is admitting there is a problem. “Honestly, whether you accept it or not, it’s in black and white now,” said Hariri. “It’s going to be really hard to say this is not a problem.”