smithsonian.com
July 7, 2016
Three million analyses point
to a problem with fMRI brain activity studies
 |
|
fMRI changed the way researchers
look at the human brain. (NIH Image Gallery (Flickr/Creative Commons)
|
Science never operates in a vacuum—reproducing results over and
over again is central to research. But a new paper published in the journal PNAS calls the reproducibility of fMRI studies into question,
Lunau writes. The analysis examined resting-state fMRI data from 499 healthy
people. Researchers split the people into groups and used three statistical
packages commonly used to analyze fMRI data to conduct three million
comparisons.
Since the data used were of people whose brains were not
particularly active, so they should not have shown any significant
trends of neural activity. Researchers expected to find false positives—that
is, results showing that people’s brains were not at rest—about five percent of
the time. But that’s where the expected results broke down: Rather than showing
a five percent chance of finding a false positive, the analysis revealed a 70
percent chance.
A bug in one of the software programs used to analyze fMRIs
seems to have been at least partly to blame. When the researchers reported
their findings to software manufacturers, writes Lunau, they responded with their own analyses and,
in one case, code changes. But the study calls into question decades of research that
relies on fMRI studies that used the flawed code.
“It is not feasible to redo 40,000 fMRI studies, and lamentable
archiving and data-sharing practices mean most could not be reanalyzed either,”
the team writes.
The study has caused a stir among scientists who rely on fMRI.
But how bad is the problem? Not as bad as you might think, says Discover’s “Neuroskeptic.” The commentator points out that the problem only applies
to one statistical package and that up to 70 percent of studies containing at
least one false positive does not mean that 70 percent of studies are, in fact,
invalid or false. Further, writes Neuroskeptic, the problem only affects a
small percentage of brain studies—those that deal with brain activation.
Regardless, the study is likely to play into a bigger debate in
the field of science and the brain: reproducibility. Since a gigantic
international effort called the ability of psychology studies to be reproduced
last year, the debate about how to make
research more reliable and reproducible has heated up. (The study in question
was controversial and continues to be debated, especially among the psychology community.) In May, the
journal Science published the results
of a study of 1,500 researchers on
reproducibility. Over 70 percent reported they had tried and failed to
reproduce others’ research, and more than 60 percent listed selective reporting
and pressure to publish as reasons studies that are not reproducible are
published. More than half of the respondents (52 percent) called
reproducibility “a significant crisis” in science—unsurprising, given that scientists have trouble
agreeing what the word even means.
Don’t despair, though: As Monya Baker writes for Slate, recent reproducibility kerfuffles are likely good for science
and spur additional visibility and funding for more reliable results.
"Taken together,” writes Baker, attempts to make work more reproducible
"...could stop researchers from blithely following up on work that cannot
be reproduced or charging down paths others have charted as dead
ends." Studies like the one that calls fMRI brain activation results
into question are sobering, but even as they potentially unseat years of
research, they may push science into a more reliable future.
http://www.smithsonianmag.com/smart-news/new-study-calls-reliability-brain-scan-research-question-180959715/
No comments:
Post a Comment