Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates

By Anders Eklund, Thomas E. Nichols, Hans Knutsson. Proceedings of the National Academy of Sciences of the United States of America, 2015

3,333 citations recorded by the source corpus.

Significance Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data. Here, we used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of a number of fMRI studies and may have a large impact on the interpretation of weakly significant neuroimaging results. The most widely used task functional magnetic resonance imaging (fMRI) analyses use parametric statistical methods that depend on a variety of assumptions. In this work, we use real resting-state data and a total of 3 million random task group analyses to compute empirical familywise error rates for the fMRI software packages SPM, FSL, and AFNI, as well as a nonparametric permutation method. For a nominal familywise error rate of 5%, the parametric statistical methods are shown to be conservative for voxelwise inference and invalid for clusterwise inference. Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape. By comparison, the nonparametric permutation test is found to produce nominal results for voxelwise as well as clusterwise inference. These findings speak to the need of validating the statistical methods being used in the field of neuroimaging.

Subfields: Cognitive Neuroscience, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/93bc678116f0a062de7f694e0552f56df924d08f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Mouth and Voice: A Relationship between Visual and Auditory Preference in the Human Superior Temporal Sulcus (2017, background): “There has also been recent controversy over the use of familywise error correction (Eklund et al., 2016).”
Reduced default mode network functional connectivity in patients with recurrent major depressive disorder (2019, neutral): “MDD studies have used diverse multiple comparison correction methods, most likely inadequate (12).”
Spatio-Temporal Data Mining (2017, background): “Ignoring these dependencies during data analysis can lead to poor accuracy and interpretability of results (Eklund et al. 2016).”
Scanning the horizon: towards transparent and reproducible neuroimaging research (2016, background): “A univariate analysis was performed using FSL to assess the correlation between activation in each voxel and the simulated behavioural regressor across subjects, and the resulting statistical map was thresholded at p 0.001 and with a 10-voxel extent threshold (which is a common heuristic correctio”
On testing for spatial correspondence between maps of human brain structure and function (2018, positive): “It is well-understood that incorrect estimation of spatial smoothing and the use of related statistical tests may result in inflated false negative or false positive rates in functional magnetic resonance imaging (fMRI) analyses (Eklund et al., 2016; Slotnick, 2017; Mueller et al., 2017).”
Adaptive Huber Regression (2017, methodology): “We refer to Fan et al. (2018) for a detailed complexity analysis of the LAMM algorithm.”
Comparing network structures on three aspects: A permutation test. (2022, methodology): “Permutation tests based on resampling are increasingly used in these situations to control the multiple testing problem and accompanying familywise error rate (FWER; e.g., as in functional neuroimaging; Dudoit & Laan, 2007; Eklund et al., 2016; Nichols & Hayasaka, 2003; Winkler et al., 2014).”
The Human Connectome Project’s Neuroimaging Approach (2016, methodology): “…attempts to compensate for imperfect cross-subject alignment, improve signal-to-noise ratio (SNR) and satisfy oversimplified statistical assumptions 13 ; (iii) analysis without using parcellations at all, or using parcellations either derived from Brodmann’s 2D schematic drawing or from locations…”
Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise (2017, methodology): “based” approach to multiple comparisons correction — this method, offered by nearly all common software packages for fMRI analysis, can therefore result in false-positive rates of up to 70% [23].”
The impact of traditional neuroimaging methods on the spatial localization of cortical areas (2018, background): “The statistical assumptions required by Gaussian randomfield theory, which have been assumed to be satisfied by using large smoothing kernels, were recently shown to be unsatisfied in some cases, leading to increased false positives (33).”
Towards the neural population doctrine. (2019, background): “Indeed, this concern has led to high-proﬁle issues in other ﬁelds confronting similarly explosive data (and data analysis) growth (e.g. [ 68]).”