Controlling the familywise error rate in functional neuroimaging: a comparative review

By Thomas E. Nichols, Satoru Hayasaka. Statistical Methods in Medical Research, 2003

1,293 citations recorded by the source corpus.

Functional neuroimaging data embodies a massive multiple testing problem, where 100 000 correlated test statistics must be assessed. The familywise error rate, the chance of any false positives is the standard measure of Type I errors in multiple testing. In this paper we review and evaluate three approaches to thresholding images of test statistics: Bonferroni, random field and the permutation test. Owing to recent developments, improved Bonferroni procedures, such as Hochberg’s methods, are now applicable to dependent data. Continuous random field methods use the smoothness of the image to adapt to the severity of the multiple testing problem. Also, increased computing power has made both permutation and bootstrap methods applicable to functional neuroimaging. We evaluate these approaches on t images using simulations and a collection of real datasets. We find that Bonferroni-related tests offer little improvement over Bonferroni, while the permutation method offers substantial improvement over the random field method for low smoothness and low degrees of freedom. We also show the limitations of trying to find an equivalent number of independent tests for an image of correlated test statistics.

Subfields: Biophysics, Cognitive Neuroscience, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/ab5e55ba170a3f5fcfe816e307640260db7a95ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Cognitive Neuroscience

In Statistics and Probability

Citations in the corpus

of Serendipitous and Unexpected Results Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon : An Argument For Proper Multiple Comparisons Correction (2010, neutral): “There are excellent articles by Brett et al. (2004) and Nichols and Hayasaka (2003) Journal of Serendipitous and Unexpected Results (jsur.org) - Vol. 1 no.1, Pages 1–5.”
Brain plasticity related to the consolidation of motor sequence learning and motor adaptation (2010, background): “In that case, we used familywise error (FWE) correction as it is known to better control for false positives than FDR (39).”
Prospective reports of chronic life stress predict decreased grey matter volume in the hippocampus (2007, methodology): “The family-wise error rate (FWE) was used to correct P-values for conducting multiple voxel-wise statistical tests within the hippocampus (Nichols and Hayasaka, 2003).”
Global connectivity of prefrontal cortex predicts cognitive control and intelligence (2012, methodology): “Statistical maps were corrected for multiple comparisons using a whole-brain false discovery rate (FDR) (Nichols and Hayasaka, 2003) threshold of 0.”
Prefrontal-subcortical pathways mediating successful emotion regulation. (2008, neutral): “We assessed the family-wise error rate (FWER) for the mediation search analyses in each ROI by using a permutation test ((Nichols and Hayasaka, 2003)), which provides estimates of the chances of obtaining a false positive result anywhere in the ROI, accounting for the observed correlations among…”
Multi-site genetic analysis of diffusion images and voxelwise heritability analysis: A pilot project of the ENIGMA–DTI working group (2013, methodology): “We use two different approaches to controlling false positives, the false discovery rate (FDR) (Genovese et al., 2002) and familywise error rate (FWE) (Nichols and Hayasaka, 2003).”
Topological FDR for neuroimaging (2010, positive): “For example, voxel-wise Bonferroni control incidentally guarantees error-control on regional peaks as we have defined them (e.g. Nichols and Hayasaka, 2003).”
Comparing network structures on three aspects: A permutation test. (2022, methodology): “This idea is represented by the following identity (Nichols & Hayasaka, 2003): {x: max f(x)>d} = Union{x: f(x) > d}.”
False discovery rate revisited: FDR and topological inference using Gaussian random fields (2009, neutral): “Conventionally, neuroimaging uses classical inference to protect againstfalsepositivesinthecontextofmultipledependentcomparisons (see Nichols and Hayasaka (2003) for a review of common approaches).”
Imaging the deep cerebellar nuclei: A probabilistic atlas and normalization procedure (2011, positive): “We therefore recommend permutation methods (Nichols and Hayasaka, 2003; Nichols and Holmes, 2002), taking advantage of their minimal assumptions and robustness.”
Handbook of Functional MRI Data Analysis: Index (2011, methodology): “When smoothing is applied to an image, the smoothness of the resulting image is FWHM = √ FWHM 2intrinsic + FWHM 2applied This is particularly important to keep in mind when using statistical methods that require an estimate of the smoothness of an image, which will be discussed in more detail in Cha”
Brainstorm: A User-Friendly Application for MEG/EEG Analysis (2011, methodology): “…available methods include Bonferroni, false discovery rate [58], which controls the expected portion of false positives among the rejected hypotheses, and familywise error rate [59], which controls the probability of at least one false positive under the null hypothesis of no experimental e ﬀ ect.”
Neural systems underlying observation of humanly impossible movements: an FMRI study. (2005, neutral): “FDR has become increasingly popular in neuroimaging research since it is generally more sensitive than traditional methods (Genovese et al., 2002; Nichols and Hayasaka, 2003).”