Type I and Type II error concerns in fMRI research: re-balancing the scale.

By Matthew D. Lieberman, William A. Cunningham. Social Cognitive and Affective Neuroscience, 2009

1,358 citations recorded by the source corpus.

Statistical thresholding (i.e. P-values) in fMRI research has become increasingly conservative over the past decade in an attempt to diminish Type I errors (i.e. false alarms) to a level traditionally allowed in behavioral science research. In this article, we examine the unintended negative consequences of this single-minded devotion to Type I errors: increased Type II errors (i.e. missing true effects), a bias toward studying large rather than small effects, a bias toward observing sensory and motor processes rather than complex cognitive and affective processes and deficient meta-analyses. Power analyses indicate that the reductions in acceptable P-values over time are producing dramatic increases in the Type II error rate. Moreover, the push for a mapwide false discovery rate (FDR) of 0.05 is based on the assumption that this is the FDR in most behavioral research; however, this is an inaccurate assessment of the conventions in actual behavioral research. We report simulations demonstrating that combined intensity and cluster size thresholds such as P < 0.005 with a 10 voxel extent produce a desirable balance between Types I and II error rates. This joint threshold produces high but acceptable Type II error rates and produces a FDR that is comparable to the effective FDR in typical behavioral science articles (while a 20 voxel extent threshold produces an actual FDR of 0.05 with relatively common imaging parameters). We recommend a greater focus on replication and meta-analysis rather than emphasizing single studies as the unit of analysis for establishing scientific truth. From this perspective, Type I errors are self-erasing because they will not replicate, thus allowing for more lenient thresholding to avoid Type II errors.

Subfields: Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/3d817620494190605c4c5869258ef4fa93236860

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Ten simple rules for neuroimaging meta-analysis (2017, background): “On the one hand, by not correcting for multiple comparisons, one is certainly more sensitive to discover meaningful (smaller) effects (Lieberman and Cunningham, 2009).”
Whole-brain, time-locked activation with simple tasks revealed using massive averaging and model-free analysis (2012, background): “As Lieberman and Cunningham stated previously (7), a longstanding preoccupation with the reduction of false-positives in fMRI creates a bias toward reporting only large and obvious effects, neglecting what perhaps represents more subtle complex cognitive and affective processes.”
On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments (2012, result): “Thus, these results suggest that some thresholding approaches are far more conservative than others, even when targeting the same corrected false positive rate – a point that has been raised in previous studies (e.g., Lieberman and Cunningham, 2009) but that merits being repeated here.”
The relation between statistical power and inference in fMRI (2017, methodology): “To strike a better balance between Type I and Type II errors, some researchers have advocated using conventional uncorrected thresholds such as p .001 or p .005 [41].”
The secret lives of experiments: Methods reporting in the fMRI literature (2012, neutral): “But these claims have been contested as well: Lieberman and Cunningham (2009) suggest that formal corrections for multiple comparisons lead to unacceptably high rates of false negative results and instead recommend the use of liberal uncorrected thresholds (e.g., a single-voxel threshold of pb0.”
Neural substrates of smoking cue reactivity: A meta-analysis of fMRI studies (2011, positive): “…rate (FDR) of q < .05 (Genovese et al., 2002), a whole-cluster p < .05 obtained via simulations based on random field theory (Friston et al., 1993; Worsley et al., 1992), or a single-voxel p < .005 combined with a minimum cluster size of 10 contiguous voxels (Lieberman & Cunningham, 2009).”
Ventromedial prefrontal volume predicts understanding of others and social network size (2011, methodology): “…overly stringent corrections for multiple comparisons in neuroimaging has recently been criticizedas aknownsourceof Type II errors (falsenegatives) (Lieberman and Cunningham 2009) we used the more liberal threshold of p=0.001 uncorrected, with an extent threshold of kN5 voxels within our a priori…”
FMRI Clustering in AFNI: False-Positive Rates Redux (2017, neutral): “In addition, the parametric methods of AFNI, SPM, and FSL were evaluated with an ‘‘ ad hoc ’’ clustering method from Cunningham and Lieberman (2009), with results shown in ENK16—ﬁgure 2; ‘‘Perm’’ method results were not provided for this test.”
Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates (2015, positive): “001 (uncorrected for multiple comparisons) is used together with an arbitrary cluster extent threshold of 10 8-mm(3) voxels (26, 28).”
Inconsistencies in spontaneous and intentional trait inferences. (2012, methodology): “005 (uncorrected) was used for all comparisons with a minimum cluster extent of 10 voxels (Lieberman and Cunningham, 2009; Ma et al., 2011).”
An fMRI investigation of empathy for 'social pain' and subsequent prosocial behavior (2011, background): “Thismeasure included 10 items, such as “I felt bad for him/her,” and “It hurt to watch him/her play,” and was highly reliable (α=.92),”