On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments

By Joshua Carp. Frontiers in Neuroscience, 2012

376 citations recorded by the source corpus.

How likely are published findings in the functional neuroimaging literature to be false? According to a recent mathematical model, the potential for false positives increases with the flexibility of analysis methods. Functional MRI (fMRI) experiments can be analyzed using a large number of commonly used tools, with little consensus on how, when, or whether to apply each one. This situation may lead to substantial variability in analysis outcomes. Thus, the present study sought to estimate the flexibility of neuroimaging analysis by submitting a single event-related fMRI experiment to a large number of unique analysis procedures. Ten analysis steps for which multiple strategies appear in the literature were identified, and two to four strategies were enumerated for each step. Considering all possible combinations of these strategies yielded 6,912 unique analysis pipelines. Activation maps from each pipeline were corrected for multiple comparisons using five thresholding approaches, yielding 34,560 significance maps. While some outcomes were relatively consistent across pipelines, others showed substantial methods-related variability in activation strength, location, and extent. Some analysis decisions contributed to this variability more than others, and different decisions were associated with distinct patterns of variability across the brain. Qualitative outcomes also varied with analysis parameters: many contrasts yielded significant activation under some pipelines but not others. Altogether, these results reveal considerable flexibility in the analysis of fMRI experiments. This observation, when combined with mathematical simulations linking analytic flexibility with elevated false positive rates, suggests that false positive results may be more prevalent than expected in the literature. This risk of inflated false positive rates may be mitigated by constraining the flexibility of analytic choices or by abstaining from selective analysis reporting.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/1b0011c33548e74edb236b6386cc7fb4a261bebb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

Citations in the corpus

Toward open sharing of task-based fMRI data: the OpenfMRI project (2013, methodology): “Given the multiplicity of different analysis streams and likelihood of different results between streams (Carp, 2012), it is expected that the results will sometimes fail to exactly match those of the original analysis.”
Comparing spatial null models for brain maps (2021, neutral): “…demonstrated in research using structural MRI ( Bhagwat et al., 2020; Khara-bian Masouleh et al., 2020 ), task fMRI ( Botvinik-Nezer et al., 2020; Carp, 2012 ), resting state fMRI ( Ciric et al., 2017; Parkes et al., 2018 ), and diﬀusion MRI ( Maier-Hein et al., 2017; Oldham et al., 2020;…”
Variability in the analysis of a single neuroimaging dataset by many teams (2019, background): “Participating teams were located in 17 countries/regions around the world: Australia (3 participants), Austria (3), Belgium (9), Brazil (3), Canada (16), China (2), Finland (1), France (4), Germany (21), Italy (13), the Netherlands (8), Spain (4), Sweden (4), Switzerland (3), Taiwan (3), UK (7) and ”
Making big data open: data sharing in neuroimaging (2014, methodology): “Similarly, the data from the OpenfMRI project have been used to examine the effects of different processing pipelines on analysis outcomes 18 and test new methods to characterize the brain systems that span across multiple cognitive tasks 10 .”
MEG and EEG data analysis with MNE-Python (2013, background): “Some recent studies have pointed out the heterogeneity of functional MRI data analysis pipelines (Carp, 2012a,b).”
Behavior, Sensitivity, and power of activation likelihood estimation characterized by massive empirical simulation (2016, neutral): “Furthermore, the substantial analytic flexibility of neuroimaging pipelines also leads to low reliability and reproducibility of neuroimaging findings (Carp, 2012b; Glatard et al., 2015; Wager et al., 2009).”
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them (2020, background): “Note that inherent flexibility in quantifying raw data is not isolated to surveys and abounds in others approaches to measurement, such as neuroimaging (Carp, 2012).”
Best practice guidance for linear mixed-effects models in psychological science (2019, background): “The same concerns have arisen in fields other than experimental Psychology, for example, following the rapid expansion in neuroimaging studies in which complex analyses with multiple analytic steps are the norm (Carp, 2012a; Carp, 2012b; Poldrack & Gorgolewski, 2014; Wager, Lindquist, & Kaplan, 2007”
Cognitive reappraisal of emotion: a meta-analysis of human neuroimaging studies. (2014, methodology): “We believe this approach enhances the interpretability of our results and also effectively combats the growing multiple comparisons problem in neuroimaging, in which greater analytic flexibility increases the likelihood of finding regions compatible with one’s hypotheses (Carp 2012, 2012).”
Benchmarking functional connectome‐based predictive models for resting‐state fMRI (2019, positive): “These various choices have a sizable impact on the accuracy of population studies, and are seldomdiscussed (Carp, 2012).”
Ten simple rules for neuroimaging meta-analysis (2017, background): “In addition, results are strongly influenced by experimental and analyses procedures (Carp, 2012) and replication studies are rare.”
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis (2021, background): “Strikingly, no two individual analysts employed precisely the same specification for either Hypothesis 1 or 2 (see Botvinik-Nezer et al., 2020, and Carp, 2012a; 2012b, for similar findings in neuroimaging studies and Bastiaansen et al., 2020, for a conceptual replication with event sampling data…”
Methods for cleaning the BOLD fMRI signal (2016, positive): “Of course, this is a challenging task and automatized adaptive frameworks must be developed, for instance following Carp (2012b), because the optimal pipeline for a given dataset will likely depend on: a) the MR acquisition parameters (type of sequence, voxel size, TR, order of slice acquisition,…”