Variability in the analysis of a single neuroimaging dataset by many teams

By Rotem Botvinik-Nezer, Felix Holzmeister, Colin F. Camerer, Anna Dreber, J. Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, Alison Adcock and 187 more. Nature, 2019

901 citations recorded by the source corpus.

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/5fe2e9ce1267547d40a3b98fedde4775a59c1eee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In Statistics and Probability

Citations in the corpus

The influence of hidden researcher decisions in applied microeconomics (2020, positive): “Botvinik-Nezer et al. (2020) recruited teams to examine a data set of fMRI imaging data on subjects playing a gambling task to evaluate nine ex ante hypotheses about the impact of gain/loss framing on risk-taking.”
Reproducible brain-wide association studies require thousands of individuals (2022, neutral): “Otherwise, we would be unable to disentangle differences in biology (rest vs. task) from differences arising from methodological variability (choice in processing, e.g., Botvinik-Nezer et al., 2020, Nature ).”
The OpenNeuro resource for sharing of neuroscience data (2021, positive): “…enhance the reuse of data by researchers who do not have the resources or expertise to preprocess these complex datasets as well as provide a standard baseline for downstream analyses, reducing the potential effects of analytic flexibility ( Botvinik-Nezer et al., 2020 ; Bowring et al., 2019 ).”
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis (2021, positive): “However, simulations and case studies suggest that the exploitation of researcher degrees of freedom could easily lead to spurious findings (Simmons, Nelson, & Simonsohn, 2011), coding different research articles from the same topic area reveals as many analytic approaches as there are publications ”
Cortical functional hierarchy disruption following subcortical stroke (2026, unknown): “Such discrepancies likely reflect the heterogeneity of lesion sites across cohorts as well as methodological variability—including differences in parcellation schemes, network definitions, preprocessing pipelines, and data quality [26, 27].”