Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

By R. Silberzahn, E. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, Š. Bahník, F. Bai, Colin Bannard, E. Bonnier and 55 more. Advances in Methods and Practices in Psychological Science, 2018

668 citations recorded by the source corpus.

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

Subfields: Applied Psychology, Economics and Econometrics, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/9dcc8921fb80e2321c20153c1154bab647cf3640

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Economics and Econometrics

In Statistics and Probability

Citations in the corpus

How Computational Modeling Can Force Theory Building in Psychological Science (2020, methodology): “This helps to contextualize studies that show divergence in data-modeling decisions given the same hypotheses (e.g., Botvinik-Nezer et al., 2020; Silberzahn et al., 2018).”
Fairness GAN (2018, methodology): “This dataset was originally assembled for a unique crowdsourcing experiment testing whether different statisticians will find evidence of racial discrimination in the calling of offenses by referees [32].”
A Hierarchy of Limitations in Machine Learning (2020, background): “The essential variability of the modeling process itself is on display in the amazing study of Silberzahn et al. (2018), where twenty-nine teams of statistical analysts given the same data set and the same target quantity to estimate, and not a single team used the same statistical model!”
The generalizability crisis (2019, background): “Hence, it is critical to find ways to share raw data, processing pipelines, and analysis scripts for digital trace research, as the degrees of freedom are vast, which causes large variance in conclusions made from the same data (Silberzahn et al., 2018; Towse et al., 2020).”
The influence of hidden researcher decisions in applied microeconomics (2020, positive): “Previous many-analysts studies include Silberzahn et al. (2018), which recruited psychological researchers to examine whether a data set of referee calls in soccer showed evidence of discrimination against darker-skinned players.”
Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence (2019, background): “However, like primary studies (Silberzahn et al., 2018), meta-analyses involve researchers’ degrees of freedom; therefore, one and the same question addressed with meta-analytical methods could lead to very different statistical approaches and answers (for diverging metaanalytical evidence on media ”
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis (2021, background): “…that complexity and ambiguity are positively correlated with dispersed results across different analysts, our findings may have wide implications for the conclusions drawn from analyses of complex datasets (see also Bastiaansen et al., 2020; Botvinik-Nezer et al., 2020; Silberzahn et al., 2018).”
50 Years of Data Science (2017, background): “Calling on examples like Cleveland’s study of graphical perception (Cleveland et al. 1985), Leek advocates for data scientists experimenting to learn how software syntax impacts learning, and how practitioners are actually working (Silberzahn et al. 2017).”