The Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research

By Andrew Gelman. 2015

125 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, General Social Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In General Social Sciences

Citations in the corpus

Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “As such, causal methods can be used to study interference [97, 87] and heterogeneous effects [45].”
Best practice guidance for linear mixed-effects models in psychological science (2019, background): “More generally, an analysis that fails to account for potential differences between sampling units in the slopes of experimental variables can mis-estimate the robustness of observed effects (Gelman, 2014).”
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis (2021, background): “However, as many scholars have argued, there are often numerous defensible analytic specifications that could be used on the same data, raising the issue of whether variations in such specifications might produce qualitatively different outcomes (Bamberger, 2019; Cortina, Green, Keeler, & Vandenberg”
Beyond subjective and objective in statistics (2015, background): “…the underlying principles are subtle Instead of pointing at speciﬁc bad examples, here is a list of some common problems (discussed, for example, in Gelman, 2015, and Gelman and Zelizer, 2015), where we believe that exercising one or more of our listed virtues would improve matters: • Presenting…”
Adjusting for Publication Bias in Meta-Analysis (2016, methodology): “Given the degree of heterogeneity present in the Many Labs studies (for which the only difference among the studies was the location of the laboratory) and in the Mechanical Turk subsample of these studies (for which the only difference among the studies was the day on which the study materials were”
Three ways to make replication mainstream (2017, background): “ortantly, they can also account for differences in methods factors such as dependent variables, moderators, and study designs (McShane and Bockenholt, 2017a, 2017b) and for varying treatment effects (Gelman, 2015) thereby allowing for a much richer characterization of a research domain and applicati”
Abandon Statistical Significance (2017, background): “…measurements, a greater use of within-person or longitudinal designs, and increased consideration of models that use informative priors, that feature varying treat-ment eﬀects, and that are multilevel or meta-analytic in nature (Gelman, 2015, 2017; McShane and B¨ockenholt, 2017, 2018).”
A researcher's guide to regression, discretization, and median splits of continuous variables (2015, methodology): “In this setting, the approach of Gelman and Park (2009) of splitting the data into three groups and discarding the middle group seems even more advantageous for researchers considering dichotomization.”
Beyond Power Calculations (2014, result): “…have been gathered, such an analysis can and should be informative to a researcher and, in this case, should suggest that, even aside from other issues (see Gelman, 2014), this statistically significant result provides only very weak evidence about the pattern of interest in the larger population.”