Why We (Usually) Don't Have to Worry About Multiple Comparisons

By Andrew Gelman, Jennifer L. Hill, Masanao Yajima. 2009

1,331 citations recorded by the source corpus.

Abstract Applied researchers often find themselves making statistical inferences in settings that would seem to require multiple comparisons adjustments. We challenge the Type I error paradigm that underlies these corrections. Moreover we posit that the problem of multiple comparisons can disappear entirely when viewed from a hierarchical Bayesian perspective. We propose building multilevel models in the settings where multiple comparisons arise. Multilevel models perform partial pooling (shifting estimates toward each other), whereas classical procedures typically keep the centers of intervals stationary, adjusting for multiple comparisons by making the intervals wider (or, equivalently, adjusting the p values corresponding to intervals of fixed width). Thus, multilevel models address the multiple comparisons problem and also yield more efficient estimates, especially in settings with low group-level variation, which is where multiple comparisons are a particular concern.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/9288a36967840d7dfb8b6401420ebb97f590b0c0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Teacher–Child Relationships, Classroom Climate, and Children’s Social-Emotional and Academic Development (2017, background): “…related to social-emotional outcomes, with girls reporting significantly higher levels of anxiety (b .09, 1 Following the recommendations of Gelman, Hill, and Yajima (2012) for multilevel models with data from applied settings such as school classrooms, we do not correct for studywise Type…”
Bayesian data analysis. (2010, background): “Thus, estimates of outlying conditions are shrunk toward the grand mean, and this shrinkage attenuates false alarms (e.g., Berry & Hochberg, 1999; Gelman, 2005; Gelman et al., 2009; Lindquist & Gelman, 2009; Meng & Dempster, 1987).”
Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis (2016, methodology): “How does Bayesian Analysis mitigate false alarms? Gelman in (Gelman et al., 2012) suggests to use multilevel analysis (in our case a hierarchical Bayesian model on multiple classifiers).”
Stan: A Probabilistic Programming Language (2017, methodology): “Applying a hierarchical model then considering posterior inference appropriately adjusts for mulitple comparisons; see (Gelman, Hill, and Yajima 2012; Efron 2010) for discussion.”
False discovery rates: a new deal (2016, background): “the more meaningful question: “is the evidence strong enough to support a belief that the observed difference has the correct sign?” Along the same lines, Gelman and co-authors [15, 26] suggest focussing on “type S errors”, meaning errors in sign, rather than the more traditional type I errors.”
A Nearby Repeating Fast Radio Burst in the Direction of M81 (2021, methodology): “Bayesian hypothesis testing is another method proposed to avoid the problem of multiple testing (Scott & Berger 2006; Gelman et al. 2012).”
Bayesian estimation supersedes the t test. (2013, background): “Further details regarding so-called “hierarchical Bayesian ANOVA” are provided by Gelman (2005, 2006), Gelman, Hill, and Yajima (2011), and Kruschke (2010b, 2010a, 2011b).”
COCOA AND TOTAL SYSTEM YIELDS OF ORGANIC AND CONVENTIONAL AGROFORESTRY VS. MONOCULTURE SYSTEMS IN A LONG-TERM FIELD TRIAL IN BOLIVIA (2016, positive): “…multi-strata, zero external input system); different superscript letters indicate signiﬁcant difference between mean values (multilevel modelling approach according to Gelman et al. (2012), p (Diff > 0) < 0.05 p value and degrees of freedom (numDf: nominator Df, denDf: denominator Df) of ﬁxed…”