Balancing Type I Error and Power in Linear Mixed Models

By Hannes Matuschek, Reinhold Kliegl, Shravan Vasishth, Harald Baayen, Douglas M. Bates. 2015

1,743 citations recorded by the source corpus.

Linear mixed-effects models have increasingly replaced mixed-model analyses of variance for statistical inference in factorial psycholinguistic experiments. Although LMMs have many advantages over ANOVA, like ANOVAs, setting them up for data analysis also requires some care. One simple option, when numerically possible, is to fit the full variance-covariance structure of random effects (the maximal model; Barr et al. 2013), presumably to keep Type I error down to the nominal alpha in the presence of random effects. Although it is true that fitting a model with only random intercepts may lead to higher Type I error, fitting a maximal model also has a cost: it can lead to a significant loss of power. We demonstrate this with simulations and suggest that for typical psychological and psycholinguistic data, higher power is achieved without inflating Type I error rate if a model selection criterion is used to select a random effect structure that is supported by the data.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/cb1a42d9458e2b847054d2ea2f6ff4e3dd99ad45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

The cave of Shadows. Addressing the human factor with generalized additive mixed models (2015, methodology): “, 2015), and accepting less than nominal power (Matuschek et al., 2016)? The problem here is that low-hanging fruit is easily plucked, often by simple linear models without any random effects.”
Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R (2020, methodology): “Note that while we work under the assumption that the final random-effect structure has already been selected, the complexity of the random-effect structure has been shown to have an influence on power as well — that is, power decreases with model complexity (Matuschek et al., 2017).”
Parsimoniously Fitting Large Multivariate Random Effects in glmmTMB (2024, positive): “The reduced-rank model contributes to the model-simplification toolbox, allowing for a more parsimonious random effect which may be necessary for some study designs (Matuschek, Kliegl, Vasishth, Baayen, and Bates 2017).”
An Introduction to Mixed Models for Experimental Psychology (2019, positive): “However, this does not necessarily a ﬀ ect the tests of ﬁxed e ﬀ ects (but see Matuschek et al., 2017).”
Best practice guidance for linear mixed-effects models in psychological science (2019, methodology): “LRTs apply when models are nested (the terms of the simpler model appear in the more complex model) and the models are compared in a pairwise fashion (see discussions in Luke, 2016; Matuschek et al., 2017).”
Analyzing dynamic phonetic data using generalized additive mixed modeling: A tutorial focusing on articulatory differences between L1 and L2 speakers of English (2018, neutral): “The advantage of such an approach is that it does not result in a lack of power (as the maximal approach does; Matuschek, Kliegl, Vasishth, Baayen, & Bates, 2017) and is more suitable to be used in conjunction with generalized additive modeling (Baayen, Vasishth, Kliegl, & Bates, 2017).”
Theta-Phase Connectivity between Medial Prefrontal and Posterior Areas Underlies Novel Instructions Implementation (2022, methodology): “, 2014), adopting a backward selection approach to define the random effect structure of the model (Matuschek et al., 2017).”