Random effects structure for confirmatory hypothesis testing: Keep it maximal
By Dale J. Barr, Roger Levy, Christoph Scheepers, Harry J. Tily. Journal of Memory and Language, 2013
9,753 citations recorded by the source corpus.
Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholinguistics and related areas. However, many researchers do not seem to appreciate how random effects structures affect the generalizability of an analysis. Here, we argue that researchers using LMEMs for confirmatory hypothesis testing should minimally adhere to the standards that have been in place for many decades. Through theoretical arguments and Monte Carlo simulation, we show that LMEMs generalize best when they include the maximal random effects structure justified by the design. The generalization performance of LMEMs including data-driven random effects structures strongly depends upon modeling criteria and sample size, yielding reasonable results on moderately-sized samples when conservative criteria are used, but with little or no power advantage over maximal models. Finally, random-intercepts-only LMEMs used on within-subjects and/or within-items data from populations where subjects and/or items vary in their sensitivity to experimental manipulations always generalize worse than separate F1 and F2 tests, and in many cases, even worse than F1 alone. Maximal LMEMs should be the ‘gold standard’ for confirmatory hypothesis testing in psycholinguistics and beyond.
Subfields: Management Science and Operations Research, Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/7c1e5c788c4fd8c5a17118ac300a05bc93a96f94
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Management Science and Operations Research
- Judith Degen
- Steven G. Luke
- Jalal Al-Tamimi
- Robert A. I. Davies
- Lotte Meteyard
- Ian Cunnings
- Richard Futrell
- J. Wiley
In Statistics and Probability
- Robert A. I. Davies
- Lotte Meteyard
- Christopher Eager
- Joseph Roy
- Jalal Al-Tamimi
- Steven G. Luke
- Shravan Vasishth
- Harald Baayen
Citations in the corpus
- Investigating the distribution of some (but not all ) implicatures using corpora and web-based methods (2015, methodology): “…suggests two things: a) the Homogeneity Assumption seems to be more warranted when the some-NP is overtly partitive and b) the partitive is nevertheless not sufficient for unambiguously generating a scalar 17 Barr et al. (2013) recommend using the maximal random effects structure whenever possible.”
- The nature and specificity of paired associate learning deficits in children with dyslexia (2014, result): “Such an approach has been shown to be more conservative than traditional by-subjects and by-items ANOVA (Barr et al., 2012).”
- But You Don’t Look Like A Scientist!: Women Scientists with Feminine Appearance are Deemed Less Likely to be Scientists (2016, neutral): “All possible random intercepts and slopes were estimated (Barr et al. 2013), but the covariances between the random effects were not estimated due to convergence problems (estimating covariances involved estimating an additional 81 parameters).”
- Why You Should Always Include a Random Slope for the Lower-Level Variable Involved in a Cross-Level Interaction (2019, neutral): “, 2017) and for statistical inference (Barr et al., 2013; Bell et al., 2016; Berkhof and Kampen, 2004; Bryan and Jenkins, 2016; Schmidt-Catran and Fairbrother, 2016).”
- Random effects structure for testing interactions in linear mixed-effects models (2013, methodology): “…Testing Service, USA Reviewed by: Hugo Quené, Utrecht University, Netherlands In a recent paper on mixed-effects models for confirmatory analysis, Barr et al. (2013) offered the following guideline for testing interactions: “one should have byunit [subject or item] random slopes for any…”
- Analyzing dynamic phonetic data using generalized additive mixed modeling: A tutorial focusing on articulatory differences between L1 and L2 speakers of English (2018, critical): “…Barr, Levy, Scheepers, and Tilly (2013) was important in that it made researchers aware that a random-effects structure only consisting of random intercepts is often problematic, we are not in favor of an approach in which the maximally possible random-effects structure is used (Barr et al., 2013).”
- A brief introduction to mixed effects modelling and multi-model inference in ecology (2018, background): “Therefore, the approach of fitting the ‘maximal’ complexity of random effects structure (Barr et al 2013) is perhaps better phrased as fitting the most complex mixed effects structure allowed by your data (Bates et al 2015), which may mean no random slopes at all.”
- To Trust or to Think (2021, background): “, all the measurements were not statistically independent) [3].”
- What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021, background): “As a ‘maximal model’ (Barr et al., 2013), it includes random intercepts to represent the general bias of individual participants and items (e.g. some participants give higher or lower ratings on average) and random slopes to represent the interactions between random and fixed effects (e.g. some…”
- Predictive translation memory: a mixed-initiative system for human language translation (2014, background): “The maximal random effects structure [5] includes random intercepts and slopes for subject, source sentence, and text genre.”
- RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency (2018, background): “Random slopes and interactions are not necessary in these models to avoid anti-conservativity (Barr et al., 2013) because we do not have repeated observations within any item/condition combination.”