Random effects structure for confirmatory hypothesis testing: Keep it maximal

By Dale J. Barr, Roger Levy, Christoph Scheepers, Harry J. Tily. Journal of Memory and Language, 2013

9,753 citations recorded by the source corpus.

Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholinguistics and related areas. However, many researchers do not seem to appreciate how random effects structures affect the generalizability of an analysis. Here, we argue that researchers using LMEMs for confirmatory hypothesis testing should minimally adhere to the standards that have been in place for many decades. Through theoretical arguments and Monte Carlo simulation, we show that LMEMs generalize best when they include the maximal random effects structure justified by the design. The generalization performance of LMEMs including data-driven random effects structures strongly depends upon modeling criteria and sample size, yielding reasonable results on moderately-sized samples when conservative criteria are used, but with little or no power advantage over maximal models. Finally, random-intercepts-only LMEMs used on within-subjects and/or within-items data from populations where subjects and/or items vary in their sensitivity to experimental manipulations always generalize worse than separate F1 and F2 tests, and in many cases, even worse than F1 alone. Maximal LMEMs should be the ‘gold standard’ for confirmatory hypothesis testing in psycholinguistics and beyond.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/7c1e5c788c4fd8c5a17118ac300a05bc93a96f94

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

  1. Judith Degen
  2. Steven G. Luke
  3. Jalal Al-Tamimi
  4. Robert A. I. Davies
  5. Lotte Meteyard
  6. Ian Cunnings
  7. Richard Futrell
  8. J. Wiley

In Statistics and Probability

  1. Robert A. I. Davies
  2. Lotte Meteyard
  3. Christopher Eager
  4. Joseph Roy
  5. Jalal Al-Tamimi
  6. Steven G. Luke
  7. Shravan Vasishth
  8. Harald Baayen

Citations in the corpus