The generalizability crisis

By Tal Yarkoni. 2019

596 citations recorded by the source corpus.

Abstract Most theories and hypotheses in psychology are verbal in nature, yet their evaluation overwhelmingly relies on inferential statistical procedures. The validity of the move from qualitative to quantitative analysis depends on the verbal and statistical expressions of a hypothesis being closely aligned – that is, that the two must refer to roughly the same set of hypothetical observations. Here, I argue that many applications of statistical inference in psychology fail to meet this basic condition. Focusing on the most widely used class of model in psychology – the linear mixed model – I explore the consequences of failing to statistically operationalize verbal hypotheses in a way that respects researchers' actual generalization intentions. I demonstrate that although the “random effect” formalism is used pervasively in psychology to model intersubject variability, few researchers accord the same treatment to other variables they clearly intend to generalize over (e.g., stimuli, tasks, or research sites). The under-specification of random effects imposes far stronger constraints on the generalizability of results than most researchers appreciate. Ignoring these constraints can dramatically inflate false-positive rates, and often leads researchers to draw sweeping verbal generalizations that lack a meaningful connection to the statistical quantities they are putatively based on. I argue that failure to take the alignment between verbal and statistical expressions seriously lies at the heart of many of psychology's ongoing problems (e.g., the replication crisis), and conclude with a discussion of several potential avenues for improvement.

Subfields: Experimental and Cognitive Psychology, History and Philosophy of Science, Statistics and Probability, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In History and Philosophy of Science

In Statistics and Probability

In Cognitive Science & Psychology

Citations in the corpus

Can language models learn from explanations in context? (2022, methodology): “Thus, appropriate hierarchical models are essential to generalizable research (Yarkoni, 2022).”
Lack of Theory Building and Testing Impedes Progress in The Factor and Network Literature (2020, background): “(Yarkoni, 2019) The first problematic way in which the literature falls short of theory building and testing is the conflation of theoretical and statistical models.”
Language models show human-like content effects on reasoning (2022, background): “Gelman and Hill, 2006); such analytic methods are broadly used in the behavioral sciences because they allow drawing generalizable inferences from data (Yarkoni, 2022), and they are similarly useful tools for analyzing the behaviors of large language models (Lampinen et al.”