The effect of word predictability on reading time is logarithmic

By Nathaniel J. Smith, Roger Levy. Cognition, 2013

758 citations recorded by the source corpus.

It is well known that real-time human language processing is highly incremental and context-driven, and that the strength of a comprehender’s expectation for each word encountered is a key determinant of the difficulty of integrating that word into the preceding context. In reading, this differential difficulty is largely manifested in the amount of time taken to read each word. While numerous studies over the past thirty years have shown expectation-based effects on reading times driven by lexical, syntactic, semantic, pragmatic, and other information sources, there has been little progress in establishing the quantitative relationship between expectation (or prediction) and reading times. Here, by combining a state-of-the-art computational language model, two large behavioral data-sets, and non-parametric statistical techniques, we establish for the first time the quantitative form of this relationship, finding that it is logarithmic over six orders of magnitude in estimated predictability. This result is problematic for a number of established models of eye movement control in reading, but lends partial support to an optimal perceptual discrimination account of word recognition. We also present a novel model in which language processing is highly incremental well below the level of the individual word, and show that it predicts both the shape and time-course of this effect. At a more general level, this result provides challenges for both anticipatory processing and semantic integration accounts of lexical predictability effects. And finally, this result provides evidence that comprehenders are highly sensitive to relative differences in predictability – even for differences between highly unpredictable words – and thus helps bring theoretical unity to our understanding of the role of prediction at multiple levels of linguistic structure in real-time language comprehension.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Developmental and Educational Psychology

View Paper: https://www.semanticscholar.org/paper/89774f17d4ae199ea6950b747bf21e390a6e023a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Developmental and Educational Psychology

Citations in the corpus

Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (2019, background): “The surprisal, or negative log-conditional probability, S(xi) of a sentence’s ith word xi, tells us how strongly xi is expected in context and is also known to correlate with human processing difficulty (Smith and Levy, 2013; Hale, 2001; Levy, 2008).”
Neural language models as psycholinguistic subjects: Representations of syntactic state (2019, background): “These reading times are often taken to reﬂect the extent to which humans expect certain words in context, and may be generally proportional to surprisal given the comprehen-der’s probabilistic language model (Hale, 2001; Levy, 2008; Smith and Levy, 2013).”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, background): “The rich sources of information that comprehenders combine to interpret language—including lexical and syntactic information, world knowledge, and information about others’ mental states (115–119)—can be used to make informed guesses about how the linguistic signal may unfold, and much behavioral an”
Communicative efficiency in language production: Optional case-marking in Japanese (2015, background): “Plausibility ratings were normalized (so that they ranged from 0 to 1) and logtransformed (following other work that has shown that log-transformed, rather than raw, predictability affects language processing, Smith & Levy, 2013, and production, Jaeger, 2006).”
RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency (2018, background): “These reading times are often taken to reﬂect the extent to which humans expect certain words in context, and may be generally proportional to surprisal given the com-prehender’s probabilistic language model (Hale, 2001; Levy, 2008; Smith and Levy, 2013).”
A hierarchy of linguistic predictions during natural language comprehension (2020, methodology): “We translated the idea of all-or-none predic1025 tion into a regression model using an insight by Smith and 1026 Levy which implied that all-or-none predictions result in 1027 a linear relationship between word probability and brain 1028 responses [11].”
Investigating locality effects and surprisal in written English syntactic choice phenomena. (2016, positive): “More predictable words induce faster processing times in reading (Boston, Hale, Patil, Kliegl, & Vasishth 2008, Demberg & Keller 2008, Smith & Levy 2013).”
Investigating the time-course of phonological prediction in native and non-native speakers of English: A visual world eye-tracking study (2018, background): “We know that successful 26 prediction can facilitate comprehension (Frisson, Rayner, & Pickering, 2005; Rayner, Slattery, Drieghe, & Liversedge, 2011; Smith & Levy, 2013), and these limitations may underlie some of the difficulty of L2 comprehension.”
What do we mean by prediction in language comprehension? (2015, background): “…& Rayner, 1981; Rayner, Binder, Ashby, & Pollatsek, 2001; Rayner & Well, 1996; see also Boston, Hale, Kliegl, Patil, & Vasishth, 2008; Demberg & Keller, 2008; Demberg, Keller, & Koller, 2013; Frank & Bod, 2011; McDonald & Shillcock, 2003; Smith & Levy, 2013; see Staub, 2015 for a recent review).”
Rapid Expectation Adaptation during Syntactic Comprehension (2013, background): “The predicted change in reading times at the disambiguating region is further exaggerated due to the log-linear relation between linguistic probabilities and reading times [56].”
Statistical Measures for Usage‐Based Linguistics (2015, positive): “…(Clark, 2013; Rescorla & Wagner, 1972; Rumelhart, Hinton, & Williams, 1986; Wills, 2009), and there is increasing evidence for surprisal-driven language processing and acquisition (Dell & Chang, in press; Demberg & Keller, 2008; Jaeger & Snider, 2013; Pickering & Garrod, 2013; Smith & Levy, 2013).”
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020, background): “In probabilistic language models, these garden-path disambiguation effects are well captured by word negative log probabilities, or SURPRISALS (Hale, 2001): S(w|C) = − log2 p(w|C), which are independently well-established to predict human incremental processing difficulty over several orders of magn”
What do RNN Language Models Learn about Filler–Gap Dependencies? (2018, background): “It is known to correlate directly with human sentence processing difﬁculty (Hale, 2001; Levy, 2008; Smith and Levy, 2013).”