Neural language models as psycholinguistic subjects: Representations of syntactic state

By Richard Futrell, Ethan Gotlieb Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, Roger Levy. 2019

237 citations recorded by the source corpus.

We investigate the extent to which the behavior of neural network language models reflects incremental representations of syntactic state. To do so, we employ experimental methodologies which were originally developed in the field of psycholinguistics to study syntactic representation in the human mind. We examine neural network model behavior on sets of artificial sentences containing a variety of syntactically complex structures. These sentences not only test whether the networks have a representation of syntactic state, they also reveal the specific lexical cues that networks use to update these states. We test four models: two publicly available LSTM sequence models of English (Jozefowicz et al., 2016; Gulordava et al., 2018) trained on large datasets; an RNN Grammar (Dyer et al., 2016) trained on a small, parsed dataset; and an LSTM trained on the same small corpus as the RNNG. We find evidence for basic syntactic state representations in all models, but only the models trained on large datasets are sensitive to subtle lexical cues signaling changes in syntactic state.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Natural Language Processing, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Natural Language Processing

In Cognitive Science & Psychology

Citations in the corpus

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “…have studied whether models prefer grammatical completions in subjectverb agreement contexts (Marvin and Linzen, 2018; van Schijndel et al., 2019; Goldberg, 2019; Mueller et al., 2020; Lakretz et al., 2021; Futrell et al., 2019), as well as in filler-gap dependencies (Wilcox et al., 2018, 2019).”
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, neutral): “This work is particularly dominated by testing of syntactic awareness in LMs, and often mirrors the present work in using targeted evaluations modeled after psycholinguistic tests (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018;Wilcox et al., 2018; Chowdhury and Zamparelli, 201”
Cross-Linguistic Syntactic Evaluation of Word Prediction Models (2020, background): “To address these limitations, a growing number of studies have used constructed materials, which improve experimental control and coverage of syntactic constructions (Marvin and Linzen, 2018; Wilcox et al., 2018; Futrell et al., 2019; Warstadt et al., 2019a).”
Learning to Recognize Dialect Features (2020, methodology): “Minimal pairs have also been used to design controlled experiments and probe neural models’ ability to capture various linguistic phenomena (Gulordava et al., 2018; Ettinger et al., 2018; Futrell et al., 2019; Gardner et al., 2020; Schuster et al., 2020).”
Refining Targeted Syntactic Evaluation of Language Models (2021, methodology): “cessing (Futrell et al., 2019).”
When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022, methodology): “paradigm (Futrell et al., 2019) for discourse entity tracking and discuss the methodological challenges that arise with using this paradigm for discourse phenomena.”
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020, methodology): “References Bever (1970); Ferreira and Clifton (1986); Trueswell et al. (1994); van Schijndel and Linzen (2018); Futrell et al. (2019)”