What do RNN Language Models Learn about Filler–Gap Dependencies?
By Ethan Gotlieb Wilcox, Roger Levy, Takashi Morita, Richard Futrell. 2018
206 citations recorded by the source corpus.
RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-of-the-art RNN language models represent long-distance filler–gap dependencies and constraints on them. Examining RNN behavior on experimentally controlled sentences designed to expose filler–gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text. Furthermore, we show that RNNs learn a subset of the known restrictions on filler–gap dependencies, known as island constraints: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands. These studies demonstrates that state-of-the-art RNN models are able to learn and generalize about empty syntactic positions.
Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Samuel R. Bowman
- Mohit Bansal
- Yoav Goldberg
- Tal Linzen
- Percy Liang
- Noah A. Smith
- Luke S. Zettlemoyer
In Natural Language Processing
- Yonatan Belinkov
- Samuel R. Bowman
- Tal Linzen
- Yoav Goldberg
- Mohit Bansal
- Noah A. Smith
- Douwe Kiela
- Percy Liang
In Computational Linguistics & Semantics
- Yonatan Belinkov
- Tal Linzen
- Yoav Goldberg
- Dieuwke Hupkes
- Roger Levy
- Peng Qian
- Ethan Gotlieb Wilcox
- Willem H. Zuidema
Citations in the corpus
- Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “…different continuations with high accuracy, even in difficult contexts (Linzen et al., 2016; Gulordava et al., 2018), for constructions such as subject-verb agreement (van Schijndel et al., 2019), filler-gap dependencies (Wilcox et al., 2018), and reflexive anaphora (Marvin and Linzen, 2018).”
- What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, background): “However, the majority of this work has focused on syntactic capabilities of LMs (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018; Wilcox et al., 2018; Futrell et al., 2019).”
- Linguistic Knowledge and Transferability of Contextual Representations (2019, background): “One particularly relevant line of work involves the construction of synthetic tasks that a model can only solve if it captures a particular phenomenon (Linzen et al., 2016; Jumelet and Hupkes, 2018; Wilcox et al., 2018; Futrell and Levy, 2019, inter alia ).”
- BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2019, methodology): “Recent studies (e.g., Linzen et al., 2016; Marvin and Linzen, 2018; Wilcox et al., 2018) have explored this question by evaluating LMs’ preferences between minimal pairs of sentences differing in grammatical acceptability as in Example (1).”
- BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance (2019, methodology): “We investigate this question using the task of natural language inference (NLI) by fine-tuning 100 instances of BERT (Devlin et al., 2019) on the MNLI dataset (Williams et al., 2018).”
- The emergence of number and syntax units in LSTM language models (2019, background): “Other recent work has extended syntax probing to other phenomena such as negative polarity items and island constraints (Chowdhury and Zamparelli, 2018; Jumelet and Hupkes, 2018; Marvin and Linzen, 2018; Wilcox et al., 2018).”
- Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (2019, methodology): “Wilcox et al. (2018) found evidence that large-data LSTMs are sensitive to adjunct islands, as evidenced by attenuated and often fully eliminated wh-licensing interactions for materials like (9-b)– (9-c) relative to (9-a) below.”
- Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (2019, background): “, 2018) and limitations (Chowdhury and Zamparelli, 2018; Marvin and Linzen, 2018; Wilcox et al., 2018).”
- Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment (2019, background): “In many later papers (e.g. Gulordava et al., 2018; Wilcox et al., 2018; Jumelet and Hupkes, 2018; Marvin and Linzen, 2018; Giulianelli et al., 2018) a wide spectrum of grammatical phenomena has been investigated, assessing these grammatical abilities in a mainly “behavioural” fashion, by…”
- A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020, background): “Test suites include Filler-Gap Dependencies (FGD) (6 suites) from Wilcox et al. (2018) and Wilcox et al. (2019b), and 2 novel Cleft suites, described in detail below.”
- Cross-Linguistic Syntactic Evaluation of Word Prediction Models (2020, background): “…(Marvin and Linzen, 2018; An et al., 2019), negative polarity item licensing (Marvin and Linzen, 2018; Jumelet and Hupkes, 2018), filler-gap dependencies (Chowdhury and Zamparelli, 2018; Wilcox et al., 2018), argument structure (Kann et al., 2019), and several others (Warstadt et al., 2019a).”
- GPT-4 passes the bar exam (2024, unknown): “20,21 37 The Uniform Bar Exam”
- Neural language models as psycholinguistic subjects: Representations of syntactic state (2019, methodology): “A similar measure was used to detect filler–gap dependencies by Wilcox et al. (2018).”
- UnNatural Language Inference (2020, background): “We show that state-of-the-art models do not rely on sentence structure the way we think they should: NLI models (Transformer-based models, RNNs, and ConvNets) are largely insensitive to permutations of word order that corrupt the original syntax.”
- Linguistic generalization and compositionality in modern artificial neural networks (2019, unknown): “, [49, 50, 51]).”
- When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022, background): “…Gulordava et al., 2018; Hu et al., 2020b), anaphora agreement and binding constraints (e.g., Marvin and Linzen, 2018; Futrell et al., 2019; Warstadt et al., 2020; Hu et al., 2020a), or filler-gap dependencies (e.g., Wilcox et al., 2018; Chowdhury and Zamparelli, 2018; Da Costa and Chaves, 2020).”