Colorless Green Recurrent Networks Dream Hierarchically

By Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni. 2018

562 citations recorded by the source corpus.

Recurrent neural networks (RNNs) achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues (“The colorless green ideas I ate with the chair sleep furiously”), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow-pattern extractors, but they also acquire deeper grammatical competence.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “, 2016), hierarchical relations (Gulordava et al., 2018), and sentiment (Radford et al.”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “…agreement, a phenomenon that requires a distinction to be made between the subject and direct object of a sentence (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018); it is unlikely that our ﬁne-tuning step on MNLI substantially changed the model’s representational…”
A Survey of Data Augmentation Approaches for NLP (2021, background): “, 2019) 3 × tok Input 3 NONCE (Gulordava et al., 2018) 3 × const Input 3 XLDA (Singh et al.”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “An alternative method to study the ability of language models and machine translation models to capture hierarchical information is to test their sensitivity to speciﬁc grammatical errors (Linzen et al., 2016; Gulordava et al., 2018; Tran et al., 2018; Sennrich, 2017; Tang et al., 2018).”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “, 2019)—as psycholinguistic subjects to be studied behaviorally (Linzen et al., 2016; Gulordava et al., 2018; Goldberg, 2019).”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “Gulordava et al. (2018) extended this to other agreement phenomena, but they relied on syntactic information available in treebanks, resulting in a smaller dataset.”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “One line of research examines the outputs of language models on carefully chosen input sentences (Linzen et al., 2016; Khandelwal et al., 2018; Gulordava et al., 2018; Marvin and Linzen, 2018).”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “Other related work uses perplexity scores to test whether language models learn to encode properties such as subject-verb agreement (Linzen et al., 2016; Gulordava et al., 2018; Marvin & Linzen, 2018; Kuncoro et al., 2018).”
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, neutral): “This work is particularly dominated by testing of syntactic awareness in LMs, and often mirrors the present work in using targeted evaluations modeled after psycholinguistic tests (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018;Wilcox et al., 2018; Chowdhury and Zamparelli, 201”
A Structural Probe for Finding Syntax in Word Representations (2019, background): “Recent work has analyzed model behavior to determine if a model understands hierarchy and other linguistic phenomena (Linzen, 2018; Gulordava et al., 2018; Kuncoro et al., 2018; Linzen and Leonard, 2018; van Schijndel and Linzen, 2018; Tang et al., 2018; Futrell et al., 2018).”
Assessing BERT's Syntactic Abilities (2019, methodology): “This differs from Linzen et al. (2016) and Gulordava et al. (2018) by considering the entire sentence (excluding the verb) and not just its preﬁx leading to the verb, and differs from Marvin and Linzen (2018) by conditioning the focus verb on bidirectional context.”
Targeted Syntactic Evaluation of Language Models (2018, methodology): “Following previous work (Lau et al., 2017; Linzen et al., 2016; Gulordava et al., 2018), we suggest that given two sentences that differ minimally from each other, one of which is grammatical and the other which is not, it is desirable for the model to assign a higher probability to the grammatical ”
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (2019, background): “A wide range of literature analyzes linguistic knowledge stored in pretrained embeddings (Jumelet and Hupkes, 2018; Gulordava et al., 2018; Giulianelli et al., 2018; McCoy et al., 2019; Dasgupta et al., 2018; Marvin and Linzen, 2018; Warstadt and Bowman, 2019; Kann et al., 2019).”
Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information (2018, methodology): “…et al., 2018), using the pretrained LM and the English test set made available by the authors of the paper.2 Following Linzen et al. (2016) and Gulordava et al. (2018), we use the LSTM language model to process a corpus of sentences containing long-distance subjectverb relations, and test…”
Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “A flurry of recent work (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018; Lakretz et al., 2019) has explored how recurrent neural language models perform in cases that require sensitivity to hierarchical syntactic structure, and study how they do so, particularly in the domain…”
Deep RNNs Encode Soft Hierarchical Syntax (2018, neutral): “However, the more recent investigation by Gulordava et al. (2018) are in line with our results.”
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2019, positive): “§ 6.1 discusses evidence that increased dependency length and the presence of agreement attractors of the kind investigated by Linzen et al. (2016) and Gulordava et al. (2019) reduce performance on agreement phenomena.”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “Nowadays, given its popularity, recurrent neural networks are the most evaluated networks, mainly investigated on the structures and linguistic properties they are encoding (Linzen et al., 2016; Enguehard et al., 2017; Kuncoro et al., 2017; Gulordava et al., 2018).”