Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

By Tal Linzen, Emmanuel Dupoux, Yoav Goldberg. 2016

1,000 citations recorded by the source corpus.

The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture’s grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “Linzen et al. (2016) assessed whether RNNs can learn number agreement in subject-verb dependencies.”
Neural Network Acceptability Judgments (2018, methodology): “This contributes to a growing effort to test ANNs’ ability to make finegrained grammatical distinctions (Linzen et al., 2016; Adi et al., 2017; Conneau et al., 2018; Ettinger et al., 2018; Marvin and Linzen, 2018).”
Universal Language Model Fine-tuning for Text Classification (2018, background): “syntax-sensitive dependencies (Linzen et al., 2016) to create a model”
Semi-supervised sequence tagging with bidirectional language models (2017, background): “Recently, there has been some interest in interpreting the activations of RNNs. Linzen et al. (2016) showed that single LSTM units can learn to predict singular-plural distinctions.”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “…based on subject-verb agreement, a phenomenon that requires a distinction to be made between the subject and direct object of a sentence (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018); it is unlikely that our ﬁne-tuning step on MNLI substantially changed the…”
Universal Transformers (2018, methodology): “Next, we consider the task of predicting number agreement between subject and verb in naturally occurring English sentences [20].”
Fine-tuned Language Models for Text Classification (2018, methodology): “Language modeling could also be augmented with additional tasks in a multi-task learning fashion (Caruana, 1993) or enriched with additional supervision, e.g. syntax-sensitive dependencies (Linzen et al., 2016) to create a model that is more general or better suited for certain downstream tasks.”
How Can We Know What Language Models Know? (2019, background): “…by using extrinsic probing tasks to examine whether certain linguistic properties can be predicted from those representations (Shi et al., 2016; Linzen et al., 2016; Belinkov et al., 2017), or by ablations to the models to investigate how behavior varies (Li et al., 2016b; Smith et al., 2017).”
Quantifying Attention Flow in Transformers (2020, positive): “We use the subject-verb agreement dataset (Linzen et al., 2016).”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “…(Del Corso et al., 2005), Evidence Inference (Lehman et al., 2019), Movie Reviews (Zaidan and Eisner, 2008), MultiRC (Khashabi et al., 2018), LGD (Linzen et al., 2016), 20 News (Lang, 1995), Amazon Re-views (McAuley and Leskovec, 2013), Beer Re-views (McAuley et al., 2012), BoolQ (Clark et al.,…”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “An alternative method to study the ability of language models and machine translation models to capture hierarchical information is to test their sensitivity to speciﬁc grammatical errors (Linzen et al., 2016; Gulordava et al., 2018; Tran et al., 2018; Sennrich, 2017; Tang et al., 2018).”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “…are able to predict the correct token from a set of grammatically minimally different continuations with high accuracy, even in difficult contexts (Linzen et al., 2016; Gulordava et al., 2018), for constructions such as subject-verb agreement (van Schijndel et al., 2019), filler-gap dependencies…”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “Many papers have found neural models (especially PLMs) to naturally encode syntax (Linzen et al., 2016; Belinkov et al., 2017; Marvin and Linzen, 2018; Belinkov and Glass, 2019; Goldberg, 2019; Hewitt and Manning, 2019).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Linzen et al. (2016), for instance, found that long short-term memory (LSTM) language models are able to capture subject-verb agreement in many common cases, while direct supervision is required for solving harder cases.”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “…of recurrent networks and related models by using visualization, input/hidden representation deletion techniques or by looking at the word-by-word behaviour of the network (e.g., Nagamine et al., 2015; Hupkes et al., 2017; Li et al., 2016; Linzen et al., 2016; Kàdàr et al., 2017; Li et al., 2017).”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “One line of research examines the outputs of language models on carefully chosen input sentences (Linzen et al., 2016; Khandelwal et al., 2018; Gulordava et al., 2018; Marvin and Linzen, 2018).”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “Other related work uses perplexity scores to test whether language models learn to encode properties such as subject-verb agreement (Linzen et al., 2016; Gulordava et al., 2018; Marvin & Linzen, 2018; Kuncoro et al., 2018).”
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, background): “Most strikingly, however, we ﬁnd that BERT fails completely to show generalizable understanding of negation, raising questions about the aptitude of LMs to learn this type of meaning.”