LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

By A. Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, Phil Blunsom. 2018

148 citations recorded by the source corpus.

Language exhibits hierarchical structure, but recent work using a subject-verb agreement diagnostic argued that state-of-the-art language models, LSTMs, fail to learn long-range syntax sensitive dependencies. Using the same diagnostic, we show that, in fact, LSTMs do succeed in learning such dependencies—provided they have enough capacity. We then explore whether models that have access to explicit syntactic information learn agreement more effectively, and how the way in which this structural information is incorporated into the model impacts performance. We find that the mere presence of syntactic information does not improve accuracy, but when model architecture is determined by syntax, number agreement is improved. Further, we find that the choice of how syntactic structure is built affects how well number agreement is learned: top-down construction outperforms left-corner and bottom-up variants in capturing non-local structural dependencies.

Subfields: Artificial Intelligence, Model Architectures, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Computational Linguistics & Semantics

Citations in the corpus

What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “Other related work uses perplexity scores to test whether language models learn to encode properties such as subject-verb agreement (Linzen et al., 2016; Gulordava et al., 2018; Marvin & Linzen, 2018; Kuncoro et al., 2018).”
A Structural Probe for Finding Syntax in Word Representations (2019, background): “Recent work has analyzed model behavior to determine if a model understands hierarchy and other linguistic phenomena (Linzen, 2018; Gulordava et al., 2018; Kuncoro et al., 2018; Linzen and Leonard, 2018; van Schijndel and Linzen, 2018; Tang et al., 2018; Futrell et al., 2018).”
Targeted Syntactic Evaluation of Language Models (2018, background): “In previous work, perplexity and syntactic judgment accuracy have been found to be partly dissociable (Kuncoro et al., 2018; Tran et al., 2018).”
Visualizing and Understanding the Effectiveness of BERT (2019, background): “Recent work of inspecting the effectiveness of the pre-trained models (Linzen et al., 2016; Kuncoro et al., 2018; Tenney et al., 2019b; Liu et al., 2019a) focuses on analyzing the syntactic and semantic properties.”
Emergent linguistic structure in artificial neural networks trained by self-supervision (2020, background): “Subsequent work has evaluated models on more challenging sentences where potential confounds are removed (25), evaluated models on other grammatical phenomena such as reflexive anaphora (26), evaluated models that explicitly model hierarchy (27), and evaluated BERT on agreement across attractors (28”
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (2019, background): “, 2016), and outperform linear models at learning subject-verb number agreement (Kuncoro et al., 2018).”
Learning compositionally through attentive guidance (2018, background): “An increasing amount of hybrid approaches that explicitly build in compositional components in connectionist models (Reed and De Freitas 2015; Socher, Manning, and Ng 2010; Kuncoro et al. 2018), but little is known about inducing compositional solutions in purely connectionist models, such LSTMs (Ho”
Cross-Linguistic Syntactic Evaluation of Word Prediction Models (2020, background): “…the performance of the two architectures on grammatical evaluations are mixed (Tran et al., 2018; van Schijndel et al., 2019), and the best reported syntactic performance on English grammatical evaluations comes from LMs trained with explicit syntactic supervision (Kuncoro et al., 2018, 2019).”
Cross-Domain Generalization of Neural Constituency Parsers (2019, background): “…modeling of syntactic structure have obtained strong performance on parsing (Dyer et al., 2016; Liu and Zhang, 2017) and a range of related tasks (Kuncoro et al., 2018; Hale et al., 2018), recent neural parsers obtain state-of-the-art F1 on benchmark datasets using rich input encoders without…”
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (2019, background): “…without syntactic annotations; these studies have identiﬁed both strengths (Linzen et al., 2016; Giulianelli et al., 2018; Gulordava et al., 2018; Kuncoro et al., 2018; van Schijndel and Linzen, 2018; Wilcox et al., 2018) and limitations (Chowdhury and Zampar-elli, 2018; Marvin and Linzen, 2018;…”
Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, background): “…to struggle with syntactic generalization and that quantitative performance metrics such as perplexity can partly be dissociated from syntactic knowledge displayed on more challenging benchmarks (e.g., Kuncoro et al. (2018); Marvin and Linzen (2018); Tran et al. (2018); McCoy et al. (2019)).”
Refining Targeted Syntactic Evaluation of Language Models (2021, background): “The syntactically correct of the two sentences is sampled from natural corpora (Linzen et al., 2016; Kuncoro et al., 2018) or constructed from templates.”
Unsupervised Recurrent Neural Network Grammars (2019, background): “…shown to outperform standard sequential language models, achieve excellent results on parsing (Dyer et al., 2016; Kun-coro et al., 2017), better encode syntactic properties of language (Kuncoro et al., 2018), and correlate with electrophysiological responses in the human brain (Hale et al., 2018).”
Neural language models as psycholinguistic subjects: Representations of syntactic state (2019, background): “RNNGs have been found to out-perform LSTMs not only in overall test-set per-plexity (Dyer et al., 2016), but also in modeling long-distance number agreement in Kuncoro et al. (2018) for certain model conﬁgurations; our work extends this comparison to a variety of syntactic state phenomena.”
Hate speech detection: Challenges and solutions (2019, background): “Some deep learning architectures, such as recurrent and transformer neural networks, challenge the bag-of-words assumption by modeling the ordering of the words by processing over a sequence of word embeddings [28].”
Compound Probabilistic Context-Free Grammars for Grammar Induction (2019, background): “17 Kuncoro et al. (2018, 2019) also observe that models that achieve lower perplexity do not necessarily perform better on syntactic evaluation tasks.”