Structural Supervision Improves Learning of Non-Local Grammatical Dependencies

By Ethan Gotlieb Wilcox, Peng Qian, Richard Futrell, Miguel Ballesteros, Roger Levy. 2019

61 citations recorded by the source corpus.

State-of-the-art LSTM language models trained on large corpora learn sequential contingencies in impressive detail, and have been shown to acquire a number of non-local grammatical dependencies with some success. Here we investigate whether supervision with hierarchical structure enhances learning of a range of grammatical dependencies, a question that has previously been addressed only for subject-verb agreement. Using controlled experimental methods from psycholinguistics, we compare the performance of word-based LSTM models versus Recurrent Neural Network Grammars (RNNGs) (Dyer et al. 2016) which represent hierarchical syntactic structure and use neural control to deploy it in left-to-right processing, on two classes of non-local grammatical dependencies in English—Negative Polarity licensing and Filler-Gap Dependencies—tested in a range of configurations. Using the same training data for both models, we find that the RNNG outperforms the LSTM on both types of grammatical dependencies and even learns many of the Island Constraints on the filler-gap dependency. Structural supervision thus provides data efficiency advantages over purely string-based training of neural language models in acquiring human-like generalizations about non-local grammatical dependencies.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Natural Language Processing

Citations in the corpus

Neural Network Acceptability Judgments (2018, methodology): “It is also increasingly used in computational linguistics (Linzen et al., 2016; Marvin and Linzen, 2018; Futrell et al., 2018; Wilcox et al., 2018, 2019).”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “lead us to models that can imitate humans’ sample efficiency (one example of such direction is neural networks with explicit symbolic structure, which are harder to scale up, but perform well on smaller data sets: Kuncoro et al. 2018; Wilcox et al. 2019).”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “…have studied whether models prefer grammatical completions in subjectverb agreement contexts (Marvin and Linzen, 2018; van Schijndel et al., 2019; Goldberg, 2019; Mueller et al., 2020; Lakretz et al., 2021; Futrell et al., 2019), as well as in filler-gap dependencies (Wilcox et al., 2018, 2019).”
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2019, neutral): “In the two-preﬁx method , used by Wilcox et al. (2019), a pair of sentences differ in their initial string, and the grammaticality difference is only revealed when a shared critical word is included (e.g., The cat eats mice vs.”
Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs (2019, methodology): “…tasks, we also explore two additional BERT-based models—(iii) BERT→MNLI: BERT fine-tuned on the Multi-Genre Natural Language Inference corpus (Williams et al., 2018), motivated both by prior work on pretraining sentence encoders on MNLI (Conneau et al., 2017) as well as work showing…”
Quantity doesn’t buy quality syntax with neural language models (2019, positive): “Overall, this body of results points to the limited data efficiency of standard RNNs and Transformers, and indicates that learning syntax from realistic amounts of data—in particular the amount of data available to humans when they learn language—may require syntactically structured architectures or”
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020, methodology): “…capture many syntactic generalizations , ranging from subject–verb agreement Materials and code can be found at https://github. com/cpllab/syntactic-generalization . to long-distance ﬁller–gap dependencies (Linzen et al., 2016; Marvin and Linzen, 2018; Futrell et al., 2018; Wilcox et al., 2019b).”
Compound Probabilistic Context-Free Grammars for Grammar Induction (2019, background): “As with NLMs, RNNGs make no independence assumptions, and have been shown to outperform NLMs in terms of perplexity and grammaticality judgment when trained on gold trees (Kuncoro et al., 2018; Wilcox et al., 2019).”