A Systematic Assessment of Syntactic Generalization in Neural Language Models

By Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Gotlieb Wilcox, Roger Levy. 2020

275 citations recorded by the source corpus.

While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M-40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “The distinct indirect effect contours for the adjacent and non-adjacent cases may indicate distinct subject-verb agreement mechanisms for short- and long-distance agreement, consistent with similar findings for LSTMs (Lakretz et al., 2019).”
When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022, background): “…Gulordava et al., 2018; Hu et al., 2020b), anaphora agreement and binding constraints (e.g., Marvin and Linzen, 2018; Futrell et al., 2019; Warstadt et al., 2020; Hu et al., 2020a), or ﬁller-gap dependencies (e.g., Wilcox et al., 2018; Chowdhury and Zamparelli, 2018; Da Costa and Chaves, 2020).”
Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023, methodology): “It is evaluated as the model with outperforming bilingual evaluation understudy (BLEU) score among four state-of-art text generation methods (RNN, GAN, GPT, and CTRL) [66] and the best syntactic generalization score among trained (LSTM, ON-LSTM, Transformer, n-gram) and ofthe-shelf (GPT, JRNN, Trans”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “Other work in this space has tried to understand what models know about syntax [24,28,34,49,54,71,83] or the complex interaction between syntactic and semantic categories [38,78,81,82].”
Refining Targeted Syntactic Evaluation of Language Models (2021, background): “Many of these phenomena are rare, so evaluations with templated minimal pairs complement perplexity as a metric for evaluating models’ syntactic generalization (Hu et al., 2020).”
UnNatural Language Inference (2020, methodology): “We verify our ﬁndings with three popular English NLI datasets—SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018b) and ANLI (Nie et al., 2020))—and one Chinese one, OCNLI (Hu et al., 2020a).”
Learning Helpful Inductive Biases from Self-Supervised Pretraining (2020, background): “Finally, Hu et al. (2020) find that adopting architectures that build in linguistic bias, such as RNNGs (Dyer et al., 2016), has a big- ger effect on the acceptability task than increasing training data from 1M to 40M words.”
Frequency Effects on Syntactic Rule Learning in Transformers (2021, background): “There has been substantial prior work on the ability of language models to perform abstract syntactic processing tasks (Hu et al., 2020) (see Linzen and Baroni (2020) for a review).”
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2021, unknown): “Prior work has shown that the predictions of neural LMs have high quality with respect to syntax (Hu et al., 2020; Zhang et al., 2021b); e.g., the annotators in Dou et al. (2022) marked less than 3% of LM-generated tokens as having grammatical errors.”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, unknown): “…et al., 2017; Pallier et al., 2011; Pereira et al., 2018; Rabovsky et al., 2018; Shain et al., 2020; Wehbe et al., 2014; Willems et al., 2016; Gauthier & Ivanova, 2018; Gauthier & Levy, 2019; Hu et al., 2020; Jain & Huth, 2018; S. Wang et al., 2020; Schwartz et al., 2019; Toneva & Wehbe, 2019).”