Quantity doesn’t buy quality syntax with neural language models

By Marten van Schijndel, Aaron Mueller, Tal Linzen. 2019

76 citations recorded by the source corpus.

Recurrent neural networks can learn to predict upcoming words remarkably well on average; in syntactically complex contexts, however, they often assign unexpectedly high probabilities to ungrammatical words. We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained. We find that gains from increasing network size are minimal beyond a certain point. Likewise, expanding the training corpus yields diminishing returns; we estimate that the training corpus would need to be unrealistically large for the models to match human performance. A comparison to GPT and BERT, Transformer-based models trained on billions of words, reveals that these models perform even more poorly than our LSTMs in some constructions. Our results make the case for more data efficient architectures.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “In response to studies comparing training corpus sizes between deep learning models and humans (e.g., van Schijndel et al. 2019), it is sometimes pointed out that humans do not learn language from text alone—we also observe the world and interact with it.”
Experience Grounds Language (2020, background): “As pre-training schemes seem to be reaching the point of diminishing returns from data, even for some syntactic phenomena (van Schijndel et al., 2019), we posit that other forms of supervision, such as multimodal perception, will be necessary to learn the remaining aspects of meaning in context.”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “Because larger models tend to exhibit correct subject-verb agreement more often than smaller ones (Hu et al., 2020; van Schijndel et al., 2019), we hypothesize that larger models will generally have larger TEs for the same structure (i.e., we predict that higher accuracy is indicative of larger…”
Pre-Trained Models: Past, Present and Future (2021, background): “To analysis the gain of pretraining on estimating the probabilities of ungrammatical words, Schijndel (Schijndel et al., 2019) show that expanding the training corpus yields diminishing returns and the training corpus would need to be unrealistically large to make PTMs match human performance.”
When Do You Need Billions of Words of Pretraining Data? (2020, background): “…of rapid improvements in linguistic knowledge up to about 10M words of pretraining data, after which improvements slow down for most phenomena. van Schijndel et al. (2019) ﬁnd large improvements in knowledge of subject-verb agreement and reﬂexive binding up to 10M words, and few improvements…”
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020, result): “Transformers have been quickly adopted in practical NLP systems (Storks et al., 2019), but the literature has reported mixed results on the beneﬁt of Transformers over LSTMs in terms of linguistic generalization (Hupkes et al., 2020; van Schijndel et al., 2019).”
Learning Helpful Inductive Biases from Self-Supervised Pretraining (2020, background): “…acceptability judgment task on minimal pairs (i.e. not during fine-tuning), and do not attempt to separate feature learning from feature preferences. van Schijndel et al. (2019) find the greatest increase in sensitivity to acceptability contrasts occurs between training on 2M and 10M words.”
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020, result): “Second, we find a larger effect of model inductive bias than training data size on SG score, a result that accords with van Schijndel et al. (2019).”
UnNatural Language Inference (2020, background): “This is especially true for syntactic phenomena (Marvin and Linzen, 2018; Hu et al., 2020b; Gauthier et al., 2020; McCoy et al., 2020; Warstadt et al., 2020), where transformers are, for some phenomena and settings, worse than RNNs (van Schijndel et al., 2019).”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, background): “Results marked with † are taken from Sennrich and Zhang (2019), those marked with ] from Kudo (2018).”