Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks

By R. Thomas McCoy, Robert Frank, Tal Linzen. 2018

87 citations recorded by the source corpus.

Syntactic rules in human language usually refer to the hierarchical structure of sentences. However, the input during language acquisition can often be explained equally well with rules based on linear order. The fact that children consistently ignore these linear explanations to instead settle on hierarchical explanations has been used to argue for an innate hierarchical bias in humans. We revisit this argument by using recurrent neural networks (RNNs), which have no hierarchical bias, to simulate the acquisition of question formation, a hierarchical transformation, in an artificial language modeled after English. Even though this transformation could be explained with a linear rule, we find that some RNN architectures consistently learn the correct hierarchical rule instead. This finding suggests that hierarchical cues within the language are sufficient to induce a preference for hierarchical generalization. This conclusion is strengthened by the finding that adding an additional hierarchical cue, namely syntactic agreement, further improves performance.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “McCoy et al. (2018) reached a similar conclusion for the task of forming a question from a declarative sentence; different architectures that all succeeded on the test set failed on the generalization set in different ways, indicating that the test set alone was not sufﬁcient to determine what the…”
Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, methodology): “Pursuing the idea of poverty of the stimulus training (McCoy et al., 2018), we train diagnostic classifiers only on sentences in which the relevant property (main auxiliary or subject noun) is stateable in either hierarchical or sequential terms, i.”
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance (2019, methodology): “Similarly, McCoy et al. (2018) trained 100 instances for each of 6 types of sequence-to-sequence recurrent neural networks, using a synthetic training set that was ambiguous between two generalizations.”
What do RNN Language Models Learn about Filler–Gap Dependencies? (2018, background): “…and Gulordava et al. (2018) demonstrated that these models are able to successfully learn the number agreement dependency between a subject and its verb, even when there are intervening elements, and McCoy et al. (2018) found that RNNs learn the hierarchical rules of English auxiliary inversion.”
Human few-shot learning of compositional instructions (2019, methodology): “We devised a new set of seq2seq problems that were intentionally ambiguous and compatible with a number of possible generalizations, related to the “poverty of the stimulus” paradigm in experimental linguistics (Wilson, 2006; McCoy, Frank, & Linzen, 2018).”
Learning Helpful Inductive Biases from Self-Supervised Pretraining (2020, background): “McCoy et al. (2018, 2020) train a seq2seq models on generated data ambiguous between a surface and a structural generalization to learn the subject-auxiliary inversion rule in English question formation.”
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (2019, methodology): “Simpler synthetic languages (not based on nat-ural corpora) have been used in a number of recent studies to examine the inductive biases of different neural architectures (Bowman et al., 2015; Lake and Baroni, 2018; McCoy et al., 2018).”
Can neural networks acquire a structural bias from raw linguistic data? (2020, methodology): “There have been several prior efforts to test neural networks for a structural bias in the domain of subject-auxiliary inversion (Lewis & Elman, 2001; Frank & Mathis, 2007; McCoy et al., 2018, 2020).”
The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models (2018, background): “While these models perform very well on standard datasets and also appear to capture some linguistic structure (McCoy et al., 2018; Belinkov et al., 2017; Linzen et al., 2016), they also can be quite brittle, typically breaking on uncharacteristic inputs (Lake and Ba-roni, 2018; Belinkov and Bisk,…”
CNNs found to jump around more skillfully than RNNs: Compositional Generalization in Seq2seq Convolutional Networks (2019, background): “ing abilities (e.g., Kirov and Cotterell, 2018; McCoy et al., 2018; Pater, 2018).”
Cross-Linguistic Syntactic Evaluation of Word Prediction Models (2020, background): “…that language models would be better able to learn hierarchical syntactic generalizations in morphologically complex languages (which provide frequent overt cues to syntactic structure) than in morphologically simpler languages (Gulordava et al., 2018; Lorimor et al., 2008; McCoy et al., 2018).”