Can neural networks acquire a structural bias from raw linguistic data?

By Alex Warstadt, Samuel R. Bowman. 2020

57 citations recorded by the source corpus.

We evaluate whether BERT, a widely used neural network for sentence processing, acquires an inductive bias towards forming structural generalizations through pretraining on raw data. We conduct four experiments testing its preference for structural vs. linear generalizations in different structure-dependent phenomena. We find that BERT makes a structural generalization in 3 out of 4 empirical domains---subject-auxiliary inversion, reflexive binding, and verb tense detection in embedded clauses---but makes a linear generalization when tested on NPI licensing. We argue that these results are the strongest evidence so far from artificial learners supporting the proposition that a structural bias can be acquired from raw data. If this conclusion is correct, it is tentative evidence that some linguistic universals can be acquired by learners without innate biases. However, the precise implications for human language acquisition are unclear, as humans learn language from significantly less data than BERT.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Pre-Trained Models: Past, Present and Future (2021, background): “Similarly, several works also reveal the strong syntax encoding of PTMs (Vilares et al., 2020; Warstadt and Bowman, 2020; Hewitt and Manning, 2019).”
LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning (2021, background): “However, there is another potential advantage of pretraining—it may distill inductive biases into the model that are helpful for training on downstream tasks (Brown et al., 2020; Warstadt & Bowman, 2020).”
Learning Helpful Inductive Biases from Self-Supervised Pretraining (2020, background): “Warstadt and Bowman (2020) conduct related experiments on subject-auxiliary inversion and other English structural rules, and find that BERT likely acquires a structural bias from pretraining.”
Predicting Inductive Biases of Pre-Trained Models (2021, background): “Evidence that similar generalization behavior can result from pre-trained representations has exciting implications for those interested in sample efﬁciency and cognitively-plausible language learning (Warstadt & Bowman, 2020; Linzen, 2020).”