Assessing BERT's Syntactic Abilities

By Yoav Goldberg. 2019

526 citations recorded by the source corpus.

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “In this way, our dataset can augment prior work on studying to what extent models such as BERT understand syntax (Goldberg 2019).”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “First, the same pretrained BERT model we used (namely, the bert-base-uncased model) was found by Goldberg (2019) to achieve strong results on syntactic evaluations, suggesting that this pretrained model does have access to relevant syntactic information.”
Language Models as Knowledge Bases? (2019, background): “The parameters of these models appear to store vast amounts of linguistic knowledge (Peters et al., 2018b; Goldberg, 2019; Tenney et al., 2019) useful for downstream tasks.”
oLMpics-On What Language Model Pre-training Captures (2019, background): “Thus, ideally, one should test LMs using the pretrained weights without fine-tuning (Linzen et al., 2016a; Goldberg, 2019).”
How Can We Know What Language Models Know? (2019, methodology): “For contextualized representations in particular, a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a,b; Jawahar e”
Compositionality Decomposed: How do Neural Networks Generalise? (2019, background): “Lake & Baroni, 2018); Some instead consider models’ ability to process hierarchical structures (Hupkes, Veldhoen, & Zuidema, 2018; Linzen, Dupoux, & Goldberg, 2016); Yet others consider if models can segment the input into reusable parts (Johnson et al., 2017). This variety of tests for compositiona”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “, 2019)—as psycholinguistic subjects to be studied behaviorally (Linzen et al., 2016; Gulordava et al., 2018; Goldberg, 2019).”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “, 2020) perform on linguistic tasks (Goldberg, 2019; Hewitt and Manning, 2019; Tenney et al., 2019; Elazar et al., 2020), commonsense (Forbes et al.”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “15These works cover a wide variety of topics, including grammatical generalization (Goldberg, 2019; Warstadt et al., 2019), syntax (Tenney et al., 2019b; Lin et al., 2019; Reif et al., 2019; Hewitt and Manning, 2019; Liu et al., 2019a), world knowledge (Petroni et al., 2019; Jiang et al., 2020),…”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, neutral): “Other work has drawn a similar conclusions from examining BERT’s predictions on agreement tasks (Goldberg, 2019) or internal vector representations (He-witt and Manning, 2019; Liu et al., 2019).”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “phenomena such as English subject-verb agreement (Goldberg, 2019; Jawahar et al., 2019), constituent types, dependency labels, NER, and (core) semantic role types (again, all in English) (Tenney et al.”
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (2019, background): “Early work probing BERT’s linguistic capabilities has found it surprisingly robust (Goldberg, 2019).”
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2019, positive): “Relevant to our case study here, using several of these tests Goldberg (2019) shows the BERT model to perform impressively on such syntactic diagnostics.”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “In particular, the opposite was observed for subject-verb agreement (Goldberg, 2019) and sentence subject detection Lin et al. (2019).”
Revealing the Dark Secrets of BERT (2019, background): “Goldberg (2019) demonstrated that BERT consistently assigns higher scores to the correct verb forms as opposed to the incorrect one in a masked language modeling task, suggesting some ability to model subject-verb agreement.”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, methodology): “Trained with massive corpora for language modeling, BERT has strong syntactic ability (Goldberg, 2019) and captures generic language features.”
On the Cross-lingual Transferability of Monolingual Representations (2019, methodology): “Following Goldberg (2019), we feed each sentence into the model masking the word in which it differs from its pair, and pick the one to which the masked language model assigns the highest probability mass. Similar to Goldberg”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “At a high level, mechanisms for interpreting BERT can be categorized into three main categories: interpreting the learned embeddings [1, 16, 23, 49, 86], BERT’s learned knowledge of syntax [27, 30, 32, 45, 47, 76], and BERT’s learned knowledge of semantics [24, 76].”
Pre-Trained Models: Past, Present and Future (2021, background): “tokens or estimate the probabilities of different sentences, it is intuitive to construct analysis tasks based on generation (Goldberg, 2019).”
Visualizing and Understanding the Effectiveness of BERT (2019, background): “Recent work (Tenney et al., 2019b; Liu et al., 2019a; Goldberg, 2019; Tenney et al., 2019a) has shown that the pre-trained models can encode syntactic and semantic information of language.”