Targeted Syntactic Evaluation of Language Models

By Rebecca Marvin, Tal Linzen. 2018

483 citations recorded by the source corpus.

We present a data set for evaluating the grammaticality of the predictions of a language model. We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence. The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items. We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one. In an experiment using this data set, an LSTM language model performed poorly on many of the constructions. Multi-task training with a syntactic objective (CCG supertagging) improved the LSTM’s accuracy, but a large gap remained between its performance and the accuracy of human participants recruited online. This suggests that there is considerable room for improvement over LSTMs in capturing syntax in a language model.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, neutral): “…from the same distribution as the ﬁne-tuning set, but from what we might term normative evaluation: expert-created controlled data sets that capture our intuitions about how an agent should perform the task (Marelli et al., 2014; Marvin and Linzen, 2018; Warstadt et al., 2019; Ettinger, 2020).”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “…agreement, a phenomenon that requires a distinction to be made between the subject and direct object of a sentence (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018); it is unlikely that our ﬁne-tuning step on MNLI substantially changed the model’s representational capabilities.”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “, 2017; Belinkov and Bisk, 2018) and language modelling (Marvin and Linzen, 2018; Ettinger, 2020).”
How Context Affects Language Models' Factual Predictions (2020, background): “A variety of “probes” have been developed to analyse the syntactic structures, such as syntax trees [Marvin and Linzen, 2018, Hewitt and Manning, 2019, Vig and Belinkov, 2019], negative polarity items [Warstadt and Bowman, 2019, Warstadt et al., 2019], semantic fragments [Richardson et al., 2019],…”
Language Models as Knowledge Bases? (2019, background): “Marvin and Linzen (2018) assess the gram-maticality of pretrained language models.”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “Thus, while excluding the complementizer can make subject-verb agreement slightly more difficult for LMs (Marvin and Linzen, 2018), it does not appear to change the mechanisms through which subject-verb agreement happens in the model.”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “Many papers have found neural models (especially PLMs) to naturally encode syntax (Linzen et al., 2016; Belinkov et al., 2017; Marvin and Linzen, 2018; Belinkov and Glass, 2019; Goldberg, 2019; Hewitt and Manning, 2019).”
BERT Rediscovers the Classical NLP Pipeline (2019, methodology): “Much of this work is behavior-based, designing controlled test sets and analyzing errors in order to reverse-engineer the types of abstractions the model may or may not be representing (e.g. Con-neau et al., 2018; Marvin and Linzen, 2018; Poliak et al., 2018).”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “One line of research examines the outputs of language models on carefully chosen input sentences (Linzen et al., 2016; Khandelwal et al., 2018; Gulordava et al., 2018; Marvin and Linzen, 2018).”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “Other related work uses perplexity scores to test whether language models learn to encode properties such as subject-verb agreement (Linzen et al., 2016; Gulordava et al., 2018; Marvin & Linzen, 2018; Kuncoro et al., 2018).”
Assessing BERT's Syntactic Abilities (2019, methodology): “This differs from Linzen et al. (2016) and Gulordava et al. (2018) by considering the entire sentence (excluding the verb) and not just its preﬁx leading to the verb, and differs from Marvin and Linzen (2018) by conditioning the focus verb on bidirectional context.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “…exist for various phenomena, including ones with “minimal” edits similar to our contrast sets, e.g., in image captioning (Shekhar et al., 2017), machine translation (Sen-nrich, 2017; Burlot and Yvon, 2017; Burlot et al., 2018), and language modeling (Marvin and Linzen, 2018; Warstadt et al., 2019).”
On the Cross-lingual Transferability of Monolingual Representations (2019, neutral): “2012), and the syntactic evaluation (Marvin and Linzen, 2018) datasets.”
Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “, 2018) argues that RNNs acquire grammatical competence in agreement that is more abstract than word collocations, although language model performance that requires sensitivity to the phenomena such as reflexive anaphora, non-local agreement and negative polarity remains low (Marvin and Linzen, 2018”
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2019, neutral): “So far, we have con-Figure sidered only the full-sentence method , advocated for by Marvin and Linzen (2018), which compares LM likelihoods of full sentences.”
Information-Theoretic Probing with Minimum Description Length (2020, background): “…deﬁned templates, either evaluating sensitivity to speciﬁc grammatical errors (Linzen et al., 2016; Gulordava et al., 2018; Tran et al., 2018; Marvin and Linzen, 2018) or understanding what language models know when applying them as knowledge bases or in question answering settings…”
Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs (2019, methodology): “4 Licensing Features We create a controlled set of 136,000 English sentences using an automated sentence generation procedure, inspired in large part by previous work by Ettinger et al. (2016, 2018), Marvin and Linzen (2018), Dasgupta et al. (2018), and Kann et al. (2019).”