BLiMP: The Benchmark of Linguistic Minimal Pairs for English

By Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman. 2019

762 citations recorded by the source corpus.

Abstract We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “In its use of contrastive cases for model evaluation, H ATE C HECK builds on a long history of minimally-contrastive pairs in NLP (e.g. Levesque et al., 2012; Sennrich, 2017; Glockner et al., 2018; Warstadt et al., 2020).”
Dynabench: Rethinking Benchmarking in NLP (2021, neutral): “…competencies (Geiger et al., 2018; Naik et al., 2018; Glockner et al., 2018; White et al., 2018; Warstadt et al., 2019; Gauthier et al., 2020; Hossain et al., 2020; Jeretic et al., 2020; Lewis et al., 2020; Saha et al., 2020; Schuster et al., 2020; Sugawara et al., 2020; Warstadt et al., 2020).”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “, 2020) which is a speech rendering of the BLIMP dataset (Warstadt et al., 2020).”
Evaluating NLP Models via Contrast Sets (2020, background): “Challenge sets exist for various phenomena, including ones with “minimal” edits similar to our contrast sets, e.g., in image captioning (Shekhar et al., 2017), machine translation (Sennrich, 2017), and language modeling (Marvin and Linzen, 2018; Warstadt et al., 2019).”
Is neural language acquisition similar to natural? A chronological probing study (2022, background): “In 1984 , PC Magazine decided to do a review of printers . a text is coherent (Warstadt et al., 2020).”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, methodology): “…to parametric probes, stimulibased probing (Linzen et al., 2016; Marvin and Linzen, 2018; Gulordava et al., 2018; Goldberg, 2019; Wolf, 2019; Warstadt et al., 2019a, 2020a,b; Ettinger, 2020) has been used to show that even without a learned probing function, BERT can predict syntactic…”
On learning and representing social meaning in NLP: a sociolinguistic perspective (2021, background): “Although there are many datasets to evaluate NLP models on various linguistic phenomena (Warstadt et al., 2019, 2020; Wang et al., 2018), such datasets are missing for social meaning.”
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models (2021, background): “Statistical models trained on large amounts of data using the language modelling objective (predicting words in context) have shown to pick up an intriguing amount of implicit knowledge about other tasks, for example syntactic knowledge (Warstadt et al., 2020; Hu et al., 2020) or world knowledge (Tr”
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020, background): “Although this paper, together with Warstadt et al. (2020), report what is to our knowledge the largestscale targeted syntactic evaluations to date, we emphasize that they are only first steps toward a comprehensive understanding of the syntactic capabilities of contemporary language models.”
PROST: Physical Reasoning about Objects through Space and Time (2021, background): “Each object in PROST is selected to be single-token compatible for a wide range of vocabularies to enable easy probing of MLMs.”