A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

By Adina Williams, Nikita Nangia, Samuel R. Bowman. 2017

5,138 citations recorded by the source corpus.

This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “MNLI Multi-Genre Natural Language Inference is a large-scale, crowdsourced entailment classification task (Williams et al., 2018).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, neutral): “The MultiNLI dataset (Williams et al., 2018) contains crowd sourced textual entailment annotations across ﬁve diverse domains for training and an additional ﬁve domains for testing.”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “(Bowman et al., 2015; Williams et al., 2018).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “MNLI The Multi-Genre Natural Language Inference Corpus (Williams et al., 2018) is a crowdsourced collection of sentence pairs with textual entailment annotations.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “• Natural language inference (MNLI [Williams et al., 2017], QNLI [Rajpurkar et al.”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “, 2016), MultiGenre NLI (MNLI) (Williams et al., 2018), Question NLI (QNLI) (Rajpurkar et al.”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, methodology): “In the syntactic domain, McCoy et al. (2019b) showed that BERT fine-tuned on the crowdsourced MultiNLI data set (Williams et al., 2018) achieves high accuracy on the MultiNLI test set, but shows very little sensitivity to word order when tested on constructed examples that require an analysis of…”
Adversarial Training for Large Neural Language Models (2020, methodology): “On ANLI, we follow the experimental setting of Nie et al. (2019) to enable a head-to-head comparison, which combines four datasets (ANLI, MNLI, SNLI and FEVER (Thorne et al., 2018)) for ﬁne-tuning.”
TTTTTackling WinoGrande Schemas (2020, positive): “ are formulated as text-to-text problems, where the inputs are cast into natural language templates that contain the task descriptors. Concretely, Raffel et al. provide the following example for MNLI [7], where the goal is to predict whether a premise implies (“entailment”) or contradicts(“contradic”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We also evaluate the model by finetuning on MNLI (Williams et al., 2018) and SST-2 (Socher et al.”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “Only for MNLI, we find language modeling to consistently help for all training set sizes considered.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “, 2015) and MNLI (Williams et al., 2017), and selected the best-performing”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “…2005), Semantic Textual Similarity Benchmark (STS; Cer et al., 2017), Quora Question Pairs (QQP; Iyer et al., 2017), Multi-Genre NLI (MNLI; Williams et al., 2018), Question NLI (QNLI; Rajpurkar et al., 2016), Recognizing Textual Entailment (RTE; Dagan et al., 2005; Bar-Haim et al., 2006;…”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, background): “Multi-Genre Natural Language Inference is a large-scale, crowd-sourced entailment classification task (Williams et al., 2018).”
Natural Questions: A Benchmark for Question Answering Research (2019, background): “One view of the long answer selection task is that it is more closely related to natural language inference (NLI) (Bowman et al., 2015; Williams et al., 2018) than short answer extraction.”
Probing Neural Network Comprehension of Natural Language Arguments (2019, methodology): “…work on ARCT (Choi and Lee, 2018; Zhao et al., 2018; Niven and Kao, 2018) involved transfer learning from Natural Language Inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2017), and utilized effective NLI models such as ESIM (Chen et al., 2016) and InferSent (Conneau et al., 2017).”