A large annotated corpus for learning natural language inference

By Samuel R. Bowman, Gabor Angeli, Christopher Potts, Christopher D. Manning. 2015

4,739 citations recorded by the source corpus.

Understanding entailment and contradiction is fundamental to understanding natural language, and inference about entailment and contradiction is a valuable testing ground for the development of semantic representations. However, machine learning research in this area has been dramatically limited by the lack of large-scale resources. To address this, we introduce the Stanford Natural Language Inference corpus, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning. At 570K pairs, it is two orders of magnitude larger than all other resources of its type. This increase in scale allows lexicalized classifiers to outperform some sophisticated existing entailment models, and it allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “These tasks include sentence-level tasks such as natural language inference (Bowman et al., 2015; Williams et al., 2018) and paraphrasing (Dolan and Brockett, 2005), which aim to predict the relationships between sentences by analyzing them holistically, as well as token-level tasks such as named…”
Long Short-Term Memory-Networks for Machine Reading (2016, methodology): “In this task we used the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015), which contains premise-hypothesis pairs and target labels indicating their relation (entailment, contradiction or neutral).”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “We use pretrained versions of these models (with ELMo embeddings) on SNLI to obtain 3-way entailment, neutral, and contradiction probabilities for each example.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “In the case of SNLI (Bowman et al., 2015), the bestperforming sentence encoding model on the leaderboard as of April 2018 achieves 86.”
Deep Contextualized Word Representations (2018, background): “The augmented con-Model Feature based (Bowman et al., 2015) 78.2 DIIN (Gong et al., 2018) 88.0 BCN+Char+CoVe (McCann et al., 2017) 88.1 ESIM (Chen et al., 2017) 88.0 ESIM+TreeLSTM (Chen et al., 2017) 88.6 ESIM+ELMo 88.7 ± 0.17 DIIN ensemble (Gong et al., 2018) 88.9 ESIM+ELMo ensemble 89.3 text…”
Learned in Translation: Contextualized Word Vectors (2017, methodology): “For entailment, we use the Stanford Natural Language Inference Corpus (SNLI) [Bow-man et al., 2015], which has 550,152 training, 10,000 validation, and 10,000 testing examples.”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “HCTI (Shao, 2017) Third place overall is obtained by HCTI with a model similar to a convolutional Deep Structured Semantic Model (CDSSM) (Chen et al., 2015; Huang et al., 2013).”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “…that an encoder based on a bi-directional LSTM architecture with max pooling, trained on the Stanford Natural Language Inference (SNLI) dataset ( Bowman et al., 2015), yields state-of-the-art sentence embeddings compared to all existing alternative unsupervised approaches like SkipThought or…”
A Decomposable Attention Model for Natural Language Inference (2016, methodology): “We evaluate our approach on the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015).”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “The NSP objective was designed to improve performance on downstream tasks, such as Natural Language Inference (Bowman et al., 2015), which require reasoning about the relationships between pairs of sentences.”
How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “Consider, for example, crowdsourced natural language inference (NLI) data sets, in which workers are asked to generate a sentence that contradicts the prompt shown to them (Bowman et al., 2015).”
Adversarial Training for Large Neural Language Models (2020, methodology): “On ANLI, we follow the experimental setting of Nie et al. (2019) to enable a head-to-head comparison, which combines four datasets (ANLI, MNLI, SNLI and FEVER (Thorne et al., 2018)) for ﬁne-tuning.”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “We (Lang, 1995), the English source side of English-German WMT16 and English-German Multi30K (Elliott et al., 2016), and concatenations of the premise and hypothesis for RTE (Dagan et al., 2005) and SNLI (Bowman et al., 2015).”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “Progress in AI has been driven by, among other things, the development of challenging large-scale benchmarks like ImageNet (Russakovsky et al., 2015) in computer vision, and SNLI (Bowman et al., 2015), SQuAD (Rajpurkar et al., 2016), and others in natural language processing (NLP).”
Probing Neural Network Comprehension of Natural Language Arguments (2019, methodology): “…work on ARCT (Choi and Lee, 2018; Zhao et al., 2018; Niven and Kao, 2018) involved transfer learning from Natural Language Inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2017), and utilized effective NLI models such as ESIM (Chen et al., 2016) and InferSent (Conneau et al., 2017).”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “…accuracy in multiple language understanding tasks such as sentiment classiﬁcation (Socher et al., 2013), natural language inference (Bowman et al., 2015), named entity recognition (Tjong Kim Sang & De Meulder, 2003) and SQuAD question answering (Rajpurkar et al., 2016), which…”