Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

By Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, Antoine Bordes. 2017

2,227 citations recorded by the source corpus.

Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features. Efforts to obtain embeddings for larger chunks of text, such as sentences, have however not been so successful. Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted. In this paper, we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Much like how computer vision uses ImageNet to obtain features, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks. Our encoder is publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “While the advantage of unsupervised pre-training is that there is a nearly unlimited amount of data available, there has also been work showing effective transfer from supervised tasks with large datasets, such as natural language inference (Conneau et al., 2017) and machine translation (McCann et a”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “Other approaches train “thought vectors” representing sentences through unsupervised (Kiros et al., 2015; Hill et al., 2016) or supervised (Conneau et al., 2017) learning.”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “While the simplest such model, Dual-BoW, obtains only 35.1% accuracy, combining In-ferSent sentence representations gives 40.5% accuracy (InferSent-Bilinear).”
Neural Network Acceptability Judgments (2018, methodology): “Pooling classifier Our real/fake classifiers and acceptability classifiers use an architecture we refer to as a pooling classifier, which is based on Conneau et al. (2017). As illustrated in Figure 2, the pooling classifier consists of two parts: (i) a sentence encoder which reduces variable-length ”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “Similarly, with the exception of InferSent, the sentence representation models are outperformed on SST by our BiLSTM and its non-CoVe variants.”
An efficient framework for learning sentence representations (2018, methodology): “Drawing inspiration from this model, Gan et al. (2016) explore the use of convolutional neural network (CNN) encoders.”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “…we believe that language model ﬁne-tuning will be particularly useful in the following settings compared to existing transfer learning approaches (Conneau et al., 2017; Mc-Cann et al., 2017; Peters et al., 2018): a) NLP for non-English languages, where training data for supervised pretraining…”
Learned in Translation: Contextualized Word Vectors (2017, background): “Attention mechanisms have also been successfully applied to NLP tasks like entailment [Conneau et al., 2017], summarization [Nallapati et al.”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “…shared task data sets have been used extensively for research on sentence level similarity and semantic representations (i.a., Arora et al. (2017); Conneau et al. (2017); Mu et al. (2017); Pagliardini et al. (2017); Wieting and Gimpel (2017); He and Lin (2016); Hill et al. (2016); Kenter et al.…”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, methodology): “The MultiNLI corpus was first released in draft form in the first half of 2017, and in the time since its initial release, work by others (Conneau et al., 2017) has shown that NLI can also be an effective source task for pre-training and transfer learning in the context of sentence-to-vector models,”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model [Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017].”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “…from the original papers: for SNLI-Hard from Gururangan et al.’s implementation of the hierarchical tensor-based Densely Inter-active Inference Network (Gong et al., 2018, DIIN) on MNLI, and for the NLI stress tests, Naik et al.’s implementation of InferSent (Conneau et al., 2017) trained on SNLI.”
Probing Neural Network Comprehension of Natural Language Arguments (2019, methodology): “…work on ARCT (Choi and Lee, 2018; Zhao et al., 2018; Niven and Kao, 2018) involved transfer learning from Natural Language Inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2017), and utilized effective NLI models such as ESIM (Chen et al., 2016) and InferSent (Conneau et al., 2017).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “Given the productive use of MultiNLI in pretraining and intermediate fine-tuning of pretrained language models (Conneau et al., 2017; Phang et al., 2018), for CB and RTE, we use MultiNLI as a transfer task.”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “In a similar vein, McCoy et al. (2019) found little evidence of compositional structure in the InferSent model (Conneau et al., 2017), which was trained on SNLI, even though other RNNs did learn clear compositional structure when trained on tasks that required such structure.”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “The use of MNLI and SNLI is motivated by prior work on using natual language inference tasks to pretrain sentence encoders (Conneau et al., 2017; Subramanian et al., 2018; Bowman et al., 2019).”
QuAC: Question Answering in Context (2018, methodology): “Pretrained InferSent To test the importance of lexical matching in our dataset, we output the sentence in s whose pretrained InferSent representation (Conneau et al., 2017) has the highest cosine similarity to that of the question.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “We change the Question Match model to a classic BiLSTM Max-Out (Conneau et al., 2017) for textual entailment, by replacing the question q and a choice c i with the premise p and the hypothesis h , resp., and perform binary classiﬁ-cation on the entailment labels (Entail, Neural).”
Annotation Artifacts in Natural Language Inference Data (2018, methodology): “In parallel to this work, Dasgupta et al. (2018) found that the InferSent model (Conneau et al., 2017) relies on word-level heuristics to achieve state-of-the-art performance on SNLI.”
Transfer Learning in Natural Language Processing (2019, methodology): “We will then describe supervised approaches which make use of large annotated datasets (Zoph et al., 2016; Yang et al., 2017; Wi-eting et al., 2016; Conneau et al., 2017; McCann et al., 2017) before turning to distant supervision approaches which use heuristics to automatically label datasets…”