Towards Universal Paraphrastic Sentence Embeddings

By John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu. 2015

573 citations recorded by the source corpus.

We consider the problem of learning general-purpose, paraphrastic sentence embeddings based on supervision from the Paraphrase Database (Ganitkevitch et al., 2013). We compare six compositional architectures, evaluating them on annotated textual similarity datasets drawn both from the same distribution as the training data and from a wide range of other domains. We find that the most complex architectures, such as long short-term memory (LSTM) recurrent neural networks, perform best on the in-domain data. However, in out-of-domain scenarios, simple architectures such as word averaging vastly outperform LSTMs. Our simplest averaging model is even competitive with systems tuned for the particular tasks while also being extremely efficient and easy to use. In order to better understand how these architectures compare, we conduct further experiments on three supervised NLP tasks: sentence similarity, entailment, and sentiment classification. We again find that the word averaging models perform well for sentence similarity and entailment, outperforming LSTMs. However, on sentiment classification, we find that the LSTM performs very strongly-even recording new state-of-the-art performance on the Stanford Sentiment Treebank. We then demonstrate how to combine our pretrained sentence embeddings with these supervised tasks, using them both as a prior and as a black box feature extractor. This leads to performance rivaling the state of the art on the SICK similarity and entailment tasks. We release all of our resources to the research community with the hope that they can serve as the new baseline for further work on universal sentence embeddings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, background): “Wieting et al. (2015) and Wieting & Gimpel (2017) use paraphrase data to learn an encoder that maps synonymous phrases to similar embeddings using a margin loss.”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “Ensembled components include: alignment similarity; TakeLab STS (Šarić et al., 2012b); string similarity measures such as matching n-grams, summarization and MT metrics (BLEU, WER, PER, ROUGE); a RNN and recurrent convolutional neural networks (RCNN) over word alignments; and a BiLSTM that is stat”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “Recent work on generating sentence embeddings range from models that compose word embeddings (Le and Mikolov, 2014; Arora et al., 2017; Wieting et al., 2016) to more complex neural network architectures.”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “These embeddings are encoded with one of three models: word averages (Wieting et al., 2016),”
Transfer Learning in Natural Language Processing (2019, methodology): “We will then describe supervised approaches which make use of large annotated datasets (Zoph et al., 2016; Yang et al., 2017; Wieting et al., 2016; Conneau et al., 2017; McCann et al., 2017) before turning to distant supervision approaches which use heuristics to automatically label datasets (Mintz ”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “Results 1 correspond to PP-Proj (Wieting et al., 2015) and 2 from Tree-LSTM (Tai et al.”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “Some recent work has addressed this by learning general-purpose sentence representations (Kiros et al., 2015; Wieting et al., 2015; Hill et al., 2016; Conneau et al., 2017; McCann et al., 2017; Jernite et al., 2017; Nie et al., 2017; Pagliardini et al., 2017).”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “Sentence Representation Learning Many approaches have been proposed to extend word embeddings to sentence or paragraph representations (Le and Mikolov, 2014; Wieting et al., 2016; Arora et al., 2017).”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “Given a training minibatch { ( S i , T i ) } Bi =1 , we construct 2 ∗ B − 1 negative pairs { ( S i , (cid:103) T i,j ) } 2 B − 1 j =1 for every positive pair ( S i , T i ) using three different methods, inspired by Wieting et al. (2016).”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “We embed each sentence using a universal para-phrastic sentence encoder (Wieting et al., 2016; Arora et al., 2017; Ethayarajh, 2018a), a model which was trained to output similar representations for sentences of similar meaning.”
Probing for semantic evidence of composition by means of simple classification tasks (2016, background): “, 2014), 2) Paraphrastic word averaging embeddings (Paragram) trained with a compositional objective (Wieting et al., 2015).”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, neutral): “tasks (Wieting et al., 2015).”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “To our knowledge no existing work has explored pre-training and extrapolation in the context of NLG. Noisy pre-training has been proposed before for other tasks such as paraphrasing (Wieting et al., 2016; Tomar et al., 2017) but generally not with synthetic data.”
Aligning AI With Shared Human Values (2020, methodology): “Our word averaging model is based on fasttext and 100-dimensional GloVe vectors (Wieting et al., 2016; Joulin et al., 2017; Pennington et al., 2014).”
Anchors: High-Precision Model-Agnostic Explanations (2018, methodology): “Text Classification: We have already alluded to Figure 1, where we trained an LSTM model with paraphrastic sentence embeddings (Wieting et al. 2015) to predict the sentiment of reviews.”
Model-Agnostic Interpretability of Machine Learning (2016, background): “The classiﬁers vary wildly in complexity and underlying representation - one is a logistic regression trained on unigrams, while the other an LSTM neural network trained on sentence embeddings (Wieting et al., 2015).”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, background): “Among them, the simplest strategy is to compute the element-wise average over word vectors for a given sequence (Wieting et al., 2015; Adi et al., 2016):”