Learning Distributed Representations of Sentences from Unlabelled Data

By Felix Hill, Kyunghyun Cho, Anna Korhonen. 2016

591 citations recorded by the source corpus.

Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, methodology): “, 2011; 2013), Sequential Denoising Autoencoders (SDAE), FastSent (Hill et al., 2016), etc.”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “Other approaches train “thought vectors” representing sentences through unsupervised (Kiros et al., 2015; Hill et al., 2016) or supervised (Conneau et al., 2017) learning.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “), including approaches leveraging unlabeled data (Hill et al., 2016; Peters et al., 2018), labeled data (Conneau and Kiela, 2018; McCann et al.”
An efficient framework for learning sentence representations (2018, methodology): “There have been variations on the decoder such as autoencoder models which predict the input sentence instead of neighboring sentences (Hill et al., 2016) and predicting properties of a window of words in the input sentence (Le & Mikolov, 2014).”
Learned in Translation: Contextualized Word Vectors (2017, background): “Hill et al. [2016] study the effect of transferring from a variety of source domains to the semantic similarity tasks in Agirre et al. [2014].”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “† indicates methods that we trained, other numbers have been extracted from (Hill et al., 2016), except for Naive Bayes - SVM and AdaSent results coming from (Zhao et al.”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “Alternately, the Fast-Sent model of Hill et al. (2016) is designed to work dramatically more quickly than systems like Skip Thought, but in service of this goal the standard sentence encoder RNN is replaced with a low-capacity CBOW model.”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “LM embeddings are related to a class of meth-ods (e.g., Le and Mikolov, 2014; Kiros et al., 2015; Hill et al., 2016) for learning sentence and document encoders from unlabeled data, which can be used for text classiﬁcation and textual entailment among other tasks.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model [Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017].”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “Skipthought (Kiros et al., 2015) and FastSent (Hill et al., 2016) sentence embeddings are learned by using an encoding of a sentence to predict words in neighboring sentences.”
Transfer Learning in Natural Language Processing (2019, neutral): “…Blei et al., 2003) before detailing deep neural network approaches like auto-encoding/skip-thoughts models (Dai and Le, 2015; Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018) and the current trend of language model-based approaches (Dai and Le, 2015; Peters et al., 2018a;…”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “Several works focused on learning reusable sentence-to-vector encodings, where the pretrained encoder produces a ﬁxed-size representation for each input sentence (Dai and Le, 2015; Kiros et al., 2015; Hill et al., 2016; Con-neau et al., 2017).”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “On the one hand, people have measured performance on intrinsic evaluations, e.g. of human judgments of word or sentence similarity ratings (Agirre et al., 2012; Hill et al., 2016b) or of word associations (Vuli´c et al., 2017).”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “Some recent work has addressed this by learning general-purpose sentence representations (Kiros et al., 2015; Wieting et al., 2015; Hill et al., 2016; Conneau et al., 2017; McCann et al., 2017; Jernite et al., 2017; Nie et al., 2017; Pagliardini et al., 2017).”
e-SNLI: Natural Language Inference with Natural Language Explanations (2018, background): “Traditional approaches make use of very large unsupervised datasets [17, 14], taking weeks to train.”
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (2016, methodology): “This is the approach taken by Li et al. (2015), Hill et al. (2016) and Kiros et al. (2015).”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, background): “Hill et al. (2016) showed, that the task on which sentence embeddings are trained signiﬁcantly impacts their quality.”