Skip-Thought Vectors

By Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, Sanja Fidler. 2015

2,486 citations recorded by the source corpus.

We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “These approaches have been generalized to coarser granularities, such as sentence embeddings (Kiros et al., 2015; Logeswaran and Lee, 2018) or paragraph embeddings (Le and Mikolov, 2014).”
A Structured Self-attentive Sentence Embedding (2017, methodology): “This includes SkipThought vectors (Kiros et al., 2015), ParagraphVector (Le & Mikolov, 2014), recursive auto-encoders (Socher et al., 2011; 2013), Sequential Denoising Autoencoders (SDAE), FastSent (Hill et al., 2016), etc.”
Layer Normalization (2016, methodology): “Given contiguous text, a sentence is [Kiros et al., 2015].”
Multi-task Sequence to Sequence Learning (2015, methodology): “We also explore two unsupervised learning objectives, sequence autoencoders (Dai & Le, 2015) and skip-thought vectors (Kiros et al., 2015), and reveal their interesting properties in the MTL setting: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Other approaches train “thought vectors” representing sentences through unsupervised (Kiros et al., 2015; Hill et al., 2016) or supervised (Conneau et al., 2017) learning.”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “Pretrained sentence encoders We consider two types of pretrained RNN sentence encoders, SkipThoughts (Kiros et al., 2015) and InferSent (Conneau et al., 2017).”
Neural Network Acceptability Judgments (2018, background): “In similar low-resource settings, transfer learning with sentence embeddings has proven to be effective (Kiros et al., 2015; Conneau et al., 2017).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “Skip-Thought (Kiros et al., 2015): a sequence-to-sequence(s) model trained to generate the previous and next sentences given the middle sentence.”
An efficient framework for learning sentence representations (2018, methodology): “Kiros et al. (2015) proposed the skip-thought vectors model, which consists of an encoder RNN that produces a vector representation of the source sentence and a decoder RNN that sequentially predicts the words of adjacent sentences.”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “(Kiros et al., 2015) propose an objective function that adapts the skip-gram model for words (Mikolov et al.”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “However, recent work has shown some success in using unsupervised pretraining with unlabeled data to both improve the performance of these methods and extend them to lower-resource settings (Dai and Le, 2015; Kiros et al., 2015; Ba-jgar et al., 2016).”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “LM embeddings are related to a class of meth-ods (e.g., Le and Mikolov, 2014; Kiros et al., 2015; Hill et al., 2016) for learning sentence and document encoders from unlabeled data, which can be used for text classiﬁcation and textual entailment among other tasks.”
context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, background): “Finally, there is considerable work on using recurrent neural networks to represent word sequences, such as phrases or sentences (Socher et al., 2011; Kiros et al., 2015).”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, methodology): “We follow the procedure (and code) of Kiros et al. (2015): a logistic regression classiﬁer is trained on top of sentence representations, with 10-fold cross-validation used when a train-test split is not pre-deﬁned.”
Semi-supervised Sequence Learning (2015, methodology): “Our semi-supervised learning approach is related to Skip-Thought vectors [13], with two differences.”
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “Figure 2: Sentence neural embedding [16].”
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “A lot of promising recent work originates from the Skip-gram model [14], which inspired the skip-thought vectors [15] and several techniques for unsupervised feature learning of images [16].”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model [Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017].”