An efficient framework for learning sentence representations

By Lajanugen Logeswaran, Honglak Lee. 2018

579 citations recorded by the source corpus.

In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and its context, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…transfer learning for text classification tasks advocated fine-tuning only the parameters of a small classifier that was fed sentence embeddings produced by a fixed pre-trained model [Subramanian et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Hill et al., 2016; Conneau et al., 2017].”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “…et al., 2006; Collobert & Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) and sentence-level representations (Kiros et al., 2015; Logeswaran & Lee, 2018; Le & Mikolov, 2014), as well as context sensitive features from the NMT model (McCann et al., 2017) and ELMo (Peters et al.,…”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “Also related are sentence representation methods (Kiros et al., 2015; Logeswaran and Lee, 2018) which focus on predicting surrounding contexts from sentence embeddings.”
Transfer Learning in Natural Language Processing (2019, methodology): “…before detailing deep neural network approaches like auto-encoding/skip-thoughts models (Dai and Le, 2015; Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018) and the current trend of language model-based approaches (Dai and Le, 2015; Peters et al., 2018a; Howard and Ruder, 2018;…”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “In this vein, Skip-thoughts—and Quick-thoughts (Logeswaran and Lee, 2018), which has similar performance— which use a next-sentence prediction objective similar to BERT, perform particularly well on STS tasks, indicating a close alignment between the pretraining and target task.”
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, background): “ICT can be considered a generalization of the skip-gram objective (Mikolov et al., 2013), with a coarser granularity, deep architecture, and in-batch negative sampling from Logeswaran and Lee (2018).”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “In-batch Negative Sampling To save the computation and improve sample-efﬁciency, we choose to use in-batch negative sampling (Logeswaran and Lee, 2018) instead of gathering negative samples for each ( q, p ) pair to pretrain the retrieval module.”
Learning Representations by Maximizing Mutual Information Across Views (2019, background): “Self-supervised learning is gaining popularity across the NLP, vision, and robotics communities – e.g., [Devlin et al., 2019, Logeswaran and Lee, 2018, Sermanet et al., 2017, Dwibedi et al., 2018].”
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “In analogy to these human cognitive processes, we suggest to use modern Hopﬁeld networks to amplify co-occurrences and covariance structures of the original data.”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, methodology): “We use the Quick-Thought (QT) loss (Logeswaran and Lee, 2018) to train the em-”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
Patient Knowledge Distillation for BERT Model Compression (2019, background): “…GloVe (Pennington et al., 2014), FastText (Bo-janowski et al., 2017)); ( ii ) sentence-level representation (e.g., Kiros et al. (2015); Conneau et al. (2017); Logeswaran and Lee (2018)); and ( iii ) contextualized word representation (e.g., Cove (McCann et al., 2017), ELMo (Peters et al., 2018)).”
A Theoretical Analysis of Contrastive Unsupervised Representation Learning (2019, methodology): “Notably the sentence representations learned by the quick thoughts (QT) method in Logeswaran & Lee (2018) that we analyze has state-of-the-art results on many text classiﬁcation tasks.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “For sentence embeddings, Logeswaran and Lee (2018) also use contrastive learning with a dual-encoder approach, by forming current sentence and next sentence as (xi, x+i ).”
How to Fine-Tune BERT for Text Classification? (2019, positive): “The generalization of word embeddings, such as sentence embeddings (Kiros et al., 2015; Logeswaran and Lee, 2018) or paragraph embeddings (Le and Mikolov, 2014), are also used as features in downstream models.”
A Mutual Information Maximization Perspective of Language Representation Learning (2019, background): “One particular lower bound that has been shown to work well in practice is InfoNCE (Logeswaran & Lee, 2018; van den Oord et al., 2019), 1 which is based on Noise Contrastive Estimation (NCE; Gutmann & Hyvarinen, 2012).”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, background): “One example is reconstructing the surface form of surrounding sentences given the encoded sentence (Le & Mikolov, 2014; Kiros et al., 2015), and another one is discriminating the next sentence from random candidates (Jernite et al., 2017; Logeswaran & Lee, 2018).”
Evaluation of Text Generation: A Survey (2020, background): “On the other hand, QUICKTHOUGHT (Logeswaran & Lee, 2018) is based on an unsupervised model of universal sentence embeddings trained on consecutive sentences.”