Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features

By Matteo Pagliardini, Prakhar Gupta, Martin Jaggi. 2017

737 citations recorded by the source corpus.

The recent tremendous success of unsupervised word embeddings in a multitude of applications raises the obvious question if similar methods could be derived to improve embeddings (i.e. semantic representations) of word sequences as well. We present a simple but efficient unsupervised objective to train distributed representations of sentences. Our method outperforms the state-of-the-art unsupervised models on most benchmark tasks, highlighting the robustness of the produced general-purpose sentence embeddings.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “…for research on sentence level similarity and semantic representations (i.a., Arora et al. (2017); Conneau et al. (2017); Mu et al. (2017); Pagliardini et al. (2017); Wieting and Gimpel (2017); He and Lin (2016); Hill et al. (2016); Kenter et al. (2016); Lau and Baldwin (2016); Wieting et…”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “Some recent work has addressed this by learning general-purpose sentence representations (Kiros et al., 2015; Wieting et al., 2015; Hill et al., 2016; Conneau et al., 2017; McCann et al., 2017; Jernite et al., 2017; Nie et al., 2017; Pagliardini et al., 2017).”
The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “(2016b), we also compared a vector-based sentence-similarity using sent2vec (Pagliardini et al., 2017) to compare sentences (VecSim).”
Learning Semantic Textual Similarity from Conversations (2018, neutral): “(2) For model comparisons, we include the state-of-the-art neural STS model CNN (HCTI) (Shao, InferSent (Conneau et al., 2017), Sent2Vec (Pagliardini et al., 2017), SIF (Arora et al., 2017), PV-DBOW (Lau and Baldwin, 2016), C-PHRASE (Kruszewski et al., 2015), ECNU (Tian et al., 2017) and BIT (Wu et ”
Continual Learning for Sentence Representations Using Conceptors (2019, background): “, (Yin and Schütze, 2015; Arora et al., 2017; Cer et al., 2018; Pagliardini et al., 2018)), most, if not all, of them are trained on a priori fixed corpora.”
A Theoretical Analysis of Contrastive Unsupervised Representation Learning (2019, methodology): “Subse-quently, similar ideas have been used by Logeswaran & Lee (2018) and Pagliardini et al. (2018) for sentences representations and by Wang & Gupta (2015) for images.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “Pagliardini et al. (2018) show that simply augmenting the idea of word2vec (Mikolov et al., 2013) with n-gram embeddings leads to strong results.”
Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings (2017, neutral): “…have pursued this goal (Socher et al., 2011; Le and Mikolov, 2014; Pham et al., 2015; Kiros et al., 2015; Hill et al., 2016; Arora et al., 2017; Pagliardini et al., 2017), though usually with the intent to extract useful features for supervised sentence tasks rather than to capture semantic…”
NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language (2019, methodology): “For MEDHOP we use bigram SENT2VEC embeddings trained on a large biomedical corpus 4, and for WIKIHOP the wikiunigrams model5 of SENT2VEC.”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, methodology): “Most recently, Wieting et al. (2016); Arora et al. (2017); Pagliardini et al. (2017) proposed methods in which sentences are represented as a weighted average of fixed (pre-trained) word vectors.”
Learning to Annotate: Modularizing Data Augmentation for Text Classifiers with Natural Language Explanations (2019, methodology): “NLPROLOG considers entity-masked support sentence as relation, calculates relation-predicate similarity and entity-entity similarity with mapped SENT2VEC embeddings (Pagliardini et al., 2018), and use these similarity scores for weak unification to solve candidate statements with a Prolog solver.”
Simple and Effective Paraphrastic Similarity from Parallel Translations (2019, methodology): “In this paper, we propose a method that trains highly performant sentence embeddings (Pham et al., 2015; Hill et al., 2016; Pagliardini et al., 2017; McCann et al., 2017; Conneau et al., 2017) directly on bitext, obviating these intermediate steps and avoiding the noise and error propagation from…”
Zero-training Sentence Embedding via Orthogonal Basis (2018, methodology): “It also outperforms most parametric models including GRAN, InferSent, and Sent2Vec.”
TagRec: Automated Tagging of Questions with Hierarchical Learning Taxonomy (2021, methodology): “We embed the hierarchical labels using sentence representations methods like Sent2Vec [9] and Sentence-BERT [11].”
Finding and Characterizing Information Warfare Campaigns (2020, methodology): “Early approaches simply averaged word embeddings for sentences and documents [192] which were expanded in sent2vec [188] using word and ngram embeddings while simultaneously training the composition and the embedding.”
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations (2018, methodology): “SIF (Arora et al., 2017) computes sentence embeddings by taking weighted averages of word embeddings and then modifying them via SVD. Sent2vec (Pagliardini et al., 2017) learns n-gram embeddings and averages them.”