A Simple but Tough-to-Beat Baseline for Sentence Embeddings

By Sanjeev Arora, Yingyu Liang, Tengyu Ma. 2017

1,419 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, methodology): “Arora et al. (2016) consider a weighted BoW model followed by simple post-processing and show that it performs better than BoW models trained on paraphrase data.”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “The approach is similar to SIF (Arora et al., 2017) but without removal of the common principle component (18)RTV took first place on track 5, English, but submitted no system description paper.”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “The results were in-conclusive and did not reach the same level as simpler approaches that directly learn a classiﬁer on top of unsupervised sentence embeddings instead (Arora et al., 2017).”
Inductive Representation Learning on Large Graphs (2017, neutral): “’s [2] sentence embedding approach, with 300-dimensional word vectors trained using the GenSim word2vec implementation [30].”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “(2015), Word embedding average in Arora et al. (2016), DiscSent from Jernite et al. (2017), Byte mLSTM from Radford et al. (2017), Infersent in Conneau et al. (2017), Neural Semantic Encoder from Munkhdalai & Yu (2017), BLSTM-2DCNN from Zhou et al.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “More recent studies have showed that cross-lingual word embeddings can be generated with no supervision whatsoever (Artetxe et al., 2017; Conneau et al., 2018b).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “into representations of larger segments of text: weighted average of word embeddings with learned weights [De Boom et al., 2016], weighted average of word embeddings followed by modiﬁcation with SVD [Arora et al., 2017], random walks [Ethayarajh, 2018], and different pooling techniques [Shen et al.,”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “Although the BOW vector ignores the word-order information in the sentence, it is empirically shown to be effective in preserving the high-level semantic information in the sentences and achieves success in sentence similarity and entailment tasks (Wieting et al., 2016; Arora et al., 2017).”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “…and construct sentence embeddings (Kiros et al., 2015; Wieting et al., 2016; Conneau et al., 2017; Artetxe and Schwenk, 2019; Cer et al., 2018; Arora et al., 2017) that allow us to retrieve domain-speciﬁc unannotated sentences, which are similar to the existing training set of the downstream…”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, background): “Our ﬁrst striking result is the good overall performance of Bag-of-Vectors, conﬁrming early insights that aggregated word embeddings capture surprising amounts of sentence information (Pham et al., 2015; Arora et al., 2017; Adi et al., 2017).”
What makes a good conversation? How controllable attributes affect human judgments (2019, methodology): “weighted average of the GloVe embeddings of the words in s, with the first principal component projected out; for full details, see Arora et al. (2017). This method of controlling response-relatedness is similar to that described in (Baheti et al.”
Learning Semantic Textual Similarity from Conversations (2018, methodology): “, 2017), SIF (Arora et al., 2017), PV-DBOW (Lau and Baldwin, 2016), C-PHRASE (Kruszewski et al.”
FRAGE: Frequency-Agnostic Word Representation (2018, background): “Different from classic one-hot representation, the learned word embeddings contain semantic information which can measure the semantic similarity between words [28], and can also be transferred into other learning tasks [29, 3].”
Learning protein sequence embeddings using information from structure (2019, methodology): “The concept of feeding LSTM language model representations as inputs for supervised learning problems as part of a larger neural network model has shown recent success in NLP but has not yet been tried for biological sequences.”
Continual Learning for Sentence Representations Using Conceptors (2019, methodology): “Among different linear sentence encoders, the smoothed inverse frequency (SIF) approach (Arora et al., 2017) is a prominent example – it outperforms many neuralnetwork based sentence encoders on a battery of NLP tasks (Arora et al.”
Joint Embedding of Words and Labels for Text Classification (2018, background): “On the contrary, recent studies show that simple manipulation of the word embeddings, e.g., mean or max-pooling, can also provide surprisingly excellent performance (Joulin et al., 2016; Wieting et al., 2016; Arora et al., 2017; Shen et al., 2018a).”
Assessing Composition in Sentence Vector Representations (2018, background): “As discussed above, the BOW model serves primarily as a sanity check for our purposes, but it is important to note that this model has had competitive results on various tasks, and is taken seriously as a sentence representation method for many purposes (Wieting et al., 2016; Arora et al., 2016).”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “…that can be used without finetuning for downstream tasks (Le & Mikolov, 2014; Kiros et al., 2015; Wieting et al., 2016; Hill et al., 2016; Arora et al., 2017; Conneau et al., 2017; Cer et al., 2018; Zhang et al., 2018; Subramanian et al., 2018; Reimers & Gurevych, 2019; Zhang et al.,…”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “A simple way to alleviate the problem is postprocessing, either to eliminate the dominant principal components (Arora et al., 2017; Mu and Viswanath, 2018), or to map embeddings to an isotropic distribution (Li et al., 2020; Su et al., 2021).”