Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline

By Kawin Ethayarajh. 2018

90 citations recorded by the source corpus.

Using a random walk model of text generation, Arora et al. (2017) proposed a strong baseline for computing sentence embeddings: take a weighted average of word embeddings and modify with SVD. This simple method even outperforms far more complex approaches such as LSTMs on textual similarity tasks. In this paper, we first show that word vector length has a confounding effect on the probability of a sentence being generated in Arora et al.’s model. We propose a random walk model that is robust to this confound, where the probability of word generation is inversely related to the angular distance between the word and sentence embeddings. Our approach beats Arora et al.’s by up to 44.4% on textual similarity tasks and is competitive with state-of-the-art methods. Unlike Arora et al.’s method, ours requires no hyperparameter tuning, which means it can be used when there is no labelled data.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Self-training Improves Pre-training for Natural Language Understanding (2020, background): “, 2013) and the uSIF approach (Ethayarajh, 2018b; Arora et al., 2017) as baselines in our experimental results.”
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2019, background): “Despite their inability to understand word order, they are surprisingly effective on sentence similarity tasks (Arora et al., 2017; Wieting and Gimpel, 2018; Ethayarajh, 2018) (b) Complex contextualized sentence encoders, such as Long Short Term Memory Networks (LSTM) (Hochreiter and Schmidhuber, 19”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, methodology): “This might motivate the creation of simpler methods whose ease-of-use makes up for weaker performance, such as weighted-average sentence embeddings (Arora et al., 2019; Ethayarajh, 2018).”
Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (2019, methodology): “For f G , we either use SIF-weighted averaged GloVe vectors (Section 3.2), or unweighted averaged ParaNMT 17 word and trigram vectors (Wieting and Gimpel, 2018), which are the current State of the Art on the unsupervised STS Benchmark test set (Ethayarajh, 2018).”
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025, unknown): “…on text embeddings focused on deriving sentence embeddings from word embeddings (Shen et al., 2018; Mu and Viswanath, 2018; Arora et al., 2017; Ethayarajh, 2018), while later methods such as InferSent (Conneau et al., 2017), Sentence-BERT (Reimers and Gurevych, 2019), and Supervised SimCSE…”