Linguistic Regularities in Sparse and Explicit Word Representations

By Omer Levy, Yoav Goldberg. 2014

738 citations recorded by the source corpus.

Recent work has shown that neuralembedded word representations capture many relational similarities, which can be recovered by means of vector arithmetic in the embedded space. We show that Mikolov et al.’s method of first adding and subtracting word vectors, and then searching for a word similar to the result, is equivalent to searching for a word that maximizes a linear combination of three pairwise word similarities. Based on this observation, we suggest an improved method of recovering relational similarities, improving the state-of-the-art results on two recent word-analogy datasets. Moreover, we demonstrate that analogy recovery is not restricted to neural word embeddings, and that a similar amount of relational similarities can be recovered from traditional distributional word representations.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

GloVe: Global Vectors for Word Representation (2014, background): “Mnih and Kavukcuoglu (2013) also proposed closely-related vector log-bilinear models, vLBL and ivLBL, and Levy et al. (2014) proposed explicit word embed-dings based on a PPMI metric.”
Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, methodology): “We use the standard 3 COSADD function Equation (1) from Mikolov et al. (2013), and 3 COSMUL , introduced by Levy and Goldberg (2014) to overcome some of the shortcomings of 3 COSADD , mainly ensuring that a single large term cannot dominate the expression Equation (2 They do not assume that B is…”
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016, background): “, [23, 30, 40, 9]), and the difficulty of evaluating embedding quality (as compared to supervised learning) parallels the difficulty of defining bias in an embedding.”
Recovery guarantee of weighted low-rank approximation via alternating minimization (2016, background): “When doing low-rank approximation on the co-o currence matrix to get word embeddings, it has been observed empirically that a simple weighting can lead to much better performance than the unweighted formulation (see, e.g., [Levy and Goldberg, 2014]).”
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (2016, background): “We hypothesize that this occurs because a sizable part of the auto-encoder performance comes from the decoder, which also improves as we add more dimensions.”
Learning visual groups from co-occurrences in space and time (2015, positive): “Previous work on visual grouping Isola et al. (2014) and word embeddings (Church & Hanks (1990); Levy et al. (2014)) has found PMI to be an effective measure for these tasks.”
Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015, background): “Positional contexts might contain relevant information for recovering syntactic analogies, explaining PPMI’s relatively high score on MSR’s analogy task in (Levy and Goldberg, 2014b).”
Inducing Relational Knowledge from BERT (2019, background): “…somewhat supported by theoretical insights (Arora et al. 2016; Allen and Hospedales 2019), simple linear transformations of word vectors have been found limiting in more general settings (Levy and Goldberg 2014; Linzen 2016; Rogers, Drozd, and Li 2017; Nissim, van Noord, and van der Goot 2019).”
Dependency-Based Word Embeddings (2014, methodology): “In particular, we found that DEPS perform dramatically worse than BOW contexts on analogy tasks as in (Mikolov et al., 2013c; Levy and Goldberg, 2014).”
code2vec: learning distributed representations of code (2018, methodology): “Previous work in NLP has suggested a variety of methods for combining similarities [Levy and Goldberg 2014a] for the task of natural language analogy recovery.”
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (2019, background): “Though they learn embeddings iteratively in practice, it has been proven that in theory, they both implicitly factorize a word-context matrix containing a co-occurrence statistic (Levy and Goldberg, 2014a,b).”
Gender-preserving Debiasing for Pre-trained Word Embeddings (2019, methodology): “We use the CosAdd (Levy and Goldberg, 2014) that ﬁnds d that has the maximum cosine similarity with ( b − a + c ) .”
The (too Many) Problems of Analogical Reasoning with Word Vectors (2017, methodology): “We also added an implementation of 3CosMul, a multiplicative objective proposed by Levy and Goldberg (2014), now available in the same script 4 .”
Evaluating Neural Word Representations in Tensor-Based Compositional Settings (2014, methodology): “PPMI has been shown to achieve state-of-the-art results (Levy et al., 2014) and is suggested by the review of Kiela and Clark (2014).”
Multilingual Reliability and “Semantic” Structure of Continuous Word Spaces (2015, methodology): “We solve analogy tasks with the 3COSMUL method (Levy and Goldberg, 2014), and similarity tasks with cosine similarity.”
Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation (2018, background): “At the same time, we would like to extend our analysis to more specialized embedding models (Kiela et al., 2015; Levy and Goldberg, 2014a) to get a more complete picture of what information they capture.”
Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015, background): “In this work, driven by the observed linear linguistic regularities in the embedding spaces (e.g., [26, 20]) 4 , we opt for simple addition (denoted by +) from [29] as the compositional operator, due to its simplicity, the ease of applicability considering the bag-of-words assumption, as well as…”
Neural Word Embedding as Implicit Matrix Factorization (2014, methodology): “The analogy questions are answered using Levy and Goldberg’s similarity multiplication method [19], which is state-of-the-art in analogy recovery: arg maxb∗∈VW \{a∗,b,a} cos(b ∗, a∗)·cos(b∗, b)/(cos(b∗, a)+ε).”