Issues in evaluating semantic spaces using word analogies

By Tal Linzen. 2016

160 citations recorded by the source corpus.

The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, background): “It has been argued that what is observed through the analogy task might be mainly due to irrelevant neighborhood structure rather than to the vector offset that supposedly captures the analogy itself (Linzen 2016; Rogers, Drozd, and Li 2017).”
Inducing Relational Knowledge from BERT (2019, background): “…somewhat supported by theoretical insights (Arora et al. 2016; Allen and Hospedales 2019), simple linear transformations of word vectors have been found limiting in more general settings (Levy and Goldberg 2014; Linzen 2016; Rogers, Drozd, and Li 2017; Nissim, van Noord, and van der Goot 2019).”
Distributional Effects of Gender Contrasts Across Categories (2019, methodology): “We preferred testing both rank and distance measurements over the more widespread cosine similarity measure so as to take into account neighborhood structure (Linzen, 2016).”
Harvesting Common-sense Navigational Knowledge for Robotics from Uncurated Text Corpora (2017, background): “This tendency of analogy operations to produce results close to their origins is well documented and easily verifiable [21, 17].”
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model (2019, background): “, 2013b), although this assumption has been subjected to criticism (Linzen, 2016).”
Geographical Evaluation of Word Embeddings (2017, background): “Linzen (2016) provides a detailed analysis of the word analogy task and shows that results in this evaluation are to a large extent based on proximity in the semantic space rather than consistent offsets between the word pairs.”
The (too Many) Problems of Analogical Reasoning with Word Vectors (2017, background): “Linzen (2016) showed that if that is not done, the accuracy drops dramatically, hitting zero for 9 out of 15 Google test categories.”
Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen (2016, background): “…fact that in L05, I05, and D05 the most frequent type of mistakes are the wrong morphological forms of a source word is consistent with the ﬁnding of Linzen (2016) that in the noun plural category of the Google test set the nearest neighbor of the c word provides the correct answer in 70% of cases.”
Metaphoria: An Algorithmic Companion for Metaphor Creation (2019, background): “We expect that Intersecting will not be particularly apt as it relies solely on the embedding space to provide meaning and embedding spaces notoriously lack consistent discrete semantics [23].”