Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

By Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, Chris Dyer. 2016

283 citations recorded by the source corpus.

Lacking standardized extrinsic evaluation methods for vector representations of words, the NLP community has relied heavily on word similarity tasks as a proxy for intrinsic evaluation of word vectors. Word similarity evaluation, which correlates the distance between vectors and human judgments of semantic similarity is attractive, because it is computationally inexpensive and fast. In this paper we present several problems associated with the evaluation of word vectors on word similarity datasets, and summarize existing solutions. Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Evaluation & Benchmarks

Citations in the corpus

SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “The evaluation of general-purpose word and sentence embeddings has been problematic (Chiu et al., 2016; Faruqui et al., 2016), leading to much discussion about the best way to go about it 1 .”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Although these datasets are widely used, this kind of evaluation has been criticized for its subjectivity and questionable correlation with downstream performance (Faruqui et al., 2016).”
A Survey of Cross-lingual Word Embedding Models (2017, neutral): “(2015) LSGNS Ω∞ Monotonic alignment Gardner et al. (2015) LLSA Ω∞ Matrix factorization Pham et al. (2015) LSGNS-P Ω∞ X Paragraph vectors Guo et al.”
Word Embeddings and Gender Stereotypes in Swedish and English (2019, background): “Moreover, there is an apparent risk of overfitting and there are no tests for statistical significance [13].”
Learning Word Embeddings from Speech (2017, unknown): “Using word similarity tasks as the only way to measure the quality of word vectors is not perfect and can sometimes lead to incorrect inferences [Faruqui et al., 2016, Schnabel et al., 2015].”
Coming to Your Senses: on Controls and Evaluation Sets in Polysemy Research (2018, background): “These conclusions join a general skepticism expressed in the literature about the use of word similarity tasks for the evaluation of word vectors (Hill et al., 2015; Avraham and Goldberg, 2016; Batchkarov et al., 2016; Faruqui et al., 2016).”
SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity (2016, methodology): “Previous works often optimise models on the entire dataset, which leads to overfitting (Faruqui et al., 2016) or use custom splits, e.g., 10-fold cross-validation (Schwartz et al., 2015), which make results incomparable with others.”
The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling (2020, methodology): “Distance-based methods can also be used to evaluate the semantic representation of words, by computing the correlation between these distances and human semantic similarity judgements (see Schnabel et al., 2015; Faruqui et al., 2016).”
Geographical Evaluation of Word Embeddings (2017, methodology): “The word similarity task was analysed and criticized by multiple authors (Faruqui et al., 2016; Chiu et al., 2016; Batchkarov et al., 2016; Gladkova and Drozd, 2016).”
Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks (2017, background): “However, in spite of being popular and computationally cheap, [Faruqui et al. 2016] suggests that word analogies are not appropriate for evaluating embeddings.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, background): “This is especially relevant because intrinsic evaluation protocols do not always correlate with downstream performance (Tsvetkov, Faruqui, Ling, Lample, & Dyer, 2015; Chiu, Korhonen, & Pyysalo, 2016; Faruqui, Tsvetkov, Rastogi, & Dyer, 2016).”
Supersense Embeddings: A Unified Model for Supersense Interpretation, Prediction, and Utilization (2016, background): “…we report the performance of our embeddings on the word similarity tasks for completeness, there has been a substantial discussion on seeking alternative ways to quantify embedding quality with the focus on their purpose in downstream applications (Li and Jurafsky, 2015; Faruqui et al., 2016).”
Evaluating word embedding models: methods and experimental results (2019, background): “For the latter, the way distributional semantic models simulate similarity is still ambiguous [35].”
Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context (2017, background): “…set of experiment to evaluate whether we should choose languages from a closer family (Indo-European languages) or farther family (Non-Indo European Languages) 5 Most works tune directly on the test dataset for Word Similarity tasks (Faruqui et al., 2016) as training data alongside English.”
Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation (2018, background): “For instance, Faruqui et al. (2016) and Batchkarov et al. (2016) argue that word similarity has many problems like the sub-jectivity and difﬁculty of the task, the lack of statistical signiﬁcance and the inability to account for polysemy, warning that results should be interpreted with care.”