Evaluation methods for unsupervised word embeddings

By Tobias Schnabel, Igor Labutov, David Mimno, Thorsten Joachims. 2015

583 citations recorded by the source corpus.

We present a comprehensive study of evaluation methods for unsupervised embedding techniques that obtain meaningful representations of words from text. Different evaluations result in different orderings of embedding methods, calling into question the common assumption that there is one single optimal vector representation. We present new evaluation techniques that directly compare embeddings with respect to specific queries. These methods reduce bias, provide greater insight, and allow us to solicit data-driven relevance judgments rapidly and accurately through crowdsourcing.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “Approaches to language model evaluation fall into two major categories: intrinsic and extrinsic evaluation [39].”
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval (2019, background): “This extent of independence from frequency signals is uncommon in previous term weighting approaches; even semantic text representations such as Word2Vec [19] and Glove [24] are reported to have high correlation with term frequency [30].”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, methodology): “Here we follow the setting and the proposed al-gorithm in (Baroni et al., 2014; Schnabel et al., 2015) – we cluster words (via their representations) using the classical k -Means algorithm (with ﬁxed k ).”
Unsupervised Post-processing of Word Vectors via Conceptor Negation (2018, methodology): “Only evaluating the post-processed word vectors using these benchmarks, however, invites an obvious critique: the success of intrinsic evaluation tasks may not transfer to downstream NLP tasks, as suggested by previous research (Schnabel et al. 2015).”
A Dual Embedding Space Model for Document Ranking (2016, background): “However, more recent works [16, 40] have shown that there does not seem to be one embedding approach that is best for all tasks.”
Embedding-based News Recommendation for Millions of Users (2017, background): “While there have been reports [11, 16] that the representations generated with these methods provide helpful features for sentiment classification tasks, it has not been clarified whether these representations would be suitable for our purposes because these methods are unsupervised approaches.”
Learning Word Embeddings from Speech (2017, unknown): “Using word similarity tasks as the only way to measure the quality of word vectors is not perfect and can sometimes lead to incorrect inferences [Faruqui et al., 2016, Schnabel et al., 2015].”
Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance (2016, background): “The use of vector representations of words is now pervasive in natural language processing, and the importance of their evaluation is increasingly recognized (Collobert and Weston, 2008; Turian et al., 2010; Mikolov et al., 2013a; Faruqui and Dyer, 2014; Chen et al., 2013; Schnabel et al., 2015).”
Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation (2017, methodology): “…in semantic models is often measured using intrinsic evaluations that capture particular types of relationships (typically semantic similarity and relatedness) be-tween word pairs (Finkelstein et al., 2002; Hill et al., 2015; Schnabel et al., 2015; Tsvetkov et al., 2015, inter alia).”
Evaluation of Word Vector Representations by Subspace Alignment (2015, background): “In a recent related study, Schnabel et al. (2015) also observe that existing word similarity and text categorization evaluations yield different orderings of word vector models.”
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images (2016, methodology): “This relative comparison approach was commonly used to evaluate and compare different word embedding models [30].”
Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates (2020, background): “Future work could draw from a rich history in NLP of evaluating representations of topic models and embeddings (Wallach et al., 2009; Bojanowski et al., 2017; Schnabel et al., 2015) and evaluating semantic similarity (Cer et al., 2017; Bojanowski et al., 2017; Reimers and Gurevych, 2019).”
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, background): “Schnabel et al. (2015) found that static embedding models encode a substantial degree of word frequency information, which leads to a frequency bias in cosine similarity.”
Cross-lingual Models of Word Embeddings: An Empirical Comparison (2016, result): “The fact that the best models according to QVEC and word similarities are different reinforces observations made in previous work that performance on word similarity tasks alone does not reflect quantification of linguistic properties of words (Tsvetkov et al., 2015; Schnabel et al., 2015).”
Geographical Evaluation of Word Embeddings (2017, background): “Many evaluation approaches have already been proposed and they can be roughly divided to intrinsic and extrinsic (Schnabel et al., 2015).”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, unknown): “Existing schemes for evaluating methods for word embeddings fall into two major categories: extrinsic and intrinsic [29].”
Issues in evaluating semantic spaces using word analogies (2016, methodology): “This dataset, which has become a standard VSM evaluation set (Baroni et al., 2014; Faruqui et al., 2015; Schnabel et al., 2015; Zhai et al., 2016), contains 14 categories; see Table 1 for a full list.”
A Comparison of Word Embeddings for the Biomedical Natural Language Processing (2018, methodology): “Schnabel et al [39] trained the CBOW model of word2vec [1], C&W embeddings [40], Hellinger PCA [41], GloVe [42], TSCCA [43] and Sparse Random Projections [44] on a 2008 GloVe dump, and tested on the same fourteen datasets.”