Evaluation of Word Vector Representations by Subspace Alignment

By Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, Chris Dyer. 2015

158 citations recorded by the source corpus.

Unsupervisedly learned word vectors have proven to provide exceptionally effective features in many NLP tasks. Most common intrinsic evaluations of vector quality measure correlation with similarity judgments. However, these often correlate poorly with how well the learned representations perform as features in downstream evaluation tasks. We present QVEC—a computationally inexpensive intrinsic evaluation measure of the quality of word embeddings based on alignment to a matrix of features extracted from manually crafted lexical resources—that obtains strong correlation with performance of the vectors in a battery of downstream semantic evaluation tasks.1

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “In addition, we also evaluate the quality of our learned individual word representations using standard benchmarks (Faruqui & Dyer, 2014; Tsvetkov et al., 2015).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, methodology): “In this regard, our method is similar to QVEC (Tsvetkov et al., 2015).”
Massively Multilingual Word Embeddings (2016, methodology): “QVEC has been shown to correlate strongly with downstream semantic tasks (Tsvetkov et al., 2015).”
Investigating Language Universal and Specific Properties in Word Embeddings (2016, methodology): “Tsvetkov et al. (2015) propose a method to evaluate word embeddings through the alignment of distributional vectors and linguistic word vectors.”
A Survey of Cross-lingual Word Embedding Models (2017, background): “Document Vulić and Moens (2016) Vulić and Moens (2013) Vulić and Moens (2014) Søgaard et al.”
Bridging CNNs, RNNs, and Weighted Finite-State Machines (2018, background): “7 The transition could also be something in between, for instance by focusing on speciﬁc dimensions of a word’s meaning encoded in the vector, such as POS or semantic features like animacy or concreteness (Rubinstein et al., 2015; Tsvetkov et al., 2015).”
Towards a Seamless Integration of Word Senses into Downstream NLP Applications (2017, background): “In fact, research has previously shown that word similarity might not constitute a reliable proxy to measure the performance of word embed-dings in downstream applications (Tsvetkov et al., 2015; Chiu et al., 2016).”
Cross-lingual Models of Word Embeddings: An Empirical Comparison (2016, result): “The fact that the best models according to QVEC and word similarities are different reinforces observations made in previous work that performance on word similarity tasks alone does not reflect quantification of linguistic properties of words (Tsvetkov et al., 2015; Schnabel et al., 2015).”
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks (2016, methodology): “sic evaluation NLP tasks like text classification, parsing, sentiment analysis (Tsvetkov et al., 2015; Schnabel et al., 2015).”
The Role of Context Types and Dimensionality in Learning Word Embeddings (2016, background): “Recent works have shown that while intrinsic evaluations are easier to perform, their correlation with results on extrinsic evaluations is not very reliable (Schnabel et al., 2015; Tsvetkov et al., 2015),”
Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation (2017, background): “Following that, the correlation scores are averaged.”
Supersense Embeddings: A Unified Model for Supersense Interpretation, Prediction, and Utilization (2016, methodology): “Tsvetkov et al. (2015) proposes the usage of SemCor supersense frequencies as a way to evaluate word embedding models, showing that a good alignment of embedding dimensions to super-senses correlates with performance of the vectors in word similarity and text classiﬁcation tasks.”
Word Embeddings and Gender Stereotypes in Swedish and English (2019, background): “Firstly there needs to exist a standard way of quantitatively evaluating embeddings involving not only word analogies but also word similarities and extrinsic measures (or intrinsic ones like QVec [42]).”
Evaluating word embedding models: methods and experimental results (2019, background): “dentiﬁed by humans are strongly inﬂuenced by the characteristics of words perceived to be important. Yet, the recognized patterns might not be immediately clear to word embedding models. E. QVEC QVEC [39] is an intrinsic evaluator that measures the component-wise correlation between word vectors fro”
Evaluation methods for unsupervised word embeddings (2015, background): “Finally, Tsvetkov et al. (2015) propose a new intrinsic measure that better correlates with extrinsic performance.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, background): “This is especially relevant because intrinsic evaluation protocols do not always correlate with downstream performance (Tsvetkov, Faruqui, Ling, Lample, & Dyer, 2015; Chiu, Korhonen, & Pyysalo, 2016; Faruqui, Tsvetkov, Rastogi, & Dyer, 2016).”
Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs (2026, unknown): “Alignment-based criteria ask whether an embedding space is consistent with an external linguistic or structural reference [67, 22].”