Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance

By Billy Chiu, Anna Korhonen, Sampo Pyysalo. 2016

113 citations recorded by the source corpus.

The quality of word representations is frequently assessed using correlation with human judgements of word similarity. Here, we question whether such intrinsic evaluation can predict the merits of the representations for downstream tasks. We study the correlation between results on ten word similarity benchmarks and tagger performance on three standard sequence labeling tasks using a variety of word vectors induced from an unannotated corpus of 3.8 billion words, and demonstrate that most intrinsic evaluations are poor predictors of downstream performance. We argue that this issue can be traced in part to a failure to distinguish specific similarity from relatedness in intrinsic evaluation datasets. We make our evaluation tools openly available to facilitate further study.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “The evaluation of general-purpose word and sentence embeddings has been problematic (Chiu et al., 2016; Faruqui et al., 2016), leading to much discussion about the best way to go about it 1 .”
Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation (2018, background): “Chiu et al. (2016) analyze the correlation between results on word similarity benchmarks and sequence labeling tasks, and conclude that most intrinsic evaluations are poor predictors of downstream performance.”
Finnish resources for evaluating language model semantics (2017, background): “Notably, performance in SimLex-999 predicts the performance in downstream applications, unlike most intrinsic evaluation benchmarks (Chiu et al., 2016).”
Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter (2018, result): “s with spam removed [21]5. However, our results did not ﬁnd any embeddings that can consistently outperform others on all tasks and datasets. In fact, this is rather unsurprising, as previous studies [6] suggested similar patterns: the superiority of one word embeddings model on intrinsic tasks (e.g”
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks (2017, methodology): “Despite popularity of word embeddings, there is no clear consensus what evaluation methods should be used, and both intrinsic and downstream evaluations are criticized [Tsvetkov et al., 2015a, Faruqui et al., 2016].”
Geographical Evaluation of Word Embeddings (2017, methodology): “The word similarity task was analysed and criticized by multiple authors (Faruqui et al., 2016; Chiu et al., 2016; Batchkarov et al., 2016; Gladkova and Drozd, 2016).”
How to Train good Word Embeddings for Biomedical NLP (2016, result): “We have previously found a similar result in general domain work (with Wikipedia text) (Chiu et al., 2016).”
Evaluating word embedding models: methods and experimental results (2019, methodology): “PTB 337,195 129,892 accuracy CoNLL’00 211,727 47,377 F-score CoNLL’03 203,621 46,435 F-score For inference tools, we use the simple window-based feedforward neural network architecture implemented by [16]. It takes inputs of ﬁve at one time and passes them through a 300-unit hidden layer, a tanh act”
Towards a Seamless Integration of Word Senses into Downstream NLP Applications (2017, background): “In fact, research has previously shown that word similarity might not constitute a reliable proxy to measure the performance of word embed-dings in downstream applications (Tsvetkov et al., 2015; Chiu et al., 2016).”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, background): “This is especially relevant because intrinsic evaluation protocols do not always correlate with downstream performance (Tsvetkov, Faruqui, Ling, Lample, & Dyer, 2015; Chiu, Korhonen, & Pyysalo, 2016; Faruqui, Tsvetkov, Rastogi, & Dyer, 2016).”