Evaluating Word Embeddings Using a Representative Suite of Practical Tasks

By Neha Nayak Kennard, Gabor Angeli, Christopher D. Manning. 2016

58 citations recorded by the source corpus.

Word embeddings are increasingly used in natural language understanding tasks requiring sophisticated semantic information. However, the quality of new embedding methods is usually evaluated based on simple word similarity benchmarks. We propose evaluating word embeddings in vivo by evaluating them on a suite of popular downstream tasks. To ensure the ease of use of the evaluation, we take care to ﬁnd a good point in the tradeoff space between (1) creating a thorough evaluation – i.e., we evaluate on a diverse set of tasks; and (2) ensuring an easy and fast evaluation – by using simple models with few tuned hyperparameters. This allows us to release this evaluation as a standardized script and online evaluation, available at http://veceval.com/ .

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “On the other hand, it has been argued that the focus should be on downstream tasks where these representations would actually be applied (Ettinger et al., 2016; Nayak et al., 2016).”
LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations (2019, background): “Early works for evaluating word representations (Faruqui and Dyer, 2014a,b; Nayak et al., 2016) have mostly focused on English and used either the word similarity or a set of downstream tasks.”
Lifelong Domain Word Embedding via Meta-Learning (2018, methodology): “Following [Nayak et al. , 2016], we use the performances of down-stream tasks to evaluate the proposed method.”
Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks (2017, background): “The former relying mostly on word analogies (e.g. [Mikolov et al. 2013]) and measuring the semantic similarity between words (e.g. the WS-353 dataset [Finkelstein et al. 2002]), while extrinsic evaluations focus on practical NLP tasks (e.g. [Nayak et al. 2016]).”
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks (2017, methodology): “Recent research in evaluation methods focuses on representative or inter-1 pretable set of tasks [Nayak et al., 2016, Köhn, 2015a], analysing intrinsic evaluation [Chiu et al., 2016, Faruqui et al., 2016], as well as proposing improvements to intrinsic evaluation [Avra-ham and Goldberg, 2016,…”
Geographical Evaluation of Word Embeddings (2017, methodology): “There exist many tasks that could be used for evaluation, but it is intractable to use all of them (Nayak et al., 2016).”
A Comparison of Word Embeddings for the Biomedical Natural Language Processing (2018, background): “In Nayak et al’s study [47], they recommended that the evaluation should test both syntactic and semantic properties of the word embeddings and that the tasks should be closer to real-word applications.”