How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

By Stanislaw Jastrzebski, Damian Lesniak, Wojciech M. Czarnecki. 2017

79 citations recorded by the source corpus.

Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non--linearly encoded in the embedding space, which questions the cosine-based, unsupervised, evaluation methods. All results and analysis scripts are available online.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “Skipgram and NMT results were obtained from Jastrzebski et al. (2017)5. Charagram and Attract-Repel results were taken from Wieting et al. (2016) and Mrkšić et al. (2017) respectively.”
DirectProbe: Studying Representations without Classifiers (2021, background): “Controlled test sets (¸Senel et al., 2018; Jastrzebski et al., 2017) are designed and errors are analyzed to reverse-engineer what information can be encoded by the model (e.g., Marvin and Linzen, 2018; Ravichander et al., 2020; Wu et al., 2020).”
Debiasing Word Embeddings from Sentiment Associations in Names (2020, methodology): “To ensure that the quality of the resulting embeddings does not suffer due to the debiasing efforts, we compare the computed embeddings based on our DebiasEmb approach, and other competitors, against the standard SGNS embeddings on standard benchmark tests [15].”