Evaluating word embedding models: methods and experimental results

By Bin Wang, Angela Wang, Fenxiao Chen, Yun Cheng Wang, C.-C. Jay Kuo. APSIPA Transactions on Signal and Information Processing, 2019

308 citations recorded by the source corpus.

Abstract Extensive evaluation on a large number of word embedding models for language processing applications is conducted in this work. First, we introduce popular word embedding models and discuss desired properties of word models and evaluation methods (or evaluators). Then, we categorize evaluators into intrinsic and extrinsic two types. Intrinsic evaluators test the quality of a representation independent of specific natural language processing tasks while extrinsic evaluators use word embeddings as input features to a downstream task and measure changes in performance metrics specific to that task. We report experimental results of intrinsic and extrinsic evaluators on six word embedding models. It is shown that different evaluators focus on different aspects of word models, and some are more correlated with natural language processing tasks. Finally, we adopt correlation analysis to study performance consistency of extrinsic and intrinsic evaluators.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/80270172e6c341626458b1daaf0c642e85d205ec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. William Timkey
2. Marten van Schijndel

Citations in the corpus

• All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, background): “…the cosine similarity of pairs of word embeddings in Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) shows a robust correlation with human similarity judgments, and embeddings cluster into natural semantic classes in Euclidean space (Baroni et al., 2014; Wang et al., 2019).”