What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation

By Arne Köhn. 2015

93 citations recorded by the source corpus.

In the last two years, there has been a surge of word embedding algorithms and research on them. However, evaluation has mostly been carried out on a narrow set of tasks, mainly word similarity/relatedness and word relation similarity and on a single language, namely English. We propose an approach to evaluate embeddings on a variety of languages that also yields insights into the structure of the embedding space by investigating how well word embeddings cluster along different syntactic features. We show that all embedding approaches behave similarly in this task, with dependency-based embeddings performing best. This effect is even more pronounced when generating low dimensional embeddings.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Perhaps the ﬁrst studies that can be cast in the framework of probing classiﬁers are by Köhn (2015) and Gupta et al. (2015), who trained classiﬁers on static word embeddings to predict various morphological, syntactic, and semantic properties.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “(Köhn, 2015) Word embeddings POS, head POS, dependency relation, gender, case, number, tense Classification”
What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “Different units have been used, from word embeddings (Köhn, 2015; Qian et al., 2016b), through LSTM gates or states (Qian et al.”
Investigating Language Universal and Specific Properties in Word Embeddings (2016, background): “It is worth noticing that K¨ohn (2015) evaluates multilingual word embedding and compares skip-gram, language model and other competitive embedding models.”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “like vision and natural language processing, including word and sentence representations [22, 23, 24], machine translation [25, 26], and joint vision-language models [27].”
The Lazy Encoder: A Fine-Grained Analysis of the Role of Morphology in Neural Machine Translation (2018, methodology): “While this methodology is similar to that of previous work (K¨ohn, 2015; Belinkov et al., 2017a,b; Dalvi et al., 2017) we make sure that our results are not affected by overﬁtting by eliminating any vocabulary overlap between the classiﬁer’s training and test sets.”
Emergent linguistic structure in artificial neural networks trained by self-supervision (2020, neutral): “Considerable part-of-speech information was also available from context-free, word type-level embeddings like word2vec and GloVe (12, 13), as shown by earlier probing work (52), but the more recent studies additionally demonstrate the ability of contextual word representations to encode information ”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Perhaps the ﬁrst studies that can be cast in the framework of probing classiﬁers are by Köhn (2015) and Gupta et al. (2015), who trained classiﬁers on static word embeddings to predict various morphological, syntactic, and semantic properties.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “eludes word embeddings [138, 188, 279], RNN hidden states or gate activations [102, 278, 306, 335, 345], and sentence embeddings [2, 3, 47, 81, 106, 117].”
Post-hoc Interpretability for Neural NLP: A Survey (2021, methodology): “Additionally, some methods only look at word embeddings (Köhn, 2015).”
Analyzing Linguistic Knowledge in Sequential Model of Sentence (2016, methodology): “The basic method is to decode interpretable linguistic features from target neuron clusters, which has been used in (Köhn, 2015; Qian et al., 2016).”
LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations (2019, background): “K¨ohn (2015) introduced an ofﬂine, multilingual probing suite for static embeddings limited in terms of the languages and the probing tasks.”
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks (2017, methodology): “Recent research in evaluation methods focuses on representative or inter-1 pretable set of tasks [Nayak et al., 2016, Köhn, 2015a], analysing intrinsic evaluation [Chiu et al., 2016, Faruqui et al., 2016], as well as proposing improvements to intrinsic evaluation [Avra-ham and Goldberg, 2016,…”