From Word to Sense Embeddings: A Survey on Vector Representations of Meaning

By José Camacho-Collados, Mohammad Taher Pilehvar. 2018

371 citations recorded by the source corpus.

Over the past years, distributed semantic representations have proved to be effective and flexible keepers of prior knowledge to be integrated into downstream applications. This survey focuses on the representation of meaning. We start from the theoretical background behind word vector space models and highlight one of their major limitations: the meaning conflation deficiency, which arises from representing a word with all its possible meanings as a single vector. Then, we explain how this deficiency can be addressed through a transition from the word level to the more fine-grained level of word senses (in its broader acceptation) as a method for modelling unambiguous lexical meaning. We present a comprehensive overview of the wide range of techniques in the two main branches of sense representation, i.e., unsupervised and knowledge-based. Finally, this survey covers the main evaluation procedures and applications for this type of representation, and provides an analysis of four of its important aspects: interpretability, sense granularity, adaptability to different domains and compositionality.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (2018, background): “Therefore, these embeddings are unable to re-ﬂect the dynamic nature of ambiguous words 1 , in that they can correspond to different (potentially unrelated) meanings depending on their usage in context (Camacho-Collados and Pilehvar, 2018).”
Counter-Interference Adapter for Multilingual Machine Translation (2021, background): “Prior research suggests that unified embedding on a joint vocabulary leads to meaning conflation deficiency in multilingual training (Camacho-Collados and Pilehvar, 2018).”
Evolution of Semantic Similarity—A Survey (2020, methodology): “[43] and IC-based methods[41], Camacho-Colladas et al.[17] discuss various vector representation methods of words, Taieb et al.[28] on the other hand describes various semantic relatedness methods and Berna AltÄśnel et al.[8] summarises various semantic similarity methods used for text…”
WiC: 10, 000 Example Pairs for Evaluating Context-Sensitive Representations (2018, background): “Therefore, these embeddings are unable to reflect the dynamic nature of ambiguous words(1), in that they can correspond to different (potentially unrelated) meanings depending on their usage in context (Camacho-Collados and Pilehvar, 2018).”
Learning and Evaluating Emotion Lexicons for 91 Languages (2020, background): “While sense embeddings form an active area of research (Camacho-Collados and Pilehvar, 2018; Chi and Chen, 2018), to the best of our knowledge, type-level embeddings yield state-of-the-art performance in downstream applications.”
Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings (2019, positive): “Panchenko et al. (2017) then also use induced sense embeddings for the downstream task of WSD. Camacho-Collados and Pilehvar (2018) provide an extensive overview of different word sense modeling approaches.”
Evaluation of sentence embeddings in downstream and linguistic probing tasks (2018, methodology): “ng (NLU) applications. These word representations improved downstream tasks in many domains such as machine translation, syntactic parsing, text classiﬁcation, and machine comprehension, among others [6]. Ranging from count-based to predictive or task-based methods, in the past years, many approache”
Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark (2024, background): “There are several reviews on text embeddings such as in [40], [41], [25], [42], [43], [44], but none of the existing work focus on the recent advances in the universal text embeddings in the fourth era.”
Open-Source Embedding Models: A Comprehensive Survey of Techniques, Benchmarks, and Applications (2026, unknown): “Classical models like Word2Vec, GloVe, and FastText [7], [8], [9] exemplify this era.”