Topical Word Embeddings

By Yang Liu, Zhiyuan Liu, Tat-seng Chua, Maosong Sun. 2015

429 citations recorded by the source corpus.

Most word embedding models typically represent each word using a single vector, which makes these models indiscriminative for ubiquitous homonymy and polysemy. In order to enhance discriminativeness, we employ latent topic models to assign topics for each word in the text corpus, and learn topical word embeddings (TWE) based on both words and their topics. In this way, contextual word embeddings can be flexibly obtained to measure contextual word similarity. We can also build document representations, which are more expressive than some widely-used document models such as latent topic models. In the experiments, we evaluate the TWE models on two tasks, contextual word similarity and text classification. The experimental results show that our models outperform typical word embedding models including the multi-prototype version on contextual word similarity, and also exceed latent topic models and other representative document models on text classification.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Joint Topic-Semantic-aware Social Recommendation for Online Voting (2017, background): “As a combination, [18] tries to learn topical word embeddings based on both words and their topics.”
Regularizing Text Categorization with Clusters of Words (2016, background): “Furthermore, topical word embeddings (Liu et al., 2015) can be considered for regularization.”
PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks (2015, neutral): “Similar results to ours are also reported in [14].”
Improving Topic Models with Latent Feature Word Representations (2015, background): “Recent approaches based on deep neural networks learn vectors by predicting words given their window-based context (Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014; Liu et al., 2015).”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, methodology): “The Neural Tensor Skip-gram (NTSG) model (Liu et al., 2015a) applies the same idea of topic modeling for sense representation but introduces a tensor to better learn the interactions between words and topics.”
BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022, methodology): “The incorporation of word embeddings into classical models, such as LDA, demonstrated the viability of using these powerful representations (Liu et al., 2015; Nguyen et al., 2015; Shi et al., 2017; Qiang et al., 2017).”
Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems (2019, background): “Embedding representations power state-of-the-art applications in diverse domains, including computer vision [7, 53], natural language processing [13, 33, 31, 49, 2, 43, 32], computational biology [5], and recommendation systems [12, 41, 54].”
De-Conflated Semantic Representations (2016, background): “Other prominent work in the category include topical word embeddings (Liu et al., 2015) which use latent topic models for assigning topics to each word in a corpus and learn topic-speciﬁc word representations, and the technique proposed by Huang et al. (2012) which incorporates “global document…”
Query Expansion with Locally-Trained Word Embeddings (2016, neutral): “, 2012) or topicspecific embeddings (Liu et al., 2015).”
Bias in word embeddings (2020, result): “Word embeddings trained by different models yield different results on benchmark tests [19, 70, 74, 79].”