Improving Topic Models with Latent Feature Word Representations

By Dat Quoc Nguyen, Richard Billingsley, Lan Du, Mark Johnson. Transactions of the Association for Computational Linguistics, 2015

358 citations recorded by the source corpus.

Probabilistic topic models are widely used to discover latent topics in document collections, while latent feature vector representations of words have been used to obtain high performance in many NLP tasks. In this paper, we extend two different Dirichlet multinomial topic models by incorporating latent feature vector representations of words trained on very large corpora to improve the word-topic mapping learnt on a smaller corpus. Experimental results show that by using information from the external corpora, our new models produce significant improvements on topic coherence, document clustering and document classification tasks, especially on datasets with few or short documents.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c4cf9211ede0dd617add15a0acae8e1be1fc11e8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Short Text Topic Modeling Techniques, Applications, and Performance: A Survey (2019, methodology): “STTM provides implementations of DMM [18], LF-DMM [28], GPU-DMM [29], GPU-PDMM [31], BTM [14], WNTM [32], SATM [20], and PTM [35].”
Topic Modeling in Embedding Spaces (2019, methodology): “Nguyen et al. (2015) mix the likelihood deﬁned by LDA with a log-linear model that uses pre-ﬁtted word embeddings; Bunk and Krestel (2018) randomly replace words drawn from a topic with their embeddings drawn from a Gaussian; and Xu et al. (2018) adopt a geometric perspective, using Wasserstein…”
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2020, background): “, 2020b), or pre-trained word embeddings (Das et al., 2015; Dieng et al., 2020; Nguyen et al., 2015; Zhao et al., 2017).”
BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022, methodology): “The incorporation of word embeddings into classical models, such as LDA, demonstrated the viability of using these powerful representations (Liu et al., 2015; Nguyen et al., 2015; Shi et al., 2017; Qiang et al., 2017).”
Neural Topic Modeling with Continual Lifelong Learning (2020, background): “To deal with this problem, there have been several attempts (Petterson et al., 2010; Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019) that introduce prior knowledge such as pre-trained word embeddings (Pennington et al.”
Topic Modeling for Short Texts with Auxiliary Word Embeddings (2016, methodology): “Most relevant to ours is the work by Nguyen et al. [24].”
Finding Names in Trove: Named Entity Recognition for Australian Historical Newspapers (2015, background): “In particular, the vector-space word representations learned by a neural network have been shown to successfully improve various NLP tasks (Collobert and Weston, 2008; Socher et al., 2013; Nguyen et al., 2015).”
Incorporating Knowledge Graph Embeddings into Topic Modeling (2017, positive): “• LF-LDA (Nguyen et al. 2015), a knowledge-based topic model using word embeddings trained from large external data2.”