Gaussian LDA for Topic Models with Word Embeddings

By Rajarshi Das, Manzil Zaheer, Chris Dyer. Annual Meeting of the Association for Computational Linguistics, 2015

363 citations recorded by the source corpus.

Continuous space word embeddings learned from large, unstructured corpora have been shown to be effective at capturing semantic regularities in language. In this paper we replace LDA’s parameterization of “topics” as categorical distributions over opaque word types with multivariate Gaussian distributions on the embedding space. This encourages the model to group words that are a priori known to be semantically related into topics. To perform inference, we introduce a fast collapsed Gibbs sampling algorithm based on Cholesky decompositions of covariance matrices of the posterior predictive distributions. We further derive a scalable algorithm that draws samples from stale posterior predictive distributions and corrects them with a Metropolis–Hastings step. Using vectors learned from a domain-general corpus (English Wikipedia), we report results on two document collections (20-newsgroups and NIPS). Qualitatively, Gaussian LDA infers different (but still very sensible) topics relative to standard LDA. Quantitatively, our technique outperforms existing models at dealing with OOV words in held-out documents.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b13a3939d4a822c246bb6ace395b450b99f1d454

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Context Learning (2017, methodology): “[79] proposes a Gaussian LDA model based on word embeddings.”
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2020, background): “, 2015), or pre-trained word embeddings (Das et al., 2015; Dieng et al., 2019; Nguyen et al., 2015).”
Topic Modeling Using Latent Dirichlet allocation (2021, background): “The proposed approach could assign high topical probability, which is a topical term but not available in the dictionary [26].”
Incorporating Knowledge Graph Embeddings into Topic Modeling (2017, neutral): “(Das, Zaheer, and Dyer 2015) replaced LDA’s parameterization of “topics” as categorical distributions over words with multivariate Gaussian distributions on the word embedding space, in which the semantic similarity is measured by Euclidean distance of word vectors.”
Statistical Model Aggregation via Parameter Matching (2019, positive): “We present a practical scenario where problem similar to our simulated experiment arises — learning Gaussian topic models [7] where local topic models are learned from the Gutenberg dataset comprising 40 books.”
Topic Modeling for Short Texts with Auxiliary Word Embeddings (2016, methodology): “Note that [11] also proposes a LDA-based model by using multivariate Gaussian distributions with auxiliary word embeddings.”
Neural Topic Modeling with Continual Lifelong Learning (2020, background): “To deal with this problem, there have been several attempts (Petterson et al., 2010; Das et al., 2015; Nguyen et al., 2015; Gupta et al., 2019) that introduce prior knowledge such as pre-trained word embeddings (Pennington et al., 2014) to guide meaningful learning.”
Feuding Families and Former Friends: Unsupervised Learning for Dynamic Fictional Relationships (2016, background): “It is also similar to recent neural topic model architectures (Cao et al., 2015; Das et al., 2015), although these models are limited to static document representations.”