Autoencoding Variational Inference For Topic Models

By Akash Srivastava, Charles Sutton. International Conference on Learning Representations, 2017

675 citations recorded by the source corpus.

Topic models are one of the most popular methods for learning representations of text, but a major challenge is that any change to the topic model requires mathematically deriving a new inference algorithm. A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven diffi- cult to apply to topic models in practice. We present what is to our knowledge the first effective AEVB based inference method for latent Dirichlet allocation (LDA), which we call Autoencoded Variational Inference For Topic Model (AVITM). This model tackles the problems caused for AEVB by the Dirichlet prior and by component collapsing. We find that AVITM matches traditional methods in accuracy with much better inference time. Indeed, because of the inference network, we find that it is unnecessary to pay the computational cost of running variational optimization on test data. Because AVITM is black box, it is readily applied to new topic models. As a dramatic illustration of this, we present a new topic model called ProdLDA, that replaces the mixture model in LDA with a product of experts. By changing only one line of code from LDA, we find that ProdLDA yields much more interpretable topics, even if LDA is trained via collapsed Gibbs sampling.

Subfields: Artificial Intelligence, General Social Sciences

View Paper: https://www.semanticscholar.org/paper/0c25e5f544167840756950e0ff3edd238a19cfd1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

Citations in the corpus

Structured Disentangled Representations (2018, methodology): “We analyze this data using ProdLDA [Srivastava and Sutton, 2017] and neural variational document model (NVDM) [Miao et al.”
Latent Alignment and Variational Attention (2018, background): “Others have also noted difﬁculty in training Dirichlet models with amortized inference [69].”
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2020, methodology): “Several topic models use neural networks (Larochelle and Lauly, 2012; Salakhutdinov and Hinton, 2009; Gupta et al., 2020) or neural variational inference (Miao et al., 2016; Mnih and Gregor, 2014; Srivastava and Sutton, 2017; Miao et al., 2017; Ding et al., 2018).”
Neural Topic Modeling with Continual Lifelong Learning (2020, methodology): “Within the OTL framework, an unsupervised neural-network based topic model named as Document Neural Autoregressive Distribution Estimation (DocNADE) (Larochelle & Lauly, 2012; Lauly et al., 2017) has shown to outperform existing topic models based on LDA (Blei et al., 2003; Srivas-tava & Sutton, 201”
WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling (2018, methodology): “2b, which utilizes the logistic-normal distribution to approximate the posterior of the latent representation of a document; even though the generative model is latent Dirichlet allocation (LDA) (Blei et al., 2003), a basic single-hidden-layer topic model, due to the insufﬁcient ability of the logis”
Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training (2019, background): “Recently, neural topic models (Iyyer et al., 2016; Srivastava and Sutton, 2071; He et al., 2017) have been shown to produce more coherent topics than earlier models such as Latent Dirichlet Allocation (LDA) (Blei et al.”
Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters (2021, methodology): “CTM combines pre-trained Sentence-Transformers embedding representations (Reimers and Gurevych 2019) with a neural topic model Neural-ProdLDA (Srivastava and Sutton 2017) which takes advantage of Bag of Words (BoW) for more coherent representation.”
Dirichlet Variational Autoencoder (2019, methodology): “When we survey the work with explicit modeling on the Dirichlet prior, we found a frequent approach such as utilizing a softmax Laplace approximation (Srivastava & Sutton, 2017).”
Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis (2021, background): “While some of the approaches [13, 14, 15, 16, 17] rely on prosodic features such as fundamental frequency, duration and energy to generate the expressive speech but these approaches can generate expressive speech for the speakers who are already part of the training.”
Topic Modeling in Embedding Spaces (2019, background): “We also consider PRODLDA (Srivastava and Sutton, 2017).”
Advances in Variational Inference (2017, background): “This allows the usage of inference networks (Section 6) [122], [180].”