Discovering Discrete Latent Topics with Neural Variational Inference

By Yishu Miao, Edward Grefenstette, Phil Blunsom. International Conference on Machine Learning, 2017

344 citations recorded by the source corpus.

Topic models have been widely explored as probabilistic generative models of documents. Traditional inference methods have sought closed-form derivations for updating the models, however as the expressiveness of these models grows, so does the difficulty of performing fast and accurate inference over their parameters. This paper presents alternative neural approaches to topic modelling by providing parameterisable distributions over topics which permit training by backpropagation in the framework of neural variational inference. In addition, with the help of a stick-breaking construction, we propose a recurrent network that is able to discover a notionally unbounded number of topics, analogous to Bayesian non-parametric topic models. Experimental results on the MXM Song Lyrics, 20NewsGroups and Reuters News datasets demonstrate the effectiveness and efficiency of these neural topic models.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/4f4837c1f8a194d8b7fe7d6073ce44876f08ad6c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lawrence Carin
2. Zhe Gan
3. Chunyuan Li
4. Jianfeng Gao
5. Mohit Bansal
6. Alexander M. Rush
7. Graham Neubig
8. Kyunghyun Cho

In Signal Processing

1. Wei Ping
2. Jiaji Huang
3. Heng Ji
4. Zhe Gan
5. Lawrence Carin
6. Sanjeev Satheesh
7. Ricardo Henao
8. David Barber

Citations in the corpus

• Sequence-to-Sequence Learning with Latent Neural Grammars (2021, neutral): “…has much work on neural parameterizations of classic probabilistic latent variable models including hidden Markov models, [109, 113, 21], finite state transducers [91, 71] topic models [82, 30, 31], dependency models [59, 51, 15, 52, 123], and context-free grammars [ 64, 61, 133, 131, 125, 124].”
• A Survey of Knowledge-enhanced Text Generation (2020, methodology): “parameterized the multinomial distributions with neural networks and jointly learned the model parameters via variational inference [97].”
• Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2020, background): “Several topic models use neural networks (Larochelle and Lauly, 2012; Salakhutdinov and Hinton, 2009; Gupta et al., 2020) or neural variational inference (Miao et al., 2016; Mnih and Gregor, 2014; Srivastava and Sutton, 2017; Miao et al., 2017; Ding et al., 2018).”
• Semi-Amortized Variational Autoencoders (2018, background): “Many works on VAEs for text have thus made simplifying conditional independence assumptions (Miao et al., 2016; 2017), used less powerful generative models such as convolutional networks (Yang et al., 2017; Semeniuta et al., 2017), or combined a recurrent generative model with a topic model (Dieng…”
• Generating Sentences Using a Dynamic Canvas (2018, methodology): “More recently, Miao, Grefenstette, and Blunsom (2017) use an RNN based generative model similar to Gen-RNN in order to perform topic allocation to documents.”
• Variational Russian Roulette for Deep Bayesian Nonparametrics (2019, background): “Similarly, recent work on amortized inference in Bayesian non-parametrics relies on truncation, albeit sometimes within an outer loop that searches over the truncation size (Miao et al., 2017; Nalisnick & Smyth, 2017; Chatzis, 2014; Singh et al., 2017).”
• NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing (2018, methodology): “Inspired by the recent success of variational autoencoders for various NLP problems (Miao et al., 2016; Bowman et al., 2015; Yang et al., 2017; Miao et al., 2017; Shen et al., 2017b; Wang et al., 2018), we approach the training of discrete (binary) latent variables from a generative perspec-”
• Topic Compositional Neural Language Model (2017, methodology): “Distinct from LDA (Blei et al., 2003), we pass a Gaussian random vector through a softmax function to parameterize the multinomial document topic distributions (Miao et al., 2017).”
• Dirichlet Variational Autoencoder (2019, background): “Recently, some neural variational topic (or document) models have been suggested, for example, ProdLDA (Srivastava & Sutton, 2017), NVDM (Miao et al., 2016), and GSM (Miao et al., 2017).”