A Neural Autoregressive Topic Model

By Hugo Larochelle, Stanislas Lauly. 2012

231 citations recorded by the source corpus.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Distributed Representations of Sentences and Documents (2014, methodology): “In this direction, autoencoder-style models have also been used to model paragraphs (Maas et al., 2011; Larochelle & Lauly, 2012; Srivastava et al., 2013).”
Neural Autoregressive Distribution Estimation (2016, methodology): “Larochelle and Lauly (2012); Zheng et al. (2015b) adapted NADE for topic modeling of documents and images, while Boulanger-Lewandowski et al. (2012) used NADE for modeling music sequential data.”
RNADE: The real-valued neural autoregressive density-estimator (2013, background): “NADE has recently been extended to count data [9].”
A Neural Autoregressive Approach to Collaborative Filtering (2016, background): “NADE together with its variants achieved competitive results on many machine learning tasks (Larochelle & Lauly, 2012; Uria et al., 2013; Zheng et al., 2014b; Uria et al., 2014; Zheng et al., 2014a; 2015).”
Neural Variational Inference for Text Processing (2015, background): “A primary feature of NVDM is that each word is generated directly from a dense continuous document representation instead of the more common binary semantic vector (Hinton & Salakhut-dinov, 2009; Larochelle & Lauly, 2012; Srivastava et al., 2013; Mnih & Gregor, 2014).”
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions (2017, methodology): “Related work (Miao et al., 2016; Larochelle & Lauly, 2012; Mnih & Gregor, 2014) has used simpler decoders that model text as a bag of words.”
Deep Exponential Families (2014, methodology): “As a baseline we consider Latent Dirichlet Allocation [5] a popular topic model, and state-of-the-art DocNADE [19].”
Neural Topic Modeling with Continual Lifelong Learning (2020, methodology): “Within the OTL framework, an unsupervised neural-network based topic model named as Document Neural Autoregressive Distribution Estimation (DocNADE) (Larochelle & Lauly, 2012; Lauly et al., 2017) has shown to outperform existing topic models based on LDA (Blei et al., 2003; Srivas-tava & Sutton,…”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “Dirichlet allocation (LDA) (Mcauliffe and Blei, 2008), DocNADE (Larochelle and Lauly, 2012), Replicated Softmax Model (RSM) (Hinton and Salakhutdinov, 2009) and Overreplicated Softmax Model (OSM) (Srivastava et al.”
Topic Compositional Neural Language Model (2017, background): “…neural topic models have attracted much attention, building upon the successful usage of restricted Boltzmann machines (Hinton and Salakhutdinov, 2009), auto-regressive models (Larochelle and Lauly, 2012), sigmoid belief networks (Gan et al., 2015), and variational autoencoders (Miao et al., 2016).”
Discovering Discrete Latent Topics with Neural Variational Inference (2017, methodology): “…a number of works have sought to leverage these, such as the Replicated Softmax (Hinton & Salakhutdinov, 2009), the Auto-Regressive Document Model (Larochelle & Lauly, 2012), Sigmoid Belief Document Model (Mnih & Gregor, 2014), Variational Auto-Encoder Document Model (NVDM) (Miao et al., 2016)…”
BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022, background): “In recent years, neural topic models have increasingly shown success in leveraging neural networks to improve upon existing topic modeling techniques (Terragni et al., 2021; Cao et al., 2015; Zhao et al., 2021; Larochelle and Lauly, 2012).”
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification (2016, methodology): “It also significantly outperforms four other methods, including supervised Latent Dirichlet allocation (LDA) [54], DocNADE [36], Replicated Softmax Model (RSM) [28] and Overreplicated Softmax Model (OSM) [65].”
Modeling Documents with Deep Boltzmann Machines (2013, methodology): “Recently, neural network based approaches, such as Neural Autoregressive Density Estimators (DocNADE) [7], have been to shown to outperform the Replicated Softmax model.”
Autoencoding Variational Inference For Topic Models (2017, unknown): “There are several examples of topic models based on neural networks and neural variational inference (Hinton & Salakhutdinov, 2009; Larochelle & Lauly, 2012; Mnih & Gregor, 2014; Miao et al., 2016) but we are unaware of meth-ods that apply AEVB generically to a topic model speciﬁed by an analyst,…”
NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing (2018, result): “Similar to (Miao et al., 2016; Larochelle and Lauly, 2012), we select the 5 nearest words according to the word vectors learned from NASH and compare with the corresponding results from NVDM.”
Topically Driven Neural Language Model (2017, background): “, 2003) have been proposed, including recent work on neural topic models (Cao et al., 2015; Wan et al., 2012; Larochelle and Lauly, 2012; Hinton and Salakhutdinov, 2009).”
Regularization and nonlinearities for neural language models: when are they needed? (2013, unknown): “In fact, a recent state-of-the-art document topic model [12] uses a similar parametrization of a neural network, there called NADE (neural autoregressor density estimation).”
Domain Randomization and Generative Models for Robotic Grasping (2017, methodology): “Autoregressive models have also been successfully applied to other forms of data including in topic modeling [19] and audio generation [33].”