Replicated Softmax: an Undirected Topic Model

By Ruslan Salakhutdinov, Geoffrey E. Hinton. 2009

540 citations recorded by the source corpus.

Subfields: Natural Language Processing, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Probabilistic & Bayesian Methods

Citations in the corpus

Least Squares Generative Adversarial Networks (2016, methodology): “tricted Boltzmann Machines (RBMs), one type of deep generative models, are the basis of many other hierarchical models, and they have been used to model the distributions of images [15] and documents [16]. Deep Belief Networks (DBNs) [17] and Deep Boltzmann Machines (DBMs) [5] are extended from the ”
On the challenges of learning with inference networks on sparse, high-dimensional data (2017, background): “…only about ten latent dimensions are Table 1: Test Perplexity on RCV1: Left: Base-lines Legend: LDA (Blei et al. , 2003), Replicated Softmax (RSM) (Hinton & Salakhutdinov, 2009), Sigmoid Belief Networks (SBN) and Deep Autoregressive Networks (DARN) (Mnih & Gregor, 2014), Neural Variational…”
Neural Variational Inference for Text Processing (2015, methodology): “All the log standard deviations are initialised as zero before training.”
Learning with Hierarchical-Deep Models (2013, background): “This gives us a familiar “bag-of-wor ds” representation [30], [36].”
Generating Multi-label Discrete Patient Records using Generative Adversarial Networks (2017, background): “There are studies that extend RBMs beyond binary variables (Hinton and Salakhutdinov, 2009; Gehler et al., 2006; Tran et al., 2011).”
Deep Exponential Families (2014, methodology): “Additionally computing perplexity from the predictive distribution requires computationally demanding sampling procedures which for most models like LDA only allow testing of only a small number (50) of documents [38, 33].”
Improving Topic Models with Latent Feature Word Representations (2015, methodology): “Rather than relying solely on a multinomial or latent feature model, as in Salakhutdinov and Hinton (2009), Srivastava et al. (2013) and Cao et al. (2015), we explore how to take advantage of both latent feature and multinomial models by using a latent feature representation trained on a large…”
Neural Networks and Deep Learning (2018, unknown): “The use of replicated softmax is discussed in [199].”
A Neural Autoregressive Topic Model (2012, methodology): “To address these issues, Salakhutdinov and Hinton [2] proposed the Replicated Softmax model, which uses the following energy function”
Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy (2016, methodology): “After the RBMs are stacked to form the deep generative models, such as Deep Belief Network (DBN) [1] and Deep Boltzmann Machine (DBM) [2], they have achieved the great success in many machine learning tasks [9][14][25], including object recognition [1][2], speech perception [3], information retrieva”
A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data (2014, unknown): “As we’ll see, the deep version of our DocNADE model will outperform the DBM approach of Srivastava and Salakhutdinov [13].”
Deep Temporal Sigmoid Belief Networks for Sequence Modeling (2015, methodology): “The conditional distributions, p(v|h) and p(h|v), in the RBM are factorial, which makes inference fast, while parameter estimation usually relies on an approximation technique known as Contrastive Divergence (CD) [18].”
Topically Driven Neural Language Model (2017, background): “, 2003) have been proposed, including recent work on neural topic models (Cao et al., 2015; Wan et al., 2012; Larochelle and Lauly, 2012; Hinton and Salakhutdinov, 2009).”
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification (2016, methodology): “It also significantly outperforms four other methods, including supervised Latent Dirichlet allocation (LDA) [54], DocNADE [36], Replicated Softmax Model (RSM) [28] and Overreplicated Softmax Model (OSM) [65].”