Modeling Documents with Deep Boltzmann Machines

By Nitish Srivastava, Ruslan Salakhutdinov, Geoffrey E. Hinton. 2013

189 citations recorded by the source corpus.

We introduce a type of Deep Boltzmann Machine (DBM) that is suitable for extracting distributed semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This enables an efficient pretraining algorithm and a state initialization scheme for fast inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Distributed Representations of Sentences and Documents (2014, methodology): “In this direction, autoencoder-style models have also been used to model paragraphs (Maas et al., 2011; Larochelle & Lauly, 2012; Srivastava et al., 2013).”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, background): “, 2013), as well as distributed representations of longer bodies of text such as paragraphs and documents (Srivastava et al., 2013; Le and Mikolov, 2014).”
Neural Variational Inference for Text Processing (2015, methodology): “A primary feature of NVDM is that each word is generated directly from a dense continuous document representation instead of the more common binary semantic vector (Hinton & Salakhut-dinov, 2009; Larochelle & Lauly, 2012; Srivastava et al., 2013; Mnih & Gregor, 2014).”
Deep Exponential Families (2014, background): “These networks differ from the undirected deep probabilistic models [30, 37] in that they allow for explaining away, where latent variables compete to explain the observations.”
Neural Networks and Deep Learning (2018, unknown): “Another approach advocated in [469] is to create the value of k in CDk by 1 after every 10,000 steps.”
Improving Topic Models with Latent Feature Word Representations (2015, methodology): “tion about word similarity, such as thesauri and dictionaries, to smooth the topic-to-word distribution. Topic models have also been constructed using latent features (Salakhutdinov and Hinton, 2009; Srivastava et al., 2013; Cao et al., 2015). Latent feature (LF) vectors have been used for a wide ra”
WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling (2018, methodology): “This type of hierarchical visual representation is difficult to achieve with other types of deep neural networks (Srivastava et al., 2013; Kingma & Welling, 2014; Rezende et al., 2014; Sonderby et al., 2016).”
A Survey on Context Learning (2017, background): “The Over-Replicated Softmax is an RBMbased languagemodel with two hidden layers [64].”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “Dirichlet allocation (LDA) (Mcauliffe and Blei, 2008), DocNADE (Larochelle and Lauly, 2012), Replicated Softmax Model (RSM) (Hinton and Salakhutdinov, 2009) and Overreplicated Softmax Model (OSM) (Srivastava et al., 2013).”