Training Restricted Boltzmann Machines on Word Observations

By George E. Dahl, Ryan P. Adams, Hugo Larochelle. 2012

96 citations recorded by the source corpus.

The restricted Boltzmann machine (RBM) is a flexible model for complex data. However, using RBMs for high-dimensional multinomial observations poses significant computational difficulties. In natural language processing applications, words are naturally modeled by K-ary discrete distributions, where K is determined by the vocabulary size and can easily be in the hundred thousands. The conventional approach to training RBMs on word observations is limited because it requires sampling the states of K-way softmax visible units during block Gibbs updates, an operation that takes time linear in K. In this work, we address this issue with a more general class of Markov chain Monte Carlo operators on the visible units, yielding updates with computational complexity independent of K. We demonstrate the success of our approach by training RBMs on hundreds of millions of word n-grams using larger vocabularies than previously feasible with RBMs and by using the learned features to improve performance on chunking and sentiment classification tasks, achieving state-of-the-art results on the latter.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Distributed Representations of Sentences and Documents (2014, background): “…2011) 32.58% Full+BoW (Maas et al., 2011) 11.67% Full+Unlabeled+BoW (Maas et al., 2011) 11.11% WRRBM (Dahl et al., 2012) 12.58% WRRBM + BoW (bnc) (Dahl et al., 2012) 10.77% MNB-uni (Wang & Manning, 2012) 16.45% MNB-bi (Wang & Manning, 2012) 13.41% SVM-uni (Wang & Manning, 2012) 13.05% SVM-bi…”
Building Large-Scale Twitter-Specific Sentiment Lexicon : A Representation Learning Approach (2014, background): “…learning algorithms have been extensively studied in recent years (Bengio et al., 2013), and are dominated by the syntactic context based algorithms (Bengio et al., 2003; Collobert et al., 2011; Dahl et al., 2012; Huang et al., 2012; Mikolov et al., 2013a; Lebret et al., 2013; Sun et al., 2014).”
Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs (2013, methodology): “SRM is related to previous work (Dahl et al., 2012) on applying RBMs to high-dimensional sparse inputs, more precisely multinomial observations, e.g., one K-ary multinomial for each word in an n-gram window.”
Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network (2014, background): “The ﬁrst block is from Maas et al. [16], the second from Dahl et al. [3], the third from Wang and Manning [24] and the fourth from Le and Mikolov [15].”
Contextual Bidirectional Long Short-Term Memory Recurrent Neural Network Language Models: A Generative Approach to Sentiment Analysis (2017, methodology): “More advanced approaches utilize word or n -gram vectors, like in (Maas et al., 2011; Dahl et al., 2012).”
Learning to Spot and Refactor Inconsistent Method Names (2019, methodology): “This technique was proposed to overcome the limitations of bag-of-words [35] features which are known to (1) lose the ordering of the words and (2) ignore the word semantics.”
Contextual Explanation Networks (2017, unknown): “67 Dahl et al. (2012) WRRBM + BoW (bnc) 10.”
Baselines and Bigrams: Simple, Good Sentiment and Topic Classification (2012, methodology): “WR-RBM : Word Representation Restricted Boltzmann Machine (Dahl et al., 2012). ones that use additional data.”
Bayesian Optimization of Text Representations (2015, methodology): “5 It outperforms SVMs and feed-forward neural networks, the restricted Boltzmann machine approach presented by Dahl et al. (2012), Congressional vote (Table 7).”
Neural Networks and Deep Learning (2018, unknown): “The challenges associated with this type of modeling are discussed in [86].”
Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy (2016, methodology): “After the RBMs are stacked to form the deep generative models, such as Deep Belief Network (DBN) [1] and Deep Boltzmann Machine (DBM) [2], they have achieved the great success in many machine learning tasks [9][14][25], including object recognition [1][2], speech perception [3], information retrieva”