On Contrastive Divergence Learning

By M. A. Carreira-Perpiñán, Geoffrey E. Hinton. 2005

863 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, methodology): “In literature, there are different kinds of DL models: Deep Multilayer Perceptron (DMLP), RNN, LSTM, CNN, Restricted Boltzmann Machines (RBMs), DBN, Autoencoder (AE), and DRL [37, 38].”
A Bayesian Data Augmentation Approach for Learning Deep Models (2017, methodology): “A similar issue was observed in contrastive divergence [2], where the computation of the approximate gradient required in theory an inﬁnite number of Markov chain Monte Carlo (MCMC) cycles, but in practice, it was noted that only a few cycles were needed to provide a robust gradient approximation.”
Deep Learning for Health Informatics (2017, methodology): “Contrastive divergence [40] (CD) algorithm is a common method used to train an RBM.”
Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models (2017, methodology): “Hence, they are usually learned by approximating the likelihood using methods such as contrastive divergence (Carreira-Perpinan and Hinton 2005) and pseudolikelihood (Besag 1977).”
Bilateral Denoising Diffusion Models (2021, neutral): “The successful methods can be mainly divided into two branches – generative adversarial network (GAN) [7] based models and likelihood-based models [8–21].”
Energy-based models for atomic-resolution protein conformations (2020, methodology): “This corresponds to maximizing the log-likelihood of the data under the model: Following Carreira-Perpinan & Hinton (2005), the gradient of this objective can be written as: Intuitively, this gradient decreases the energy of samples from the data distribution x + and increases the energy of samples ”
Deep Reconstruction Models for Image Set Classification (2015, background): “Later (in Section 3.3) this template will be separately fine-tuned for all classes of the training image sets.”
Integrative Data Analysis of Multi-Platform Cancer Data with a Multimodal Deep Learning Approach (2015, methodology): “Basically, the KullbackLeibler divergence is computed over the empirical distribution function of the visible data and the model [13], [14].”
The rate adapting poisson model for information retrieval and object recognition (2006, methodology): “Although truncating the Markov chain will introduce a bias in the estimates of the gradients, the final bias of the parameter estimates has been shown to be small empirically for a number of applications ( Carreira-Perpinan & Hinton, 2005 ).”
Parallel tempering is efficient for learning restricted Boltzmann machines (2010, background): “Due to the previously mentioned characteristics, it is expected thate samples collected during the negative phase would explain the model distribution better, and that the learning process would be done well even with a smaller number of samples than those required if Gibbs sampling is used.”
Restricted Boltzmann machines for collaborative filtering (2007, background): “By increasing T to a sucien tly large value, it is possible to approximate maximum likelihood learning arbitrarily well ( Carreira-Perpinan & Hinton, 2005 ), but large values of T are seldom needed in practice.”
Why adaptively collected data have negative bias and how to correct for it (2017, methodology): “We solve the cMLE optimization problem using contrastive divergence [4].”
Taming the Curse of Dimensionality: Discrete Integration by Hashing and Optimization (2013, methodology): “We train Restricted Boltzmann Machines (RBM) [15] using Contrastive Divergence (CD) [5, 33] on MNIST hand-written digits dataset.”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, background): “Much has been written about the properties and alternative interpretations of CD, e.g. Carreira-Perpi˜nan and Hinton (2005); Yuille (2005); Bengio and Delalleau (2009); Sutskever and Tieleman (2010).”
Regularized estimation of image statistics by Score Matching (2010, background): “SM and NCE are consistent estimators [10, 6], while CD estimation has been shown to be generally asymptotically biased [4].”
Improving Traffic Flow Prediction With Weather Information in Connected Cars: A Deep Learning Approach (2016, methodology): “To address this problem, we use the contrastive divergence algorithm proposed in [34].”