Training Products of Experts by Minimizing Contrastive Divergence

By Geoffrey E. Hinton. 2002

5,632 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Media Technology, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Media Technology

In Probabilistic & Bayesian Methods

Citations in the corpus

Improving neural networks by preventing co-adaptation of feature detectors (2012, background): “Assuming the dropout networks do not all make identical predictions, the prediction of the mean network is guaranteed to assign a higher log probability to the correct answer than the mean of the log probabilities assigned by the individual dropout networks [8].”
Pylearn2: a machine learning research library (2013, methodology): “Other costs implemented in the library include dropout [27] , contrastive divergence [26], noise contrastive estimation [24], score matching [28], denoising s core matching [52], softmax log likelihood for classification with MLPs, and Gaussian log likelihood fo r regression with MLPs.”
Neural Autoregressive Distribution Estimation (2016, methodology): “The proposal of Contrastive Divergence (CD; Hinton, 2002) was instrumental in the popularization of the RBM.”
Residual Energy-Based Models for Text Generation (2020, methodology): “Other approaches like Gibbs sampling (Hinton, 2002) were applied to binary inputs but do not scale well to large dictionaries once the energy function is a large bidirectional transformer model like the one used in this work.”
What can the brain teach us about building artificial intelligence? (2016, background): “rule (Bienenstock, Cooper, & Munro, 1982), backpropagation (Rumelhart, Hinton, & Williams, 1986), the wake-sleep algorithm (Hinton, Dayan, Frey, & Neal, 1995), and contrastive divergence (Hinton, 2002). Whether the goal is supervised or unsupervised learning, these algorithms implement l”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, methodology): “Samples are typically generated from p ( x ) by an MCMC algorithm.”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “For example, Restricted Boltzmann Machines (Hinton, 2002) were trained using a block Gibbs sampler to approximate the expectation in Eq.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “A prominent approximation of MLE is the contrastive divergence (CD) [25] framework, requiring MCMC initialized from the data distribution.”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “Different choices of energy function recover well-known probabilistic models including Markov random ﬁelds [35], (restricted) Boltzmann machines [62, 24, 29], and conditional random ﬁelds [40].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “, the evidence lower bound used in variational auto-encoders [29], contrastive divergence in energy-based models [21]) for training.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, background): “For short-run MCMC, contrastive divergence (CD) [18] is the most prominent framework for theoretical underpinning.”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “In this section, we review the central principles of the MCMC-based ML learning used in previous studies such as [13, 43, 38].”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, methodology): “This paper studies the problem of learning energy-based generative ConvNet models [24, 10, 12, 11, 39, 25, 35, 28, 45, 46, 15] of images.”
A Theory of Generative ConvNet (2016, methodology): “pproximated by re-lu max(0;r), and the bias term b l;kthat is determined by and Z(l) k (w;x). Finally, taking the product of P (l) k (I;w;x) over kand xgives rise to a Product of Experts (PoE) model (Hinton, 2002), which is the generative ConvNet (7) using ﬁlters at layer l. See Lu et al. (2016) for”
GSNs : Generative Stochastic Networks (2015, methodology): “Algorithms for training Boltzmann machines (especially the Restricted Boltzmann Machine or RBM) such as contrastive divergence (Hinton, 2000; Hinton et al., 2006) and persistent contrastive divergence (Tieleman, 2008; Younes, 1998) directly aim at estimating the gradient of the log-likelihood using…”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “…but do not remove, this tradeoff – for instance mean ﬁeld theory and its expansions [39, 40], variational Bayes [19], contrastive divergence [44, 13], minimum probability ﬂow [36, 35], minimum KL contraction [28], proper scoring rules [9, 32], score matching [15], pseudolikelihood [5], loopy…”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Latent variable generative models Most deep generative models explicitly define a latent representation, except for some energy-based models [41, 31] and autoregressive models [94, 93, 10].”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “Energy-based models are another class of likelihood-based models with a rich history [1, 13, 30].”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “Early variants of EBMs include models whose energy is defined over both data and auxiliary latent variables (Salakhutdinov & Hinton, 2009; Hinton, 2012), and models using only data variables (Hinton, 2002; Mnih & Hinton, 2005).”