Unsupervised Discovery of Nonlinear Structure Using Contrastive Backpropagation

By Geoffrey E. Hinton, Simon Osindero, Max Welling, Yee Whye Teh. 2006

155 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Biophysics, Cognitive Neuroscience, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Cognitive Neuroscience

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [29, 51, 19, 40, 30, 34, 6, 32, 48, 49, 22, 7, 47] follows what Grenander [15] called “analysis by synthesis” scheme.”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, background): “The pioneering work (Hinton et al. 2006) studies the hierarchical energy-based model.”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, background): “This paper studies the problem of learning energy-based generative ConvNet models [24, 10, 12, 11, 39, 25, 35, 28, 45, 46, 15] of images.”
A Theory of Generative ConvNet (2016, methodology): “Compared to hierarchical models with explicit binary latent variables such as those based on the Boltzmann machine (Hinton et al., 2006a; Salakhutdinov & Hinton, 2009; Lee et al., 2009), the generative ConvNet is directly derived from the discriminative ConvNet.”
Representation Learning: A Review and New Perspectives (2012, methodology): “Object Recognition The beginnings of deep learning in 2006 have focused on the MNIST digit image classification problem (Hinton et al., 2006; Bengio et al., 2007), breaking the supremacy of SVMs (1.4% error) on this dataset3.”
Generative Modeling of Convolutional Neural Networks (2014, background): “Such models include field of experts [22], product of experts [10], Boltzmann machines [8], model based on neural networks [9], etc.”
Learning FRAME Models Using CNN Filters (2015, background): “See (Hinton et al. 2006; Ngiam et al. 2011) for earlier work along this direction.”
Deep Energy Estimator Networks (2018, background): “…potentials in beating the curse of dimensionality, the estimation of the derivative of the partition function Z ( θ ) plagues the learning, and approximations like the contrastive divergence (Hinton, 2006; Hinton et al., 2006) has yet to overcome the curse of inference for complex distributions.”
Implicit Generation and Generalization in Energy-Based Models (2019, methodology): “…where Z ( θ ) denotes the partition function: Generating samples from this distribution is challenging, with previous work relying on MCMC methods such as random walk or Gibbs sampling (Hinton, 2006) which has long mixing times, especially for high-dimensional complex data such as images.”
On Learning Non-Convergent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [28, 56, 19, 41, 29, 35, 4, 32, 49, 54, 21, 5, 48], or the Gibbs distribution, follows what Grenander [15] called “analysis by synthesis” scheme.”
Deep Reconstruction Models for Image Set Classification (2015, background): “Later (in Section 3.3) this template will be separately fine-tuned for all classes of the training image sets.”
End-to-End Learning for Structured Prediction Energy Networks (2017, methodology): “In contrastive backprop, this is circumvented by performing contrastive divergence training, with Hamiltonian Monte Carlo sampling from the energy surface (Mnih & Hinton, 2005; Hinton et al., 2006; Ngiam et al., 2011).”
Energy-Based Models for Deep Probabilistic Regression (2020, methodology): “Energy-Based Models Our approach is of course also related to the theoretical framework of energy-based models, which often has been employed for machine learning problems in the past [35, 20, 30].”
Learning Deep Energy Models (2011, methodology): “Our work builds on Hinton et al. (2006b), which introduced the contrastive backpropagation algorithm, and also the work of Mnih & Hinton (2005), who first showed how to stack a product of Student-t model.”
Mapping dominant leaf type based on combined Sentinel-1/-2 data – Challenges for mountainous countries (2021, background): “Since being proposed by Hinton et al. (2006), there has been increasing interest in using deep learning in earth observations (Zhu et al., 2017; Rammer and Seidl, 2019; Ma et al., 2019; Hoeser and Kuenzer, 2020), and lately also for classifying tree species (e. g. Fricker et al., 2019; Hartling et…”
Modeling pixel means and covariances using factorized third-order boltzmann machines (2010, methodology): “[11] we approximate a sample from the model by running a dynamical simulation for 20 “leap-frog steps” starting at the current training sample and adjusting the step size to keep the rejection rate at 10%.”
Regularized estimation of image statistics by Score Matching (2010, background): “(ICA) [9], and convolution variants of ICA and RBM’s [21, 19].”
Learning generative models of mid-level structure in natural images (2012, background): “…hopes with respect to the possibility of learning image structure at different levels of complexity are models from the deep learning literature (Hinton et al., 2006b; Bengio, 2009) which emphasizes the power of hierarchical, distributed representations in the context of AI-style tasks such as…”
How Much Position Information Do Convolutional Neural Networks Encode? (2020, background): “A simple idea is to compute losses and pass these backwards to the input space to generate a pattern image that can maximize the activation of a given unit (Hinton et al., 2006; Erhan et al., 2009).”