A Tutorial on Energy-Based Learning

By Yann LeCun, Sumit Chopra, Raia Hadsell, Aurelio Ranzato, Fu Jie Huang. 2006

1,869 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep Generative Stochastic Networks Trainable by Backprop (2013, background): “In addition, a challenge related to inference is sampling (not just from P (h | x) but also from P (h, x) or P (x)), which like inference is often needed in the inner loop of learning algorithms for probabilistic models with latent variables such as energybased models (LeCun et al., 2006) or Markov ”
Residual Energy-Based Models for Text Generation (2020, methodology): “Energy-based models (EBMs) (Hinton, 2002; LeCun et al., 2006; Ranzato et al., 2007) are a more general framework which potentially address all these issues, as they do not require any local normalization.”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Most work focuses on the maximum likelihood learning of energy-based models (LeCun et al., 2006; Du & Mordatch, 2019; Salakhutdinov & Hinton, 2009).”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “2 ENERGY BASED MODELS Energy based models (LeCun et al., 2006) hinge on the observation that any probability density p(x) for x ∈ R can be expressed as”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “Energy-based models [43] deﬁne p ( x ) in terms of an energy function U ( x ) where ⇡ is a tractable “prior” distribution and Z = ⇡ ( x ) exp( U ( x )) dx is a generally intractable partition function.”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “field model, a Gibbs distribution, or an energy-based model [23], [24] that is in the form of exponential tilting of a reference distribution such as the Gaussian white noise distribution or the uniform distribution.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “The maximum likelihood learning of the energy-based model (EBM) [29, 51, 19, 40, 30, 34, 6, 32, 48, 49, 22, 7, 47] follows what Grenander [15] called “analysis by synthesis” scheme.”
Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, methodology): “Speciﬁcally, we present a novel framework for probabilistic modeling of volumetric shape patterns by combining the merits of energy-based model [14] and volumetric convolutional neural network [16].”
A Theory of Generative ConvNet (2016, methodology): “The hierarchical energy-based models (LeCun et al., 2006) were studied by the pioneering work of (Hinton et al., 2006b) and (Ngiam et al., 2011).”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “, 2016b), and energy-based models (EBMs) (Lecun et al., 2006; Salakhutdinov et al., 2007).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Formally, p(x) is often defined through a non-negative function P (x) (aka an energy-based model or EBM) such that p(x) ∝ P (x).”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “This paradigm consists of two steps: first a target distribution incorporating the desired preferences is defined as an Energy-Based Model (LeCun et al., 2006).”
Neural Text Generation with Unlikelihood Training (2019, background): “Assume there is a probability distribution p ∗ ( x ) over variable-length sequences x ∈ V × · · · × V where V is a ﬁnite set, and x = ( x 1 , . . . , x | x | ) is a sequence composed of tokens x t ∈ V .”
Energy-based Generative Adversarial Network (2016, methodology): “The probabilistic binary discriminator in the original formulation of GAN can be seen as one way among many to deﬁne the contrast function and loss functional, as described in LeCun et al. (2006) for the supervised and weakly supervised settings, and Ranzato et al. (2007) for unsupervised learning.”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, methodology): “Among many other applications, these architectures are well-studied for generic classification and structured prediction tasks (Goodfellow et al., 2013; Stoyanov et al., 2011; Brakel et al., 2013; LeCun et al., 2006; Belanger & McCallum, 2016; Belanger et al., 2017; Amos et al., 2017); in vision for”
Energy-based Latent Aligner for Incremental Learning (2022, methodology): “Energy-based Models: EBMs [26] are a type of maximum likelihood estimation models that can assign low energies to observed data-label pairs and high energies otherwise [11].”