How to Train Your Energy-Based Models

By Yang Song, Diederik P. Kingma. 2021

343 citations recorded by the source corpus.

Energy-Based Models (EBMs), also known as non-normalized probabilistic models, specify probability density or mass functions up to an unknown normalizing constant. Unlike most other probabilistic models, EBMs do not place a restriction on the tractability of the normalizing constant, thus are more flexible to parameterize and can model a more expressive family of probability distributions. However, the unknown normalizing constant of EBMs makes training particularly difficult. Our goal is to provide a friendly introduction to modern approaches for EBM training. We start by explaining maximum likelihood training with Markov chain Monte Carlo (MCMC), and proceed to elaborate on MCMC-free approaches, including Score Matching (SM) and Noise Constrastive Estimation (NCE). We highlight theoretical connections among these three approaches, and end with a brief survey on alternative training methods, which are still under active research. Our tutorial is targeted at an audience with basic understanding of generative models who want to apply EBMs or start a research project in this direction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Implicit Behavioral Cloning (2021, methodology): “To demonstrate a breadth of approaches, we present results with three 49 different EBM training and inference methods discussed below, however a comprehensive comparison of all 50 EBM variants is outside the scope of this paper; see [19] for a comprehensive reference.”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, methodology): “We particularly note that other sampling steps such as MCMC corrector steps or alternative predictor steps (e.g., following the reverse SDE) [Song et al., 2021] can also be incorporated into computation budget, and general learning frameworks like reinforcement learning are well-suited to explore…”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “Score matching for energy-based models Besides the connection to diffusion models, score matching is also often used as a method for learning energy based models (EBM)— see Song & Kingma (2021) for a comprehensive review on useful techniques—.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “Recent advances in training non-normalized probabilistic models, such as score-based generative models [52, 53, 55, 23, 51, 28] and energy-based models [3, 15, 14, 62, 63, 54], have achieved comparable image sample quality as GANs.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Adversarial purification [18, 147, 206, 213, 234] denotes a class of defense methods that eliminate adversarial perturbations using a generative model [147].”
An approximate sampler for energy-based models with divergence diagnostics (2022, background): “This opens a range of optimized training techniques (see the survey by Song and Kingma 2021), including Langevin and Hamiltonian dynamics (Parisi, 1981; Duane et al., 1987), where the local Markov chain moves are informed by ∇ x log P ( x ).”
Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, background): “EBMs perform well in learning high dimensional distributions at the cost of being difficult to train [174] and have been employed in forecasting [150].”
State-of-the-Art Estimation of Protein Model Accuracy Using AlphaFold (2022, unknown): “In theory, it should often be possible to predict protein structures without using MSAs, since protein structures are fully determined by their amino acid sequences [5].”
Pre-training Molecular Graph Representation with 3D Geometry (2021, unknown): “Recently, there have been quite a lot progress along this direction [19, 32, 80, 81].”
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, background): “Another resembling perspective links energy-based models to this class of generative methods [28], [29].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, unknown): “[175], generative adversarial networks [70], [249], and energy-based models [202], however, naturally these dive into the intricacies of their respective method rather than”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, background): “For LD, the grid search is performed with N ∈ [50 , 100 , 200 , 300 , 400 , 500 , 600 , 1000] , η ∈ [0 .”
Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting (2021, methodology): “EBMs have been shown to perform well in learning high dimensional data distributions at the cost of being difﬁcult to train (Song & Kingma, 2021).”
MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “Each method has been widely explored, and for more details, we refer to this comprehensive summary [102].”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “…& Ermon, 2019; 2020; Song et al., 2021; Ho et al., 2020; Song et al., 2020; Jolicoeur-Martineau et al., 2021) and energy-based models (Ackley et al., 1985; Gao et al., 2017; Du & Mordatch, 2019; Xie et al., 2018; 2016; Song & Kingma, 2021), have achieved comparable image sample quality as GANs.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “An example of this approach, demonstrated in recent research [37], reframes BC as an energy-based modeling problem [38].”
Understanding Diffusion Models: A Unified Perspective (2022, background): “To begin to understand why optimizing a score function makes sense, we take a detour and revisit energybased models [15, 16].”