Learning Energy-Based Models by Diffusion Recovery Likelihood

By Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, Diederik P. Kingma. 2020

150 citations recorded by the source corpus.

While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset. Each EBM is trained by maximizing the recovery likelihood: the conditional probability of the data at a certain noise level given their noisy versions at a higher noise level. The recovery likelihood objective is more tractable than the marginal likelihood objective, since it only requires MCMC sampling from a relatively concentrated conditional distribution. Moreover, we show that this estimation method is theoretically consistent: it learns the correct conditional and marginal distributions at each noise level, given sufficient data. After training, synthesized images can be generated efficiently by a sampling process that initializes from a spherical Gaussian distribution and progressively samples the conditional distributions at decreasingly lower noise levels. Our method generates high fidelity samples on various image datasets. On unconditional CIFAR-10 our method achieves FID 9.60 and inception score 8.58, superior to the majority of GANs. Moreover, we demonstrate that unlike previous work on EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us to accurately estimate the normalized density of data even for high-dimensional datasets.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Statistical and Nonlinear Physics

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Variational Diffusion Models (2021, methodology): “Gao et al. [2020] show how diffusion can also be used to efficiently optimize energy-based models (EBMs) towards a close approximation of the log-likelihood objective, resulting in high-fidelity samples even after long MCMC chains. Song and Ermon [2019] first proposed learning generative models thro”
Score-based Generative Modeling in Latent Space (2021, background): “[39] add an adversarial objective, [5] introduce non-Markovian diffusion processes that allow to trade off synthesis speed, quality, and sample diversity, [40] learn a sequence of conditional energy-based models for denoising, [41] distill the iterative sampling process into single shot synthesis, a”
On Fast Sampling of Diffusion Probabilistic Models (2021, background): “Diffusion models are connected with scored-based models (Song and Ermon, 2019, 2020; Song et al., 2020b), and there have been a series of research extending and improving diffusion models (Song et al., 2020b; Gao et al., 2020; Dhariwal and Nichol, 2021; San-Roman et al., 2021; Meng et al., 2021).”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “There have been a number of improvements and alternatives proposed to the diffusion framework, for example generalization to continuous time [28], deterministic sampling [25], adversarial training [11], and others [6].”
Noise Estimation for Generative Diffusion Models (2021, methodology): “An alternative approach trains an energy-based generative model using a diffusion process that is applied to increasingly noisy versions of a dataset [5], also presenting results on CIFAR-10.”
Improved Denoising Diffusion Probabilistic Models (2021, methodology): “Also in parallel, Gao et al. (2020) develops a diffusion model with reverse diffusion steps modeled by an energy-based model.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “While EBMs offer powerful representation ability due to unnormalized likelihoods, they can suffer from high variance training, long training and sampling times, and struggle to support the entire data space.”
Compositional Visual Generation with Composable Diffusion Models (2022, methodology): “The denoising procedure can be seen as parameterizing the gradients of the data distribution [40], connecting diffusion models to EBMs [7, 11, 12, 22, 30].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “EBMs maximize the recovery likelihood because it is more tractable than marginal likelihood, as sampling from the conditional distributions is much easier than sampling from the marginal distributions.”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, methodology): “Among variants of diffusion models, Gao et al. (2021) have the closest connection with our method.”
Energy-Based Open-World Uncertainty Modeling for Confidence Calibration (2021, methodology): “Energy-Based Models (EBMs) have been widely used in the area of generative modeling [41, 8, 1, 4].”
Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling (2021, background): “…performance on several audio and image synthesis tasks; see e.g. Song and Ermon (2019); Cai et al. (2020); Chen et al. (2021a); Kong et al. (2021); Gao et al. (2020); Jolicoeur-Martineau et al. (2021b); Ho et al. (2020); Song and Ermon (2020); Song et al. (2020, 2021); Niu et al. (2020); Durkan…”