Implicit Generation and Generalization in Energy-Based Models

By Yilun Du, Igor Mordatch. 2019

42 citations recorded by the source corpus.

Energy based models (EBMs) are appealing due to their generality and simplicity in likelihood modeling, but have been traditionally difficult to train. We present techniques to scale MCMC based EBM training on continuous neural networks, and we show its success on the high-dimensional data domains of ImageNet32x32, ImageNet128x128, CIFAR-10, and robotic hand trajectories, achieving better samples than other likelihood models and nearing the performance of contemporary GAN approaches, while covering all modes of the data. We highlight some unique capabilities of implicit generation such as compositionality and corrupt image reconstruction and inpainting. Finally, we show that EBMs are useful models across a wide variety of tasks, achieving state-of-the-art out-of-distribution classification, adversarially robust classification, state-of-the-art continual online class learning, and coherent long term predicted trajectory rollouts.

Subfields: Artificial Intelligence, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

Citations in the corpus

Residual Energy-Based Models for Text Generation (2020, methodology): “In maximum likelihood training negatives are generated from the model, but in text application we cannot use gradient-based MCMC methods (Teh et al., 2003; Du & Mordatch, 2019) and Gibbs sampling (Welling et al., 2005) is too slow to be practical.”
Flow Contrastive Estimation of Energy-Based Models (2019, neutral): “[55, 9] scale up samplingbased methods to large image datasets with white noise as the starting point of sampling.”
Learning Gradient Fields for Shape Generation (2020, neutral): “For inference, we set T = 10 and α = 2 × 10 − 4 .”
Real or Fake? Learning to Discriminate Machine from Human Generated Text (2019, methodology): “EBMs have been mostly applied to vision applications [Teh et al., 2003, Du and Mordatch, 2019] where search is easier thanks to the continuous nature of the input space.”
Energy-based models for atomic-resolution protein conformations (2020, methodology): “This is motivated by their simplicity and scalability, as well as their compelling results in other domains, such as image generation (Du & Mordatch, 2019).”
Energy-Based Models for Text (2020, methodology): “Unfortunately, gradient-based MCMC approaches like Hybrid Monte Carlo (Teh et al., 2003) and Langevyn dynamics (Ranzato et al., 2007; Du and Mordatch, 2019; Xie et al., 2016, 2017, 2019, 2018; Gao et al., 2018; Nijkamp et al., 2019) are not applicable when the input is discrete like in text…”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, methodology): “3 Sampling through an ODE solver Sampling with LD requires hand-crafted tricks to speed up and stabilize its convergence [14, 18].”
Energy-Based Imitation Learning (2020, methodology): “More specifically, MaxEnt IRL models the paths in demonstrations using a Boltzmann distribution, where the energy is given by the unknown reward function r̂∗: pr̂(τ) = 1 Z exp (r̂∗(τ)) , (14) where τ = {s0, a0, · · · , sT , aT } is a trajectory of state-action pairs and T is its length; r̂∗(τ) =∑ t ”
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, background): “Another resembling perspective links energy-based models to this class of generative methods [28], [29].”
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model (2020, background): “The CIFAR-10 training for ALICE, SVAE follow their optimal networks and hyperparameters and scores for EBM are taken directly from [8].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “Energy Based Models IGEBM [41] ????? ????? ????? ????? ????? ????? 37.”