Energy-based Generative Adversarial Network

By Junbo Jake Zhao, Michaël Mathieu, Yann LeCun. 2016

1,148 citations recorded by the source corpus.

We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discriminator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the discriminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “The connection between GANs and EBMs has been studied by many authors [45–48].”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”
Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, background): “, 2016b), absolute deviation with margin (Zhao et al., 2017), and Wasserstein distance (Arjovsky et al.”
Projected GANs Converge Faster (2021, background): “Notable exceptions are [55,70] which utilize an encoder-decoder discriminator architecture.”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “Zhao et al. (2017); Che et al. (2020); Song et al. (2020); Arbel et al. (2020) formulate EBMs with GANs, and use the discriminator to assign an energy.”
Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “Fortunately, this is exactly what is done by the recently proposed Generative Adversarial Networks (GANs) [22, 12, 41, 49, 59].”
Unrolled Generative Adversarial Networks (2016, methodology): “…the GAN framework has also enabled a number of successful extensions of the technique, for instance for structured prediction (Reed et al., 2016a;b; Odena et al., 2016), training energy based models (Zhao et al., 2016), and combining the GAN loss with a mutual information loss (Chen et al., 2016).”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “GANs enable a wide variety of applications such as image generation [1, 41, 60], representation learning [44], image manipulation [62], object detection [32], and video applications [37, 50, 52].”
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, background): “Generative adversarial networks (GANs) [3] have shown remarkable results in various computer vision tasks such as image generation [1, 6, 23, 31], image translation [7, 8, 32], super-resolution imaging [13], and face image synthesis [9, 15, 25, 30].”
Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields (2017, neutral): “Energy-Based GANs [41] and their later improvement BEGANs [4] optimize an energy landscape based on auto-encoders.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “ions, and show that our method outperforms these baselines. We provide bothPyTorchandTorchimplementations. Check out more results at ourwebsite. 2. Related work Generative Adversarial Networks (GANs) [16,63] have achieved impressive results in image generation [6, 39], image editing [66], and repres”
Wasserstein GAN (2017, background): “• Energy-based GANs (EBGANs) [25] can be thought of as the generative approach to the total variation distance.”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “…generally fall into two categories: methods that use Markov chain Monte Carlo (MCMC) based sampling (Sallans & Hinton, 2004), and methods that learn a stochastic sampling network trained to output approximate samples from the target distribution (Zhao et al., 2016; Kim & Bengio, 2016).”
Self-Attention Generative Adversarial Networks (2018, methodology): “, 2018; 2019), modifying the learning objectives and dynamics (Arjovsky et al., 2017; Salimans et al., 2018; Metz et al., 2017; Che et al., 2017; Zhao et al., 2017; Jolicoeur-Martineau, 2019), adding regularization methods (Gulrajani et al.”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “An energy-based GAN [38] has also been proposed for more stable training behavior.”
Which Training Methods for GANs do actually Converge? (2018, background): “For the ﬁrst class of stable equilibria, which we call energy solutions , the equilibrium discriminator forms an energy function for the true data distributions and might be a partial explanation for the success of autoencoder-based discrim-inators (Zhao et al., 2016; Berthelot et al., 2017).”
Least Squares Generative Adversarial Networks (2016, methodology): “Viewing the discriminator as an energy function, [33] used an auto-encoder architecture to improve the stability of GANs learning.”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, methodology): “[56], a DCGAN model [32] is trained for quantitative comparison using the public available source code on the ImageNet Dog dataset.”