Maximum Entropy Generators for Energy-Based Models

By Rithesh Kumar, Anirudh Goyal, Aaron C. Courville, Yoshua Bengio. 2019

119 citations recorded by the source corpus.

Maximum likelihood estimation of energy-based models is a challenging problem due to the intractability of the log-likelihood gradient. In this work, we propose learning both the energy function and an amortized approximate sampling mechanism using a neural generator network, which provides an efficient approximation of the log-likelihood gradient. The resulting objective requires maximizing entropy of the generated samples, which we perform using recently proposed nonparametric mutual information estimators. Finally, to stabilize the resulting adversarial game, we use a zero-centered gradient penalty derived as a necessary condition from the score matching literature. The proposed technique can generate sharp images with Inception and FID scores competitive with recent GAN techniques, does not suffer from mode collapse, and is competitive with state-of-the-art anomaly detection techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “, 2019), via training another generator network (Kim & Bengio, 2016; Kumar et al., 2019), or via surrogate objectives to maximum likelihood (Hyvärinen, 2005; Gutmann & Hyvrinen, 2010; Sohl-Dickstein et al.”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, background): “Learning a ConvNet potential with the help of a generator network as approximative direct sampler is explored in [18, 5, 36, 37, 11, 20].”
Adversarial score matching and improved sampling for image generation (2020, neutral): “We proposed a hybrid approach between 3-Stacked MNIST Modes (Max 1000) KL DCGAN (Radford et al., 2015b) 99.0 3.40 ALI (Dumoulin et al., 2016) 16.0 5.40 Unrolled GAN (Metz et al., 2016) 48.7 4.32 VEEGAN (Srivastava et al., 2017) 150.0 2.95 PacDCGAN2 (Lin et al., 2017b) 1000.0 0.06 WGAN-GP (Kumar et a”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
LOGAN: Latent Optimisation for Generative Adversarial Networks (2019, methodology): “LOGAN is related to the energy-based formulation of a GAN’s discriminator (Dai et al., 2017; Kumar et al., 2019; Du & Mordatch, 2019), when latent optimisation is viewed as descending the energy function defined by the discriminator.”
Training Deep Energy-Based Models with f-Divergence Minimization (2020, methodology): “…the generative process as MCMC sampling in the latent space together with a decoder that transforms latent codes to samples (Girolami & Calderhead, 2011; Kumar et al., 2019), such that the dimension of the latent space is much smaller than the sample space and computing Hessians is more efﬁcient.”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, methodology): “We repeated, with our model, the three-channel MNIST mode coverage experiment similar to the one in [16].”
Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces (2019, methodology): “, 2017), better mode coverage (Kumar et al., 2019) and flexibility (Du and Mordatch, 2019).”
How to Train Your Energy-Based Models (2021, background): “Kumar et al. (2019) propose to estimate the entropy through its connection to mutual information: H ( q φ ( z )) = I ( g φ ( z ) , z ) , which can be estimated from samples with variational lower bounds (Nguyen et al., 2010; Nowozin et al., 2016; Belghazi et al., 2018).”
Joint Training of Variational Auto-Encoder and Latent Energy-Based Model (2020, methodology): “The generator and energy-based model can be trained separately using maximum likelihood criteria as in [13, 33], and they can also be trained jointly as recently explored by [20, 38, 14, 24].”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, background): “EBMs allow more flexible architectures [9, 11] with simpler compositionality [15, 41] compared to other explicit generative models [46, 16], and have better stability and mode coverage in training [30, 31, 71] compared to implicit generative models [10].”
Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator (2020, methodology): “Besides, turning the discriminator of 267 WGAN into an energy function is closely related to the amortized generation methods in energy-based 268 model (EBM) literature [19, 39, 22] where a separate network is proposed to learn to sample from 269 the partition function in [9].”
GFlowNet Foundations (2021, unknown): “Note that GFlowNets have been designed for generating discrete variable-size compositional structures (like sets or graphs), for both latent and observed variables, whereas GANs, VAEs or normalizing flows start from the point of view of modeling real-valued fixed-size vectors using real-valued fixed”
Prescribed Generative Adversarial Networks (2019, background): “More recently, Kumar et al. (2019) have developed maximum-entropy generators for energy-based models using mutual information as a proxy for entropy.”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “…this issue, a separate line of work utilizes an auxiliary network to amortize the negative portions of the sampling procedure (Kim & Bengio, 2016; Kumar et al., 2019; Han et al., 2019; Xie et al., 2018a; Song & Ou, 2018; Dai et al., 2019; Che et al., 2020; Grathwohl et al., 2020a; Arbel et al.,…”
Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, methodology): “Taking advantage of the approximation capacity of modern deep networks such as convolutional network (ConvNet) [39, 34], recent papers [57, 13, 36, 47, 11] parametrize the energy function by ConvNet.”
Implicit Generation and Modeling with Energy Based Models (2019, methodology): “Kim and Bengio [2016], Zhao et al. [2016], Haarnoja et al. [2017], Kumar et al. [2019] propose learning a separate network to generate samples, which makes these methods closely connected to GANs [Finn et al., 2016], but these methods do not have the advantages of implicit sampling noted in the…”