Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

By Tong Che, Ruixiang Zhang, Jascha Narain Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, Yoshua Bengio. 2020

127 citations recorded by the source corpus.

We show that the sum of the implicit generator log-density $\log p_g$ of a GAN with the logit score of the discriminator defines an energy function which yields the true data density when the generator is imperfect but the discriminator is optimal, thus making it possible to improve on the typical generator (with implicit density $p_g$). To make that practical, we show that sampling from this modified density can be achieved by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score. This can be achieved by running a Langevin MCMC in latent space and then applying the generator function, which we call Discriminator Driven Latent Sampling~(DDLS). We show that DDLS is highly efficient compared to previous methods which work in the high-dimensional pixel space and can be applied to improve on previously trained GANs of many types. We evaluate DDLS on both synthetic and real-world datasets qualitatively and quantitatively. On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN~\citep{sngan} from $8.22$ to $9.09$ which is even comparable to the class-conditional BigGAN~\citep{biggan} model. This achieves a new state-of-the-art in unconditional image synthesis setting without introducing extra parameters or additional training.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, neutral): “Our work also bears a resemblance to recent methods on improving the sampling quality in generative adversarial networks using gradient flows in the latent space [65, 66, 67, 68], with the main difference that these prior works use a discriminator to update the latent variables, whereas we train an ”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “This dataset is also studied in Che et al. (2020).”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, methodology): “Similar to [6], if we do rejection sampling with the proposal distribution p g ( x ) and the acceptance probability p g ( x ) , we get samples x from p θ ( x | c ) .”
Exponential Tilting of Generative Models: Improving Sample Quality by Training and Sampling from Latent Energy (2020, methodology): “In particular, [5] sample latent variables from an energy-based model (EBM) deﬁned jointly by the generator and discriminator.”
Instance Selection for GANs (2020, background): “…of current techniques attempt to eliminate low quality samples after the model is trained, either by changing the model distribution by truncating the latent space [2, 10] or by performing some form of rejection sampling using a trained discriminator to inform the rejection process [1, 4, 26].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Energy-based Models (EBMs) [26, 48, 58, 64, 67, 68, 75, 78, 79, 120, 129, 132, 165, 170, 182, 196, 254, 281] can be viewed as one generative version of discriminators [79, 104, 131, 134], while can be learned from unlabeled input data.”
Refining Deep Generative Models via Discriminator Gradient Flow (2020, methodology): “, 2019), or require a sensitive diffusion term to ensure sample diversity (Che et al., 2020).”
Generalized Energy Based Models (2020, background): “The closest related approach appears in a study concurrent to the present work (Che et al., 2020), where the authors propose to use Langevin dynamics on the latent space of a GAN generator, but with a different discriminator to ours (derived from the Jensen-Shannon divergence or a Wasserstein-based ”
Energy-Based Open-World Uncertainty Modeling for Confidence Calibration (2021, methodology): “Energy-Based Models (EBMs) have been widely used in the area of generative modeling [41, 8, 1, 4].”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “…issue, a separate line of work utilizes an auxiliary network to amortize the negative portions of the sampling procedure (Kim & Bengio, 2016; Kumar et al., 2019; Han et al., 2019; Xie et al., 2018a; Song & Ou, 2018; Dai et al., 2019; Che et al., 2020; Grathwohl et al., 2020a; Arbel et al., 2020).”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, background): “In spirit similar are recent works (Azadi et al., 2018; Turner et al., 2019; Che et al., 2020) that link GANs, deﬁned in the pixels space, to EBMs.”
Training GANs with Stronger Augmentations via Contrastive Discriminator (2021, methodology): “We adopt a modiﬁed version of SNDCGAN architecture by Kurach et al. (2019) following the experimental setups of other baselines (Zhang et al., 2020; Zhao et al., 2020c).”