f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization

By Sebastian Nowozin, Botond Cseke, Ryota Tomioka. 2016

1,831 citations recorded by the source corpus.

Generative neural samplers are probabilistic models that implement sampling using feedforward neural networks: they take a random input vector and produce a sample from a probability distribution defined by the network weights. These models are expressive and allow efficient computation of samples and derivatives, but cannot be used for computing likelihoods or for marginalization. The generative-adversarial training method allows to train such models through the use of an auxiliary discriminative neural network. We show that the generative-adversarial approach is a special case of an existing more general variational divergence estimation approach. We show that any f-divergence can be used for training generative neural samplers. We discuss the benefits of various choices of divergence functions on training complexity and the quality of the obtained generative models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “The former uses log-likelihood (or a suitable surrogate) as the training objective, while the latter uses adversarial training to minimize f -divergences [40] or integral probability metrics [2, 55] between model and data distributions.”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “Much recent research has accordingly focused on modiﬁcations to the vanilla GAN procedure to impart stability, drawing on a growing body of empirical and theoretical insights (Nowozin et al., 2016; Sønderby et al., 2017; Fedus et al., 2018).”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “Note that we do not modify the generator in any way, and our work is orthogonal to the ongoing research on architectural changes of the generator [20, 27], divergence measures [25, 1, 37], and regularizations [40, 15, 34].”
Improved Training of Wasserstein GANs (2017, background): “Despite this, consistently stable training of GANs remains an open problem.”
Unrolled Generative Adversarial Networks (2016, background): “minimizes the Jensen-Shannon divergence between the data distribution and the generator, and extensions generalize this to a wider class of divergences (Nowozin et al., 2016; Sonderby et al., 2016; Poole et al., 2016).”
On Mutual Information Maximization for Representation Learning (2019, background): “Their proof starts from the NWJ lower bound of the KL divergence, namely that for any function ˜ f the following lower bound holds (Nguyen et al., 2010; Nowozin et al., 2016): x,y ) ] .”
Learning deep representations by mutual information estimation and maximization (2018, background): “For example, one could deﬁne a Jensen-Shannon MI estimator (following the formulation of Nowozin et al., 2016), where x is an input sample, x (cid:48) is an input sampled from ˜ P = P , and sp ( z ) = log(1+ e z ) is the softplus function.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Nowozin et al. (2016) proposed a variational expression of f -divergences as a loss function for GANs.”
The Numerics of GANs (2017, methodology): “Most divergences that are used in practice can be represented in the following form [10, 16, 4]:”
Wasserstein GAN (2017, background): “GANs oﬀer much more ﬂexibility in the deﬁnition of the objective function, including Jensen-Shannon [4], and all f -divergences [17] as well as some exotic combinations [6].”
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, methodology): “Nowozin et al. (Nowozin et al., 2016) proposed to generalize Generative Adversarial Networks (Goodfellow et al., 2014) to f-divergences (Ali & Silvey, 1966) based on results by Nguyen et al. (Nguyen et al., 2010).”
Which Training Methods for GANs do actually Converge? (2018, background): “The two-player GAN game can be interpreted as minimizing a probabilistic divergence between the true data distribution and the distribution produced by the generator (Nowozin et al., 2016; Goodfellow et al., 2014).”
Least Squares Generative Adversarial Networks (2016, background): “Many works have been proposed to address this problem by analyzing the objective functions of GANs [2, 3, 19, 23, 24].”
cGANs with Projection Discriminator (2018, background): “The most distinctive feature of GANs is the discriminator D ( x ) that evaluates the divergence between the current generative distribution p G ( x ) and the target distribution q ( x ) (Goodfellow et al., 2014; Nowozin et al., 2016; Arjovsky et al., 2017).”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, neutral): “See for example Nowozin et al. (2016).”
Demystifying MMD GANs (2018, background): “This result has been extended to general f -divergences by Nowozin et al. (2016).”
Amortised MAP Inference for Image Super-resolution (2016, background): “Recently, Nowozin et al. (2016) presented a more general treatment that connects GANs to f -divergence minimisation.”