Comparison of Maximum Likelihood and GAN-based training of Real NVPs

By Ivo Danihelka, Balaji Lakshminarayanan, Benigno Uria, Daan Wierstra, Peter S. Dayan. 2017

55 citations recorded by the source corpus.

We train a generator by maximum likelihood and we also train the same generator architecture by Wasserstein GAN. We then compare the generated samples, exact log-probability densities and approximate Wasserstein distances. We show that an independent critic trained to approximate Wasserstein distance between the validation set and the generator distribution helps detect overfitting. Finally, we use ideas from the one-shot learning literature to develop a novel fast learning critic.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, background): “Lossless compression metrics for GANs often give absurdly large bits-per-dimension (e.g. 10 14 ) which fails to reﬂect the true performance of the model (Grover et al., 2018; Danihelka et al., 2017).”
Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step (2017, methodology): “We use the Inception Score [22] to measure how visually appealing CIFAR-10 samples are, the MS-SSIM metric [18, 24] to check sample diversity, and an Improved Wasserstein independent critic to assess overfitting, as well as sample quality [4].”
Demystifying MMD GANs (2018, background): “Likelihoods do not necessarily correspond well to sample quality (Theis et al., 2016), and GAN-type objectives focus much more on producing plausible samples, as illustrated particularly directly by Danihelka et al. (2017).”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “If we choose the generator of a generative adversarial network as a normalizing flow, we can train the flow parameters using adversarial training (Grover et al., 2018; Danihelka et al., 2017), with Wasserstein losses (Arjovsky et al.”
The Cramer Distance as a Solution to Biased Wasserstein Gradients (2017, methodology): “To compare performance between the two architectures, we measured the loss computed by an independent WGAN-GP critic trained on the validation set, following a similar evaluation previously done by Danihelka et al. (2017).”
Variational Approaches for Auto-Encoding Generative Adversarial Networks (2017, methodology): “[5] proposed training an independent Wasserstein GAN critic to distinguish between held out validation data and generated samples.”
Is Generator Conditioning Causally Related to GAN Performance? (2018, methodology): “Invertible Density Estimators and Adversarial Training: The recent work most closely related to ours is a pair of papers (Grover et al., 2017; Danihelka et al., 2017) wherein the authors compare adversarial training to maximum likelihood training of generative image models using an invertible decode”
Distribution Matching in Variational Inference (2018, background): “…2003), an image similarity metric ranging between 0.0 (low similarity) and 1.0 (high similarity) that has been shown to correlate well with human judgment. et al., 2018), and if the independent critic is trained on validation data, it can also be used to measure overﬁtting (Danihelka et al., 2017).”
Transferring GANs: generating images from limited data (2018, methodology): “Independent Wasserstein (IW) critic [7] This metric uses an independent critic D̂ only for evaluation.”
Analyzing Inverse Problems with Invertible Neural Networks (2018, methodology): “Despite their simple design and training, these networks were rarely studied: [5, 15] showed that minimization of an adversarial loss is superior to maximum likelihood training in RealNVPs, [13] used a NICE-like design as a memory-efficient alternative to residual networks, [19] demonstrated that th”
Parametric Adversarial Divergences are Good Task Losses for Generative Modeling (2017, methodology): “Danihelka et al. (2017) compare parametric adversarial divergence and likelihood objectives in the special case of RealNVP, a generator with explicit density, and obtain better visual results with the adversarial divergence.”
Quantitatively Evaluating GANs With Divergences Proposed for Training (2018, methodology): “(10) Note that IW (Danihelka et al., 2017) and MMD (Sutherland et al.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “To address this issue, an independent Wasserstein critic [218] is proposed to be trained independently for the validation dataset to measure mode collapse and overfitting.”
PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows (2019, background): “It has also been explored as a metric for evaluating GANs [50].”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, background): “[18] suggest an independent Wasserstein critic which is trained independently for the validation dataset to measure over-fitting and mode collapse.”
Guided Image Generation with Conditional Invertible Neural Networks (2019, background): “For instance, [5] found minimization of an adversarial loss to be superior to maximum likelihood training in RealNVPs, [36] trained i-2 RevNets in the same manner as adversarial auto-encoders, i.e. with a discriminator acting in latent rather than data space, and Flow-GANs [11] performed best using…”
Fair Generative Modeling via Weak Supervision (2019, background): “, 2016; Mohamed & Lakshminarayanan, 2016; Grover & Ermon, 2018), evaluation metrics based on two-sample tests (Gretton et al., 2007; Bowman et al., 2015; Lopez-Paz & Oquab, 2016; Danihelka et al., 2017; Rosca et al., 2017; Im et al., 2018; Gulrajani et al., 2018), or improved Monte Carlo inference v”
GILBO: One Metric to Measure Them All (2018, methodology): “…like estimated marginal log likelihoods (often reported for VAE s and very expensive to estimate for GAN s), an independent Wasser-stein critic (Danihelka et al., 2017), or the common INCEPTION (Gao et al., 2017) and FID (Heusel et al., 2017) scores which attempt to measure how well the…”