Demystifying MMD GANs

By Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, Arthur Gretton. 2018

2,101 citations recorded by the source corpus.

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, methodology): “Despite sharing the known drawbacks of FID (Bikowski et al., 2018), FVD has shown to correlate much stronger with human raters compared to both PSNR and SSIM (Unterthiner et al.”
Training Generative Adversarial Networks with Limited Data (2020, background): “We note that FID has a general tendency to increase as the training set gets smaller — not only because of the lower image quality, but also due to inherent bias in FID itself [2].”
Projected GANs Converge Faster (2021, methodology): “On all datasets, KID, SwaV-FID, CLIP-FID, VirTex-FID, and SWD mirror the ranking obtained via FID.”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “We focus our experiments on the widely adopted FID metric, and show additional results on the KID metric [5] as well as IS [64] and Perceptual Path Length (PPL) metrics [37] (in the supplement).”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “The time complexity of training a linear SVM is between O(n2) and O(n3) (Bottou & Lin, 2007), compared to O(nd2 + d3) of FID (Heusel et al., 2017) and O(n2d) of KID (Bińkowski et al., 2018), where n is the sampling size and d is the dimension of feature space.”
Improved Precision and Recall Metric for Assessing Generative Models (2019, methodology): “(KID) [2], group these two aspects to a single value without a clear tradeoff.”
Diagnosing and Enhancing VAE Models (2019, methodology): “For example, although the FID score is a widely-adopted measure that signiﬁcantly improves upon the earlier Inception Score (Salimans et al., 2016), it has been shown to exhibit bias in certain circumstances (Bi´nkowski et al., 2018).”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Another reason to consider other metrics is computational; the Wasserstein has high computational cost and suffers from biased stochastic gradient estimates (Bikowski et al., 2018; Bellemare et al., 2017b), so minimizing a simpler metric, such as the KL, may be more convenient.”
Assessing Generative Models via Precision and Recall (2018, methodology): “[4] recently introduced an unbiased alternative to FID, the Kernel Inception Distance.”
Classification Accuracy Score for Conditional Generative Models (2019, methodology): “[24] proposed KID, an unbiased alternative to FID, but the variance of this metric is too large to be reliable when using the number of per-class samples in the ImageNet training set (roughly 1,000 per class), and is worse when using the 50 in the validation set.”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, methodology): “To address this problem, the Kernel Inception Distance (KID) was proposed by Bińkowski et al. [35].”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, methodology): “We emphasize that for fairness in comparisons, we used the same number of real and fake samples when computing FID, since FID can produce highly bias estimates at different sample sizes [7].”
High-Fidelity Generative Image Compression (2020, methodology): “Metrics We evaluate our models and the baselines in PSNR as well as the perceptual distortions LPIPS [57] and MS-SSIM, and we use NIQE [35], FID [18], and KID [8] as perceptual quality indices.”
The GAN Landscape: Losses, Architectures, Regularization, and Normalization (2018, methodology): “The KID metric introduced by Bi´nkowski et al. (2018) is an alternative to FID.”
Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, background): “These metrics are not without problems however [2, 5], and it is unclear if they correlate well with human perception when used to evaluate non-adversarial generative models.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “…of a generative adversarial network as a normalizing flow, we can train the flow parameters using adversarial training (Grover et al., 2018; Danihelka et al., 2017), with Wasserstein losses (Arjovsky et al., 2017), using maximum mean discrepancy (Bińkowski et al., 2018), or other approaches.”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, background): “We note that there is a family of GANs whose discriminators take pairs of images [65, 37, 5], albeit for very different reasons.”