The Cramer Distance as a Solution to Biased Wasserstein Gradients

By Marc G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, R´emi Munos. 2017

397 citations recorded by the source corpus.

The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes. The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling, and most recently in reinforcement learning. In this paper we describe three natural properties of probability divergences that we believe reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients. The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third. We provide empirical evidence suggesting this is a serious issue in practice. Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cramer distance. We show that the Cramer distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences. We give empirical results on a number of domains comparing these three divergences. To illustrate the practical relevance of the Cramer distance we design a new algorithm, the Cramer Generative Adversarial Network (GAN), and show that it has a number of desirable properties over the related Wasserstein GAN.

Subfields: Applied Mathematics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Generative Models

Citations in the corpus

Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, background): “One line of work is focused on changing the objective function (Arjovsky et al., 2017; Mao et al., 2016; Lim & Ye, 2017; Bellemare et al., 2017; Salimans et al., 2018) to encourage convergence.”
Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “Meanwhile, WGAN-GP has been seen to produce deterministic completions (Bellemare et al., 2017).”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “For additional hyperparameter details, see Table 1 and (Bellemare et al., 2017a).”
Demystifying MMD GANs (2018, methodology): “(2015), with numerous recent follow-up works (Sutherland et al., 2017; Liu, 2017; Li et al., 2017b; Bellemare et al., 2017).”
Improving GANs Using Optimal Transport (2018, methodology): “A complication in using primal form optimal transport is that it may give biased gradients when used with mini-batches (see Bellemare et al., 2017) and may therefore be inconsistent as a technique for statistical estimation.”
Distributional Reinforcement Learning with Quantile Regression (2017, methodology): “Theorem 1 (Theorem 1, Bellemare et al. 2017) .”
A Distributional Perspective on Reinforcement Learning (2017, neutral): “We note that, while these algorithms appear unrelated to the Wasserstein metric, recent work (Bellemare et al., 2017) hints at a deeper connection.”
MMD GAN: Towards Deeper Understanding of Moment Matching Network (2017, methodology): “A parallel work [30] use energy distance, which can be treated as MMD GAN with different kernel.”
Learning Generative Models with Sinkhorn Divergences (2017, methodology): “It was also used to fit generative models in [3], while [24] uses MMD with a gaussian kernel.”
Is Generator Conditioning Causally Related to GAN Performance? (2018, methodology): “Loosely following Bellemare et al. (2017), we jointly vary the number of discriminator steps and whether Jacobian Clamping is applied.”
Learning Implicit Generative Models with the Method of Learned Moments (2018, background): “, 2015), Cramér (Bellemare et al., 2017) – in pixel space has led to poor sample quality, indirect minimization using adversarial training (Goodfellow et al.”
A Probabilistic U-Net for Segmentation of Ambiguous Images (2018, methodology): “To do so, we use the generalized energy distance [26, 27, 28], which leverages distances between observations:”
CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE) (2018, background): “It has been also observed that the recent literature [94] mentions that the Wasserstein distance may yield the biased sample gradients, is subject to the risk of incorrect minimum, and not well suitable for stochastic gradient descent searching.”
Improving GAN with neighbors embedding and gradient matching (2018, neutral): “The discriminator can be formed via the f-divergence (Nowozin, Cseke, and Tomioka 2016), or distance metrics (Arjovsky and Bottou 2017; Bellemare et al. 2017).”
Distributional Deep Reinforcement Learning with a Mixture of Gaussians (2019, methodology): “The latter is proven in the following theorem in a similar way with [10].”
Leveraging Frequency Analysis for Deep Fake Image Recognition (2020, methodology): “, 2016), CramerGAN (Bellemare et al., 2017), CycleGAN (Zhu et al.”
Learning deep kernels for exponential family densities (2018, background): “These types of kernels have seen success in supervised learning (Wilson et al. 2016; Jean et al. 2018) and critic functions for implicit generative models (Li et al. 2017; Bellemare et al. 2017; Bi´nkowski et al. 2018; Arbel et al. 2018).”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, background): “[35]: Standard GAN, [73]: LSGAN, [138]: EBGAN, [6]: WGAN, [97]: Improved WGAN, [80]: McGAN, [10]: CramerGAN, [65]: MMDGAN, [79]: FisherGAN, [76]: Unrolled GAN, [14]: MRGAN, [59]: DRAGAN, [77]: Conditional GAN, [16]: InfoGAN, [106]: ss-InfoGAN, [27]: ALI, [25]: BiGAN, [116]: AGE, [58]: ARAE, [61]: VA”
Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints (2018, methodology): “For the classification task, we configure training and testing sets with 5 classes: {real, ProGAN, SNGAN, CramerGAN, MMDGAN}.”