Generative Modeling Using the Sliced Wasserstein Distance

By Ishan Deshpande, Ziyu Zhang, Alexander Gerhard Schwing. 2018

260 citations recorded by the source corpus.

Generative Adversarial Nets (GANs) are very successful at modeling distributions from given samples, even in the high-dimensional case. However, their formulation is also known to be hard to optimize and often not stable. While this is particularly true for early GAN formulations, there has been significant empirically motivated and theoretically founded progress to improve stability, for instance, by using the Wasserstein distance rather than the Jenson-Shannon divergence. Here, we consider an alternative formulation for generative modeling based on random projections which, in its simplest form, results in a single objective rather than a saddle-point formulation. By augmenting this approach with a discriminator we improve its accuracy. We found our approach to be significantly more stable compared to even the improved Wasserstein GAN. Further, unlike the traditional GAN loss, the loss formulated in our method is a good measure of the actual distance between the distributions and, for the first time for GAN training, we are able to show estimates for the same.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Wasserstein Fair Classification (2019, background): “Adversarial methods maximize the system ability to predict Y while minimizing the ability to predict A (Zhang et al., 2018).”
Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation (2019, background): “The Wasserstein distance has also recently raised interest in stabilizing generative modeling [1, 14, 73], learning introspective neural networks [32], and obtaining Gaussian mixture models [29] thanks to its geometrically meaningful distance measure even when the supports of the distributions do no”
Latent Translation: Crossing Modalities by Bridging Generative Models (2019, methodology): “For computational efficiency and inspired by (Deshpande et al., 2018), we use SWD, or Sliced Wasserstein Distance (Bonneel et al., 2015) between S′1 and S ′ 2 as a loss term to encourage cross-domain overlapping in shared latent space.”
Subspace Robust Wasserstein distances (2019, background): “, 2015) but also to train generative models (Kolouri et al., 2018; Deshpande et al., 2018).”
Diffusion-GAN: Training GANs with Diffusion (2022, background): “, 2017; 2018), changing the objective functions (Arjovsky et al., 2017; Bellemare et al., 2017; Deshpande et al., 2018; Li et al., 2017a; Nowozin et al., 2016; Zheng and Zhou, 2021; Yang et al., 2021), regularizing the weights and/or gradients (Arjovsky et al.”
Generalized Sliced Wasserstein Distances (2019, methodology): “Therefore, it has recently attracted ample attention and successfully been applied to a variety of practical tasks [22, 24, 25, 26, 27, 28, 29, 30, 31].”
Orthogonal Estimation of Wasserstein Distances (2019, methodology): “Recently, Wu et al. (2017); Deshpande et al. (2018) propose to take D ( · , · ) to be the sliced Wasserstein distance, so as to bypass the potential instability due to min-max optimization formulation with GAN.”
Learning Transferable Knowledge Through Embedding Spaces (2019, methodology): “Similar to the previous chapter, we use the Sliced Wasserstein Distance (SWD) to measure and minimize the distance between the probability distributions.”
Potential Flow Generator With L2 Optimal Transport Regularity for Generative Models (2019, methodology): “1 ) 0.990 ± 0.002 0 .503 ± 0.003 0 .025 ± 0.008 As for the GAN loss for generators we use the sliced Wasserstein distance 1 , due to its relatively low computational cost, robustness, and clear mathematical interpretation in low dimensional problems Deshpande et al. [2018].”