Learning Generative Models with Sinkhorn Divergences

By Aude Genevay, G. Peyr'e, Marco Cuturi. 2017

725 citations recorded by the source corpus.

The ability to compare two degenerate probability distributions (i.e. two probability distributions supported on two distinct low-dimensional manifolds living in a much higher-dimensional space) is a crucial problem arising in the estimation of generative models for high-dimensional observations such as those arising in computer vision or natural language. It is known that optimal transport metrics can represent a cure for this problem, since they were specifically designed as an alternative to information divergences to handle such problematic scenarios. Unfortunately, training generative machines using OT raises formidable computational and statistical challenges, because of (i) the computational burden of evaluating OT losses, (ii) the instability and lack of smoothness of these losses, (iii) the difficulty to estimate robustly these losses and their gradients in high dimension. This paper presents the first tractable computational method to train large scale generative models using an optimal transport loss, and tackles these three issues by relying on two key ideas: (a) entropic smoothing, which turns the original OT loss into one that can be computed using Sinkhorn fixed point iterations; (b) algorithmic (automatic) differentiation of these iterations. These two approximations result in a robust and differentiable approximation of the OT loss with streamlined GPU execution. Entropic smoothing generates a family of losses interpolating between Wasserstein (OT) and Maximum Mean Discrepancy (MMD), thus allowing to find a sweet spot leveraging the geometry of OT and the favorable high-dimensional sample complexity of MMD which comes with unbiased gradient estimates. The resulting computational architecture complements nicely standard deep network generative models by a stack of extra layers implementing the loss function.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAIT: A Geometric Approach to Information Theory (2019, methodology): “…(Goodfellow et al., 2014; Arjovsky et al., 2017; Salimans et al., 2018); and (ii) the development of alternative formulations and relaxations of the original optimal transport objective which make it feasible to approximately compute in practice (Cuturi and Doucet, 2014; Genevay et al., 2018).”
Demystifying MMD GANs (2018, background): “ GAN training. Bellemare et al. (2017) pointed out a particular ﬂaw of the Wasserstein distance as an objective: that gradients of both the duality-based estimator and of the alternative estimator of Genevay et al. (2017) are biased. Optimization over biased gradients can lead to incorrect parameter”
Improving GANs Using Optimal Transport (2018, background): “In concurrent work, Genevay et al. (2018) independently propose a very similar loss function to (8), but using a single sample from the data and generator distributions.”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, background): “The exact calculation of the OT loss is generally difficult [25].”
Sorting out Lipschitz function approximation (2018, background): “Wasserstein Distance Estimation Wasserstein-1 distance (also called Earth Mover Distance) is a principled distance metric between two probability distributions and has found many applications in machine learning in recent years [38, 17].”
Neural Network Encapsulation (2018, background): “both φ, ψ, instead of the practice in [15,19,14] via an adversarial manner.”
Wasserstein Barycenter Model Ensembling (2019, background): “More recently a new angle on generative adversarial networks learning with the Wasserstein distance was introduced in (Arjovsky et al., 2017; Genevay et al., 2017; Salimans et al., 2018).”
Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints (2020, background): “Exact minimization over U in the above problem is in general computational intractable (Genevay et al., 2018).”
Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training (2019, methodology): “(4) over T is intractable in general [1, 49, 21, 12].”
Revisiting Reverse Distillation for Anomaly Detection (2023, methodology): “To achieve this goal, we propose using the de-biased Sinkhorn divergence, a variant of the Optimal Transport distance [7,10,11,26,32].”
Prescribed Generative Adversarial Networks (2019, methodology): “…and optimization methods (see, e.g., Li et al., 2015; Dziugaite et al., 2015; Nowozin et al., 2016; Arjovsky et al., 2017; Ravuri et al., 2018; Genevay et al., 2017) or using ideas from VAE s (Makhzani et al., 2015; Mescheder et al., 2017; Dumoulin et al., 2016; Donahue et al., 2016;…”
Interpolating between Optimal Transport and MMD using Sinkhorn Divergences (2018, methodology): “…from computer graphics (Bonneel et al., 2016) to supervised learning (Frogner et al., 2015), unsupervised density ﬁtting (Bassetti et al., 2006) and generative model learning (Montavon et al., 2016; Arjovsky et al., 2017; Salimans et al., 2018; Genevay et al., 2018; Sanjabi et al., 2018).”
Sinkhorn Barycenters with Free Support via Frank-Wolfe Algorithm (2019, methodology): “To compensate for this bias, [25] introduced the Sinkhorn divergence”
Super-efficiency of automatic differentiation for functions defined as a minimum (2020, methodology): “It has also been used to differentiate through the Sinkhorn algorithm in optimal transport applications (Boursier and Perchet, 2019; Genevay et al., 2018).”
Learning to Generate Novel Domains for Domain Generalization (2020, neutral): “ datasets. dened as Wc(;) = inf M2M X i;j [MC]i;j; (10) where the soft-matching matrix M represents the coupling distribution ˇ in Eq. (8) and can be eciently computed using the Sinkhorn algorithm [14]; Cis the pairwise distance matrix computed over two sets of samples. Following [45], we dene the c”