Learning with a Wasserstein Loss

By Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya-Polo, Tomaso A. Poggio. Neural Information Processing Systems, 2015

679 citations recorded by the source corpus.

Learning to predict multi-label outputs is challenging, but in many problems there is a natural metric on the outputs that can be used to improve predictions. In this paper we develop a loss function for multi-label learning, based on the Wasserstein distance. The Wasserstein distance provides a natural notion of dissimilarity for probability measures. Although optimizing with respect to the exact Wasserstein distance is costly, recent work has described a regularized approximation that is efficiently computed. We describe an efficient learning algorithm based on this regularization, as well as a novel extension of the Wasserstein distance from probability measures to unnormalized measures. We also describe a statistical learning bound for the loss. The Wasserstein loss can encourage smoothness of the predictions with respect to a chosen metric on the output space. We demonstrate this property on a real-data tag prediction problem, using the Yahoo Flickr Creative Commons dataset, outperforming a baseline that doesn't use the metric.

Subfields: Computational Theory and Mathematics, Genetics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/ae23eda2faddd824480de2d90638795f797cc66e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Genetics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Sample Complexity of Sinkhorn Divergences (2018, background): “Among the many learning tasks performed with this regularization, one may cite domain adaptation (Courty et al., 2014), text retrieval (Kus-ner et al., 2015) or multi-label classiﬁcation (Frogner et al., 2015).”
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling (2016, background): “…(RNNLM) are based on the conventional classiﬁcation framework, which has two major drawbacks: First, there is no assumed metric on the output classes, whereas there is evidence suggesting that learning is improved when one can deﬁne a natural metric on the output space (Frogner et al., 2015).”
Wasserstein Barycenter Model Ensembling (2019, methodology): “Wasserstein distances were originally defined between normalized probability vectors (Balanced OT) (Villani, 2008; Peyré & Cuturi, 2017), but they have been extended to deal with unnormalized measures and this problem is referred to as unbalanced OT (Chizat et al., 2018; Frogner et al., 2015).”
Unbalanced optimal transport: Dynamic and Kantorovich formulations (2015, background): “Quite interestingly, the same idea is used in an informal and heuristic way by [10] for applications in machine learning, where soft marginal constraints is the key to stabilize numerical results.”
Robust Wasserstein profile inference and applications to machine learning (2016, background): “More recently, optimal transport metrics and Wasserstein distances are being actively investigated for its use in various machine learning applications (see [41, 35, 18, 48] and references therein for a growing list of new applications).”
Regularization via Mass Transportation (2017, methodology): “[33] propose a learning algorithm based on the Wasserstein distance to predict multi-label outputs.”
Gromov-Wasserstein Averaging of Kernel and Distance Matrices (2016, background): “, 2014), designing loss functions (Frogner et al., 2015), low-rank approximation (Seguy & Cuturi, 2015), and dictionary learning (Rolet et al.”
Soft-DTW: a Differentiable Loss Function for Time-Series (2017, methodology): “ observation, in the same way that, for instance, a regularized Wasserstein distance was used to compute barycenters (Cuturi & Doucet,2014), and later to ﬁt discriminators that output histograms (Zhang et al., 2015;Rolet et al.,2016). When paired with a ﬂexible learning architecture such as a ne”
Sliced Wasserstein Distance for Learning Gaussian Mixture Models (2017, methodology): “It has been shown that optimizing with respect to the Wasserstein loss has various practical benefits over the KL-divergence loss [43, 15, 38, 3, 19].”
Stochastic Optimization for Large-scale Optimal Transport (2016, methodology): “As a set of representative examples, let us quote: bag-of-visual-words comparison in computer vision [17], color and shape processing in computer graphics [21], bag-of-words for natural language processing [11] and multi-label classification [9].”
Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation (2019, background): “Although various optimization approaches have been proposed in the past [11, 16], it is unclear how we can directly optimize W1(p1, p2) in an end-to-end trainable fashion efficiently.”
Interpolating between Optimal Transport and MMD using Sinkhorn Divergences (2018, methodology): “With applications ranging from shape matching (Vaillant and Glaunès, 2005; Kaltenmark et al., 2017) to classiﬁcation (Frogner et al., 2015) and generative model training (Goodfellow et al., 2014), a common setting is that of measure ﬁtting : given a unit-mass, positive empirical distribution β ∈ M…”
Wasserstein Training of Restricted Boltzmann Machines (2016, methodology): “That distance was for instance introduced in the context of supervised inference by [8], who used it to compute a predictive loss between the output of a multilabel classifier against its ground truth, or for domain adaptation, by [4]. high low high low small overlap small distance large distance…”
Artificial intelligence in the creative industries: a review (2020, background): “Wasserstein loss is designed to prevent this [321,322].”
Learning Generative Models with Sinkhorn Divergences (2017, methodology): “It is however important to realize that for large scale and high dimensional learning applications, empirical considerations [9, 22, 13] suggest that, unlike relevant applications of the same scheme in graphics [33], a relatively strong regularization—a large ε—leads not only to more stable results ”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Among them, some works [7, 40, 50, 92] proposed improved versions of the adversarial loss to stabilize GAN training and convergence, whereas others introduced additional losses [10, 58, 75, 83] for specific task learning.”
Earth Mover’s Distance Minimization for Unsupervised Bilingual Lexicon Induction (2017, background): “Apart from the usage as a regularizer (Zhang et al., 2016a), the EMD can also play other roles in optimization programs designed for various applications (Cuturi and Doucet, 2014; Frogner et al., 2015; Montavon et al., 2016).”