Wasserstein Distributional Robustness and Regularization in Statistical Learning

By Rui Gao, Xi Chen, Anton J. Kleywegt. 2017

129 citations recorded by the source corpus.

A central question in statistical learning is to design algorithms that not only perform well on training data, but also generalize to new and unseen data. In this paper, we tackle this question by formulating a distributionally robust stochastic optimization (DRSO) problem, which seeks a solution that minimizes the worst-case expected loss over a family of distributions that are close to the empirical distribution in Wasserstein distances. We establish a connection between such Wasserstein DRSO and regularization. More precisely, we identify a broad class of loss functions, for which the Wasserstein DRSO is asymptotically equivalent to a regularization problem with a gradient-norm penalty. Such relation provides new interpretations for problems involving regularization, including a great number of statistical learning problems and discrete choice models (e.g. multinomial logit). The connection suggests a principled way to regularize high-dimensional, non-convex problems. This is demonstrated through the training of Wasserstein generative adversarial networks in deep learning.

Subfields: Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

Citations in the corpus

Certifying Some Distributional Robustness with Principled Adversarial Training (2017, background): “[25], Volpi et al.”
Conditional variance penalties and domain shift robustness (2017, background): “…to be of the form F = F (cid:15) ( P 0 ) with with (cid:15) > 0 a small constant and D ( Q, P 0 ) being, for example, a φ -divergence (Namkoong and Duchi, 2017; Ben-Tal et al., 2013; Bagnell, 2005) or a Wasserstein-distance (Shaﬁeezadeh-Abadeh et al., 2017; Sinha et al., 2017; Gao et al., 2017).”
Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning (2019, background): “For further details on distributionally robust regression see [10, 37, 96].”