Robust Wasserstein profile inference and applications to machine learning

By Jose H. Blanchet, Yang Kang, Karthyek Murthy. 2016

385 citations recorded by the source corpus.

We show that several machine learning estimators, including square-root least absolute shrinkage and selection and regularized logistic regression, can be represented as solutions to distributionally robust optimization problems. The associated uncertainty regions are based on suitably defined Wasserstein distances. Hence, our representations allow us to view regularization as a result of introducing an artificial adversary that perturbs the empirical distribution to account for out-of-sample effects in loss estimation. In addition, we introduce RWPI (robust Wasserstein profile inference), a novel inference methodology which extends the use of methods inspired by empirical likelihood to the setting of optimal transport costs (of which Wasserstein distances are a particular case). We use RWPI to show how to optimally select the size of uncertainty regions, and as a consequence we are able to choose regularization parameters for these machine learning estimators without the use of cross validation. Numerical experiments are also given to validate our theoretical findings.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Certifying Some Distributional Robustness with Principled Adversarial Training (2017, neutral): “, 2016; Namkoong & Duchi, 2016), and Wasserstein distances (Esfahani & Kuhn, 2015; Shafieezadeh-Abadeh et al., 2015; Blanchet et al., 2016).”
Data-Driven Optimal Transport Cost Selection For Distributionally Robust Optimization (2017, methodology): “Then, use such data-driven c(·) in the definition of Dc(P, Pn) and the construction Uδ(Pn) in (3).”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “DRO methods optimize machine learning systems to be robust to adversarial data distributions, thus optimizing for worst case performance against distribution shift [21, 5, 42, 16, 47, 15, 7].”
Certifiable Distributional Robustness with Principled Adversarial Training (2017, background): “Previous approaches to distributional robustness have considered finite-dimensional parametrizations for P, such as constraint sets for moments, support, or directional deviations [12, 15, 19], as well as non-parametric distances for probability measures such as f -divergences [5, 6, 30, 27, 16, 32]”
Does Distributionally Robust Supervised Learning Give Robust Classifiers? (2016, background): “DRSL by Esfahani & Kuhn (2015), Blanchet et al. (2016) and Sinha et al. (2017) lets the uncertainty set of test distributions be a Wasserstein ball from the training distribution.”
Distributionally Robust Stochastic Optimization with Wasserstein Distance (2016, background): “[10, 12] that such choice would lead to a N−1/2-bound asymptotically under proper conditions.”
Deep Learning in Asset Pricing (2019, background): “12 (12)See Blanchet et al. (2016) for a discussion on robust estimation with an adversarial approach.”
Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning (2019, background): “Extensions to other classification models are discussed in [10, 37, 96].”
Learning Models with Uniform Performance via Distributionally Robust Optimization (2018, methodology): “An alternative to our f -divergence based sets {Q : Df (Q||P0) ≤ ρ} are Wasserstein balls [99, 72, 100, 81, 18, 17, 41, 38, 87, 65].”
Regularization via Mass Transportation (2017, background): “[8] show, however, that radii of the order O(N−1/2) are asymptotically optimal even though the corresponding Wasserstein balls are too small to contain the true distribution with constant confidence.”
Beyond Gaussian approximation: Bootstrap for maxima of sums of independent random vectors (2017, background): “These examples cover many practical problems and applications as discussed in Chernozhukov et al. (2013, 2017), and many publications citing their work (Dezeure et al., 2017; Ning and Liu, 2017; Zhang and Wu, 2017b; Chen et al., 2018; Blanchet et al., 2019; Horowitz, 2019).”
Lipschitz Networks and Distributional Robustness (2018, background): “…the uncertainty set is a transportation cost-or Wasserstein-ball containing µ (deﬁned formally in § 2), under certain technical conditions, (1) has the dual form (Blanchet et al., 2016, 2017; Gao and Kleywegt, 2016; Shaﬁeezadeh-Abadeh et al., 2017) risk ℓ ( f ; P ) = risk ℓ ( f ; µ ) + L ( f ) .”
Quantifying Distributional Model Risk via Optimal Transport (2016, neutral): “Other applications of Theorem 1, broadly in the context of distributionally robust optimization, are available in Esfahani and Kuhn (2015); Zhao and Guan (2015), Gao and Kleywegt (2016) and Blanchet et al. (2016), and as well in Example 4 in Section 6.2 of this paper.”
Wasserstein Distributional Robustness and Regularization in Statistical Learning (2017, background): “Indeed, such an equivalence has been established in Esfahani and Kuhn (2017) for piecewise-linear convex loss, Shaﬁeezadeh-Abadeh et al. (2015) for logistic regression, and in Blanchet et al. (2016) for linear regression and support vector machines.”