Regularization via Mass Transportation

By Soroosh Shafieezadeh-Abadeh, D. Kühn, Peyman Mohajerin Esfahani. 2017

245 citations recorded by the source corpus.

The goal of regression and classification methods in supervised learning is to minimize the empirical risk, that is, the expectation of some loss function quantifying the prediction error under the empirical distribution. When facing scarce training data, overfitting is typically mitigated by adding regularization terms to the objective that penalize hypothesis complexity. In this paper we introduce new regularization techniques using ideas from distributionally robust optimization, and we give new probabilistic interpretations to existing techniques. Specifically, we propose to minimize the worst-case expected loss, where the worst case is taken over the ball of all (continuous or discrete) distributions that have a bounded transportation distance from the (discrete) empirical distribution. By choosing the radius of this ball judiciously, we can guarantee that the worst-case expected loss provides an upper confidence bound on the loss on test data, thus offering new generalization bounds. We prove that the resulting regularized learning problems are tractable and can be tractably kernelized for many popular loss functions. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Conditional variance penalties and domain shift robustness (2017, neutral): “, 2018) or a Wasserstein distance (Shafieezadeh-Abadeh et al., 2017; Sinha et al., 2018; Gao et al., 2017).”
Lipschitz Networks and Distributional Robustness (2018, background): “…the uncertainty set is a transportation cost-or Wasserstein-ball containing µ (deﬁned formally in § 2), under certain technical conditions, (1) has the dual form (Blanchet et al., 2016, 2017; Gao and Kleywegt, 2016; Shaﬁeezadeh-Abadeh et al., 2017) risk ℓ ( f ; P ) = risk ℓ ( f ; µ ) + L ( f ) .”
Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning (2019, background): “Evaluating Model Noise Estimates Model estimates of internal climate variability are a key component of detection and attribution (D&A) studies (5, 35).”
Data-Enabled Predictive Control: In the Shallows of the DeePC (2018, background): “We conjecture that this regularization is related to distributionally robust optimization problems, in which similar regularization terms arise [19].”
Towards Out-Of-Distribution Generalization: A Survey (2021, background): “…[13], [118], [119], [120], [121], [122], [123], [124](Section 4.3) ( Section 5 ) ( Section 5.1 ) [1], [125], [126], [127], [128], [129], [130], [131], [132], [133], [134](Section 5.1.1 ∼ 5.1.3) [135], [136], [137], [138] (Section 5.1.4) Invariance-BasedOptimization ( Section 5.2 ) [5], [139],…”
Minimax Classification under Concept Drift with Multidimensional Adaptation and Performance Guarantees (2022, methodology): “Other robust risk minimization (RRM) methods utilize uncertainty sets determined by metrics such as Wasserstein distances (Shafieezadeh-Abadeh et al., 2019) and fdivergences (Namkoong & Duchi, 2017).”