Distributionally Robust Logistic Regression

By Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani, D. Kühn. 2015

353 citations recorded by the source corpus.

This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distribution-ally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Certifying Some Distributional Robustness with Principled Adversarial Training (2017, neutral): “, 2016; Namkoong & Duchi, 2016), and Wasserstein distances (Esfahani & Kuhn, 2015; Shafieezadeh-Abadeh et al., 2015; Blanchet et al., 2016).”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, background): “With small divergence balls of radii O (1 /n ) , DRO acts as a regularizer (Shaﬁeezadeh-Abadeh et al., 2015; Namkoong & Duchi, 2017).”
On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems (2019, background): “[2009], Abadeh et al. [2015], online learning Cesa-Bianchi and Lugosi [2006], deep learning Sinha et al.”
Robust Wasserstein profile inference and applications to machine learning (2016, methodology): “For an application of these concentration inequalities to choose the size of uncertainty set in the context of distributionally robust logistic regression and data-driven DRO, refer [42, 28].”
Regularization via Mass Transportation (2017, methodology): “n distance to predict multi-label outputs. Distributionally robust optimization models with Wasserstein ambiguity sets were introduced to the realm of supervised learning by Shaeezadeh-Abadeh et al. [61], who show that distributionally robust logistic regression problems admit a tractable reformulat”
It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks (2021, background): “…to measurement error from econometrics [116], classiﬁcation-specialized approaches modifying the (surrogate) loss function [117], or agnostic robustness/sensitivity analysis (sensitivity analysis relative to adversarial perturbationos, or e.g. adapting robust logistic regression such as [118]).”
Distributionally Robust Optimization: A Review (2019, background): “[273] and study distributionally robust supervised learning (regression and classification) models.”
Learning Models with Uniform Performance via Distributionally Robust Optimization (2018, methodology): “This power, however, means that tractable reformulations are only available under restrictive scenarios [81, 38, 87], and they remain computationally challenging.”
Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning (2019, background): “observed variability comparisons is to estimate and remove externally forced climate signals from the observations, and then compare the residual variability with control run internal variability (4).”
Evaluating Machine Accuracy on ImageNet (2020, background): “…goal of distributionally robust optimization (DRO) is to ﬁnd models that minimize the worst case expected error over a set of probability distributions (Abadeh et al., 2015; Ben-Tal et al., 2013; Delage & Ye, 2010; Duchi et al., 2019; Esfahani & Kuhn, 2018; Sagawa et al., 2019; Sinha et al., 2017).”
Distributionally Robust Neural Networks (2020, background): “With small divergence balls of radii O(1/n), DRO acts as a regularizer (Shafieezadeh-Abadeh et al., 2015; Duchi & Namkoong, 2016).”
Wasserstein Distributional Robustness and Regularization in Statistical Learning (2017, result): “15 in Shafieezadeh-Abadeh et al. (2017). The result in Theorem 1 can be easily generalized to the following case, whose proof is similar to that of Theorem 1 and thus omitted.”
Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach (2016, methodology): “e, to allow easier computation), as well as specifyinga desired conﬁdence level for appropriate robustness. The recent line of work on distributionally robust optimization using Wasserstein distances [53, 70, 28, 59, 12] is similar in spirit to the formulation considered here. Unlike f-divergences, ”
Minimax Statistical Learning and Domain Adaptation with Wasserstein Distances (2017, methodology): “Following [5, 6, 7], we let the ambiguity set A(P ) be the p-Wasserstein ball of radius ε ≥ 0 centered at P : A(P ) = B ε,p(P ) := { Q ∈ Pp(Z) : Wp(P,Q) ≤ ε } ,”
Towards Out-Of-Distribution Generalization: A Survey (2021, methodology): “For Wasserstein-based DRO methods, Shaﬁeezadeh-Abadeh et al. [125] investigate the Wasserstein DRO of logistic regression, and show that the regularized logistic regression is one special case of it.”
Reweighted Data for Robust Probabilistic Models (2016, background): “The first is a rich body of work on robust statistics and machine learning (Provost and Fawcett, 2001; Song et al., 2002; Yu et al., 2012; McWilliams et al., 2014; Feng et al., 2014; Shafieezadeh-Abadeh et al., 2015).”