Learning Models with Uniform Performance via Distributionally Robust Optimization

By John C. Duchi, Hongseok Namkoong. 2018

526 citations recorded by the source corpus.

A common goal in statistics and machine learning is to learn models that can perform well against distributional shifts, such as latent heterogeneous subpopulations, unknown covariate shifts, or unmodeled temporal effects. We develop and analyze a distributionally robust stochastic optimization (DRO) framework that learns a model providing good performance against perturbations to the data-generating distribution. We give a convex formulation for the problem, providing several convergence guarantees. We prove finite-sample minimax upper and lower bounds, showing that distributional robustness sometimes comes at a cost in convergence rates. We give limit theorems for the learned parameters, where we fully specify the limiting distribution so that confidence intervals can be computed. On real tasks including generalizing to unknown subpopulations, fine-grained recognition, and providing good tail performance, the distributionally robust approach often exhibits improved performance.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Safety, Robustness & Interpretability, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

In Safety, Robustness & Interpretability

In Mathematical Optimization

Citations in the corpus

Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, methodology): “For general non-convex DRO problems, two types of stochastic optimization methods have been proposed: (i) stochastic gradient descent (SGD) on the Lagrangian dual of the objective (Duchi & Namkoong, 2018; Hashimoto et al., 2018), and (ii) direct minimax optimization (Namkoong & Duchi, 2016).”
Knowing what you know: valid confidence sets in multiclass and multilabel prediction (2020, methodology): “believe exploring approaches to this—perhaps by distributionally robust optimization [8, 10], perhaps by uniform convergence arguments [1, Sec.”
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (2019, background): “Our objective is more related to distributionally robust optimization (Duchi and Namkoong, 2018; Hu et al., 2018), which does not assume access to target data and optimizes the worst-case performance under unknown, bounded distribution shift.”
Knowing what You Know: valid and validated confidence sets in multiclass and multilabel prediction (2020, background): “Our work, then, is a stepping stone toward more uniform notions of validity, and we believe exploring approaches to this—perhaps by distributionally robust optimization (Delage and Ye, 2010; Duchi and Namkoong, 2018), perhaps by uniform convergence arguments (Barber et al., 2019, Sec.”
Robustness to Spurious Correlations via Human Annotations (2020, methodology): “Although there exist many techniques for efﬁ-cient distributionally robust optimization (Ben-Tal et al., 2013; Namkoong & Duchi, 2016; Duchi & Namkoong, 2018), the UV-DRO objective is challenging to estimate from ﬁnite samples, as the outer supremum depends on the conditional risk E [ (cid:96) ( θ…”
DORO: Distributional and Outlier Robust Optimization (2021, unknown): “When η ∗ ( P ′ ) < − C ρ σ 2 k , by Lemma 9, we have Therefore, we can lower bound Var P ′ [ l (ˆ θ, Z )] as: p and consequently, we have a lower bound for R D χ 2 ,ρ ( θ ; P ′ ) : p On the other hand, by setting the dual variable η = 0 , we have a simple upper bound for R D χ 2 ,ρ ( ˆ θ ; P ) : p p”
Distributionally Robust Language Modeling (2019, methodology): “While an existing DRO framework called the conditional value at risk (CVaR) ensures uniformly good performance across subpopulations (Rockafellar and Uryasev, 2000; Duchi and Namkoong, 2018), we demonstrate that na¨ıvely applying this approach to language modeling fails due to three challenges.”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, methodology): “Another assumption is the availability of clean and protected attributes at test time, which is a re-occurring challenge for the development of fairness methods when working in healthcare applications that limit access to protected health information 108–115 .”
Class-Weighted Classification: Trade-offs and Robust Approaches (2020, methodology): “Distributionally robust optimization generally and CVaR specifically have also previously been used in machine learning to deal with imbalance (Duchi et al., 2018; Duchi and Namkoong, 2018), but in these works, the imbalance was considered to exist in the covariates, whether known to the algorithm…”
Counterfactual Sensitivity and Robustness (2019, background): “, Shapiro (2017), Duchi and Namkoong (2021), and references therein.”
Robust Optimization for Fairness with Noisy Protected Groups (2020, methodology): “Much existing work on DRO focuses on how to solve the DRO problem for different divergence metrics D. Namkoong and Duchi [46] provide methods for efficiently and optimally solving the DRO problem for f -divergences, and other work has provided methods for solving the DRO problem for Wasserstein dist”
Towards Out-Of-Distribution Generalization: A Survey (2021, methodology): “The distribution set P ( P tr ) in f -divergence DRO [1] is formulated as: where ρ > 0 controls the extent of the distributional shift, and D f ( Q (cid:107) P tr ) = f ( tr ) dP tr is the f -divergence be-tween Q and P tr .”