Stochastic Gradient Methods for Distributionally Robust Optimization with f-divergences

By Hongseok Namkoong, John C. Duchi. 2016

409 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, methodology): “However, one can use the min-max mirror descent algorithm or stochastic minmax mirror descent algorithms (Juditsky et al., 2011; Namkoong and Duchi, 2016; Cotter et al., 2019; Mohri et al., 2019) to obtain heuristic solutions.”
Fairness Without Demographics in Repeated Loss Minimization (2018, methodology): “While there are many possible robustness balls B which could provide upper bounds on group risk, we opt to use the chi-squared ball since it is straightforward to optimize (Ben-Tal et al., 2013; Namkoong & Duchi, 2016; 2017) and we found it empirically outperformed other f -divergence balls.”
Certifying Some Distributional Robustness with Principled Adversarial Training (2017, background): “…ﬁnite-dimensional parametrizations for P , such as constraint sets for moments, support, or directional deviations [17, 20, 27], as well as non-parametric distances for probability measures such as f -divergences [5, 6, 40, 44, 21, 47], and Wasserstein distances [23, 59, 9, 24, 10, 25, 37].”
Variance-based Regularization with Convex Objectives (2016, background): “Namkoong and Duchi [21] propose a stochastic procedure for minimizing (4) almost as fast as stochastic gradient descent.”
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization (2019, methodology): “Recall that we aim to solve the optimization problem (4), which can be rewritten as Extending existing minimax algorithms for DRO (Namkoong & Duchi, 2016; Oren et al., 2019), we interleave gradient-based updates on θ and q .”
Robustness to Spurious Correlations via Human Annotations (2020, methodology): “Although there exist many techniques for efﬁ-cient distributionally robust optimization (Ben-Tal et al., 2013; Namkoong & Duchi, 2016; Duchi & Namkoong, 2018), the UV-DRO objective is challenging to estimate from ﬁnite samples, as the outer supremum depends on the conditional risk E [ (cid:96) ( θ…”
Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach (2016, methodology): “imation and stochastic gradient methods [54, 48, 26, 22, 36, 30], there are fewer established and computationally eﬃcient solution methods for minimax problems of the form (4a) (though see the papers [48, 17, 6, 60, 47] for work in this direction). Eﬃcient solution methods need to be developed to sc”
Learning Models with Uniform Performance via Distributionally Robust Optimization (2018, methodology): “In comparison, our f -divergence formulation is computationally efficient to solve, even in large-scale learning scenarios [68, 69].”
Certifiable Distributional Robustness with Principled Adversarial Training (2017, neutral): “…have considered ﬁnite-dimensional parametrizations for P , such as constraint sets for moments, support, or directional deviations [14, 17, 21], as well as non-parametric distances for probability measures such as f -divergences [4, 5, 32, 29, 18, 35], and Wasserstein distances [8, 19, 45].”
Distributionally Robust Language Modeling (2019, background): “Online algorithms for DRO exist (Namkoong and Duchi, 2016), but do not handle the nested maximization-expectation structure arising in topic CVaR (Eq.”
Robust Optimization for Non-Convex Objectives (2017, methodology): “For continuous optimization, the work that is closest to ours is perhaps that by Shalev-Shwartz and Wexler [17] and Namkoong and Duchi [14] that use robust optimization to train against convex loss functions.”
Global Convergence and Variance-Reduced Optimization for a Class of Nonconvex-Nonconcave Minimax Problems (2020, positive): “Other applications include robust optimization [Namkoong and Duchi, 2016, 2017], adversarial machine learning [Sinha et al., 2017, Madry et al., 2017], unsupervised learning [Xu et al., 2005], and so on.”
DORO: Distributional and Outlier Robust Optimization (2021, neutral): “robust optimization (DRO) (Namkoong & Duchi, 2016; Duchi & Namkoong, 2018) refers to a family of learning algorithms that minimize the model’s loss over the worst-case distribution in a neighborhood of the observed training distribution.”
Linear Bandits on Uniformly Convex Sets (2021, background): “Linear Bandit algorithms are instrumental in solving minimax problems with convex-linear structure stemming from learning applications, see, e.g. , SVMs [HKS11; CHW12] or Distributional Robust Optimization [ND16; Cur+20].”
Robust Optimal Transport with Applications in Generative Modeling and Domain Adaptation (2020, methodology): “Following [18], we use weighted empirical distribution for the perturbed distribution PX̃ , i.”