Diametrical Risk Minimization: theory and computations
By Matthew Norton, Johannes O. Royset. 2019
22 citations recorded by the source corpus.
The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in sharp empirical risk landscapes from benchmark neural network classification problems with corrupted labels.
Subfields: Artificial Intelligence, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- SWAD: Domain Generalization by Seeking Flat Minima (2021, methodology): “Hence, solving the robust risk minimization (RRM), i.e. , arg min θ ˆ E γ D ( θ ) , will find a near solution of a flat optimum showing better generalizability [26, 34].”
- Adversarial Weight Perturbation Helps Robust Generalization (2020, background): “Recalling that weight loss landscape is a widely used indicator to characterize the standard generalization gap in standard training scenario [33, 22, 12, 34], however, there are few explorations under adversarial training[38, 59, 23], among which, Prabhu et al.”