Sharp Minima Can Generalize For Deep Nets
By Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio. 2017
918 citations recorded by the source corpus.
Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice. However, explaining why this is the case is still an open area of research. One standing hypothesis that is gaining popularity, e.g. Hochreiter & Schmidhuber (1997); Keskar et al. (2017), is that the flatness of minima of the loss function found by stochastic gradient based methods results in good generalization. This paper argues that most notions of flatness are problematic for deep models and can not be directly applied to explain generalization. Specifically, when focusing on deep networks with rectifier units, we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima. Furthermore, if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Percy Liang
- Sergey Levine
- Dacheng Tao
- Ruslan Salakhutdinov
- Jianfeng Gao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Xiaogang Wang
- Bernt Schiele
- Yi Yang
In Scaling & Learning Theory
- Yuanzhi Li
- Tengyu Ma
- Jason D. Lee
- Greg Yang
- Zhi-Qin John Xu
- Behnam Neyshabur
- Michael W. Mahoney
- Yoshua Bengio
Citations in the corpus
- Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “For a fair comparison between batch sizes, batch normalization statistics should be computed over the same number of examples or else the training objective differs between batch sizes (Goyal et al., 2017). Indeed, Hoffer et al. (2017) found that computing batch normalization statistics over larger ”
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”
- Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “The idea of searching for “flat” minima can be traced back to Hochreiter & Schmidhuber (1995), and its connection to generalization has seen significant study (Shirish Keskar et al., 2016; Dziugaite & Roy, 2017; Neyshabur et al., 2017; Dinh et al., 2017).”
- The large learning rate phase of deep learning: the catapult mechanism (2020, background): “…is that noise from stochastic gradient descent leads to flat minima, and that flat minima generalize better than sharp minima (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Smith & Le, 2018; Jiang et al., 2020; Park et al., 2019) (though see Dinh et al. (2017) for discussion of some caveats).”
- Fixing Weight Decay Regularization in Adam (2017, background): “Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima (Keskar et al., 2016; Dinh et al., 2017) and inherent problems of adaptive gradient methods (Wilson et al., 2017).”
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “The training objectives of overparameterized networks in principle have many (approximate) global optima and some generalize better than the others [14, 8, 2], while empirical observations imply that the optimization process in practice prefers those with better generalization.”
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, background): “An example is the recent, Dinh et al. [2017], work shows how sharp minima can still generalize with proper modifications to the loss function.”
- Swish: a Self-Gated Activation Function (2017, methodology): “…example, He et al. (2015) derived a simple extension of the linear initialization of Glorot & Bengio (2010) that is used widely for ReLU networks, Dinh et al. (2017) used the linearity of ReLU to form an equivalent sharp repa-rameterization of networks, and Raghu et al. (2016) measured…”
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “The Rashomon volume or other statistics that characterize the Rashomon set can be computed more easily if the volume in parameter space has a single global minimum (Hochreiter and Schmidhuber, 1997; Dinh et al., 2017; Keskar et al., 2016; Chaudhari et al., 2016; Keskar et al., 2016; Chaudhari et al.”
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “This ability of deep networks to generalize well on real data, while simultaneously being able to fit arbitrary data has recently received a lot of attention (e.g. Soudry et al. [2018], Dinh et al. [2017], Zhang et al. [2018], Arpit et al. [2017], Kawaguchi et al. [2017]).”
- Wireless Network Intelligence at the Edge (2018, methodology): “d) Minimum energy path: Finding flat minima by using EL has recently been tackled by a counterexample [96].”
- Don't Use Large Mini-Batches, Use Local SGD (2018, background): “(2014) and then widely used to study the “sharpness” and “flatness” of different minima (Keskar et al., 2016; Dinh et al., 2017; Li et al., 2018).”
- Visualizing the Loss Landscape of Neural Nets (2017, methodology): “[7] to build pairs of equivalent networks that have different apparent sharpness.”
- Averaging Weights Leads to Wider Optima and Better Generalization (2018, background): “On the other hand Dinh et al. [2017] argue that all the known definitions of sharpness are unsatisfactory and cannot on their own explain generalization.”
- CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, neutral): “[31,12] analyzed that the forgetting of old tasks in continual learning can be bounded by the variation of parameters between tasks and the eigenvalues of the Hessian matrix, where the lower eigenvalues indicate a flatter curvature of the solution.”
- Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “Recently, a lot of work [6, 8, 38] has been developed to study the neural network optimization landscape.”
- Understanding the Role of Training Regimes in Continual Learning (2020, methodology): “We also want to highlight λ 1 has been used previously to describe the width of a local minima [28, 39], with similar notes regarding the role of the entire spectrum [12, 39].”
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018, background): “[4] argue that most existing notions of flatness cannot directly explain generalization.”
- Where is the Information in a Deep Neural Network? (2019, background): “Note that stability or flatness alone may be insufficient to explain generalization: Dinh et al. (2017) show examples of sharp but not necessarily stable, minima that generalize.”