Asymmetric Valleys: Beyond Sharp and Flat Local Minima

By Haowei He, Gao Huang, Yang Yuan. 2019

166 citations recorded by the source corpus.

Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good generalization performance. In this paper, we observe that local minima of modern deep networks are more than being flat or sharp. Specifically, at a local minimum there exist many asymmetric directions such that the loss increases abruptly along one side, and slowly along the opposite side--we formally define such minima as asymmetric valleys. Under mild assumptions, we prove that for asymmetric valleys, a solution biased towards the flat side generalizes better than the exact minimizer. Further, we show that simply averaging the weights along the SGD trajectory gives rise to such biased solutions implicitly. This provides a theoretical explanation for the intriguing phenomenon observed by Izmailov et al. (2018). In addition, we empirically find that batch normalization (BN) appears to be a major cause for asymmetric valleys.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, background): “A substantial body of methods [2, 37, 11, 15] have been proposed to optimize neural networks towards flat local minima.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “One of the solutions for mitigating the distribution variation problem is to seek the ﬂat-minimal region in the loss landscape [22, 11, 24, 21], as illustrated in Figure 2.”
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning (2020, background): “However, recent evidence [2, 22] shows that (1) for deep neural networks, the minima at the asymmetric is produced under the same setting in [23].”
SelfReg: Self-supervised Contrastive Regularization for Domain Generalization (2021, methodology): “It is known that finding a flatter minima guarantees better generalization performance [19], and thus it has been used in domain adaptation and generalization fields that require high generalization performance [48, 6].”
Questions for Flat-Minima Optimization of Modern Neural Networks (2022, methodology): “This ﬁnding conﬁrms and extends those of He et al. (2019) to models beyond ResNet and PreResNet, and datasets outside of CIFAR100.”
A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast (2020, background): “They try to measure ﬂatness (Hochreiter and Schmidhuber, 1997; Keskar et al., 2017; Sagun et al., 2017; Yao et al., 2018), rescale ﬂatness (Tsuzuku et al., 2019), and ﬁnd ﬂatter minima (Hoffer et al., 2017; Chaudhari et al., 2017; He et al., 2019b).”
Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond (2020, background): “Training Neural Networks with Guaranteed Flatness Recently, some researchers [13, 18, 12, 15] have hypothesized that DNNs optimized with stochastic gradient descent (SGD) can find wide and flat local minima which may be associated with good generalization performance.”