Sharpness-Aware Minimization for Efficiently Improving Generalization

By Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur. 2020

2,017 citations recorded by the source corpus.

In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by the connection between geometry of the loss landscape and generalization---including a generalization bound that we prove here---we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-{10, 100}, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Both models are trained with learning rate 3× 10−4 and SAM [15] with ρ = 0.1, where the perturbations are applied independently by the replicas, each of which uses batch size 64.”
VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “As depicted in Figure 1, compared to the previous stateof-the-art CNN-based model (NFNet-F6 [3] with SAM [15]), and transformer-based model (CaiT-M48 [51] with KD), our best model VOLO-D5 leverages the least amount of learnable parameters but achieves the best accuracy.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “We also train four models with sharpness-aware minimization (SAM) (Foret et al., 2021).”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “So we list the baseline results in (Foret et al., 2021) without using SAM optimization for a fairer comparison.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “Accelerating Sharpness-Aware Minimization Sharpness-Aware Minimization (SAM, Foret et al. (2021)) has been shown to improve the performance of various classifier models by seeking flat minima which are hypothesized to generalize better.”
The Efficiency Misnomer (2021, background): “For instance, different optimizers use different amounts of memory on the device, for instance, SGD-Momentum (Qian, 1999) vs SAM (Foret et al., 2020).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “SAM [120], [295] encourages finding a wide basin in downstream continual learning by optimizing the flatness metric.”
Incorporating neuro-inspired adaptability for continual learning in artificial intelligence (2023, background): “A48 are derived from Theorem 2 in [80], which hold by assuming E t ( θ With active forgetting by taking the form of L 2 norm for parameters, we need to substitute the the third term in Eq.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, methodology): “For the randomly initialized model (Figure 2a), the euclidean distances between the model parameter vectors are approximately the same as for the pre-trained model, but the differences in Task 1 loss levels are signiﬁcantly higher.”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “The overfitting problem of DNNs is often attributed to the complex loss landscape containing multiple local optima, and the sharpness of the loss landscape has been widely used to characterize the generalization gap in standard training scenarios from both theoretical and empirical perspectives [27,”
SWAD: Domain Generalization by Seeking Flat Minima (2021, methodology): “Since the importance of ﬂatness in loss landscapes has emerged [23–28], several methods have been proposed to ﬁnd ﬂat minima [25, 26, 39].”
Transformers in Vision: A Survey (2021, methodology): “by smoothing the local loss surface using sharpness-aware minimizer (SAM) [258], ViTs can be trained with simple data augmentation scheme (random crop, and horizontal”
Meta Pseudo Labels (2020, background): “72 EfficientNet-L2-NoisyStudent + SAM [16] 480M 300M unlabeled JFT 88.”
ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks (2021, methodology): “…and p = ∞ , respectively, and select 5e 1 and 5e 3 as ρ s for adaptive sharpness of p = 2 and p = ∞ , respectively. follow To calculate maximizers of each loss function for calculation of sharpness and adaptive sharpness, we m -sharpness strategy suggested by Foret et al. (2021) and m is set to 8 .”
Questions for Flat-Minima Optimization of Modern Neural Networks (2022, methodology): “2 , similarly to previous work (Foret et al., 2021; Bahri et al., 2021).”
Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, background): “We highlight that the affirmative answer provided in this paper could prove extremely practical, as attention entropy is easier to compute and potentially manipulate then directly tackling the loss geometry, which typically involves computing second derivatives, as in (Foret et al., 2021).”
Regularizing Neural Networks via Adversarial Model Perturbation (2020, background): “The concurrent work [9] also independently discovered a similar regularization scheme as we present in this paper.”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, methodology): “After applying augmentation multiplicity, our NFNet-F5 with SAM achieves 86 .”