ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
By Jungmin Kwon, Jeongseop Kim, Hyunseong Park, Inae Choi. 2021
419 citations recorded by the source corpus.
Recently, learning algorithms motivated from sharpness of loss surface as an effective measure of generalization gap have shown state-of-the-art performances. Nevertheless, sharpness defined in a rigid region with a fixed radius, has a drawback in sensitivity to parameter re-scaling which leaves the loss unaffected, leading to weakening of the connection between sharpness and generalization gap. In this paper, we introduce the concept of adaptive sharpness which is scale-invariant and propose the corresponding generalization bound. We suggest a novel learning method, adaptive sharpness-aware minimization (ASAM), utilizing the proposed generalization bound. Experimental results in various benchmark datasets show that ASAM contributes to significant improvement of model generalization performance.
Subfields: Artificial Intelligence, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sebastian Riedel
- Armand Joulin
- Edouard Grave
- Luke S. Zettlemoyer
- Matthieu Cord
- Hannaneh Hajishirzi
- Piotr Bojanowski
- Mohit Bansal
In Optimization & Training
- Gautier Izacard
- Shuaicheng Niu
- Peilin Zhao
- Guillaume Couairon
- Arthur Douillard
- Jianfei Cai
- Yaofo Chen
- Mathilde Caron
Citations in the corpus
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “We use the ASAM optimizer (Kwon et al., 2021), with a learning rate of 10 − 5 and a batch size of 1024 with a temperature of 0 .”
- DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, methodology): “We now refine DyTox+ into DyTox++ by adding a new component during the training: the Sharpness-Aware Minimizer (SAM) [16].”
- Towards Stable Test-Time Adaptation in Dynamic Wild World (2023, background): “SAM and its variants (Kwon et al., 2021; Zheng et al., 2021; Du et al., 2022; Chen et al., 2022b) have shown outstanding performance on several deep learning benchmarks.”
- Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Apart from the efficiency issues, several methods including ASAM [34], GSAM [78] and Fisher SAM [31] have been proposed to improve the performance of SAM.”
- Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “We use the ASAM optimizer (Kwon et al., 2021), with a learning rate of 10 − 5 and a batch size of 1024 with a temperature of 0 .”