Averaging Weights Leads to Wider Optima and Better Generalization
By Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, Andrew Gordon Wilson. 2018
2,084 citations recorded by the source corpus.
Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Sergey Levine
- Luke S. Zettlemoyer
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Bernt Schiele
In Optimization & Training
- Quoc V. Le
- Yoshua Bengio
- Frank Hutter
- Richard Socher
- Alan L. Yuille
- Mohit Bansal
- Caiming Xiong
- Andrew Gordon Wilson
In Scaling & Learning Theory
- Andrew Gordon Wilson
- Pavel Izmailov
- Stanislav Fort
- Neil Houlsby
- Xiaohua Zhai
- Ari S. Morcos
- Lucas Beyer
- Sven Gowal
Citations in the corpus
- A Hybrid Neural Network Model for Commonsense Reasoning (2019, methodology): “We also apply SWA (Izmailov et al., 2018) to improve the generalization of models.”
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “, 2019), and connectivity between global modes in the context of model averaging (Izmailov et al., 2018; Fort et al., 2019; Wilson and Izmailov, 2020) and network pruning (Frankle et al.”
- Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “…algorithms that specifically seek out flatter minima and furthermore effectively improve generalization on a range of state-of-the-art models have thus far been elusive (e.g., see (Chaudhari et al., 2016; Izmailov et al., 2018); we include a more detailed discussion of prior work in Section 5).”
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Most applications study models along the same optimization trajectory, e.g. (Ruppert, 1988; Polyak, 1990; Szegedy et al., 2016; Izmailov et al., 2018; Zhang et al., 2019; Kaddour et al., 2022; Junczys-Dowmunt et al., 2016).”
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “The alternatives include using an approximation of the Hessian [89], [309] or calculating δ only along the trajectory of forward and backward parameter changes [58], [173], [182], [295], [308].”
- Don't Use Large Mini-Batches, Use Local SGD (2018, neutral): “Jastrzębski et al. (2017) argue that thus mainly the ratio ρ controls the stochastic noise and training dynamics, and that ρ positively correlates with wider minima and better generalization— matching with recent experimental successes of ImageNet training (Goyal et al.”
- A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “In order to compare SWAG with existing alternatives we report the results for standard SGD and SWA [20] solutions (single models), MC-Dropout [9], temperature scaling [12] applied to SWA and SGD solutions, SGLD [46], and K-FAC Laplace [43] methods.”
- Rehearsal revealed: The limits and merits of revisiting samples in continual learning (2021, background): “Such regions are an important part of the parameters space because they are linked to better generalization capabilities of a model [16, 17].”
- SWAD: Domain Generalization by Seeking Flat Minima (2021, methodology): “Since the importance of flatness in loss landscapes has emerged [23–28], several methods have been proposed to find flat minima [25, 26, 39].”
- Visualizing and Understanding the Effectiveness of BERT (2019, methodology): “The assumption is also used to design optimization algorithms (Chaudhari et al., 2017; Izmailov et al., 2018), which aims at finding broader optima with better generalization than standard SGD.”
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018, background): “See Izmailov et al. [15] for a detailed discussion of this phenomenon.”
- Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, methodology): “If a single set of weights is suffi-cient (as opposed to a distribution for model uncertainty), there are also empirically successful averaging techniques such as Polyak-Ruppert (Ruppert, 1988), checkpointing, and stochastic weight averaging (Izmailov et al., 2018).”
- Lookahead Optimizer: k steps forward, 1 step back (2019, background): “[14] proposes Stochastic Weight Averaging (SWA), which averages the weights at different checkpoints obtained during training.”
- There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, methodology): “• Motivated by our observations in Section 3 we propose to apply Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) to the Π and Mean Teacher models.”
- The Unusual Effectiveness of Averaging in GAN Training (2018, methodology): “Averaging at maximum likelihood objective: Uniform Average and Exponential Moving Average over model parameters has been used for a long time (Polyak & Juditsky, 1992; van den Oord et al., 2017; Athiwaratkun et al., 2018; Izmailov et al., 2018).”
- Deep Ensembles: A Loss Landscape Perspective (2019, background): “Izmailov et al. (2018) proposed stochastic weight averaging (SWA) for better generalization.”
- Robust fine-tuning of zero-shot models (2021, background): “Izmailov et al. (2018) average checkpoints saved throughout training for improved performance.”