SWALP : Stochastic Weight Averaging in Low-Precision Training
By Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, Christopher De Sa. 2019
107 citations recorded by the source corpus.
Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Pavel Izmailov
- Andrew Gordon Wilson
- Wesley J. Maddox
- Dmitry P. Vetrov
- Timur Garipov
- Haowei He
- Gao Huang
- Yang Yuan
Citations in the corpus
- A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “SWA and related ideas have been successfully applied to a wide range of applications [see e.g. 2, 61, 62, 51].”
- Asymmetric Valleys: Beyond Sharp and Flat Local Minima (2019, background): “Our result nicely complements a series of recent empirical observations, which demonstrated that averaged SGD has better performance over plain SGD, for various scenarios including supervised/unsupervised/low-precision training [25, 5, 51].”
- Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (2019, unknown): “This scheme, however, results in > 4% accuracy degradation on ResNet18/ImageNet [24].”
- Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “Furthermore, we apply the proposed EB Train on top of the low-precision training method (Yang et al., 2019a) and obtain experiment results of another two variants of EB Train: 1) EB Train with low-precision search and full-precision retrain (i.e., EB Train LF), and 2) EB Train with both…”