High-Accuracy Low-Precision Training

By Christopher De Sa, Megan Leszczynski, Jian Zhang, Alana Marzoev, Christopher R. Aberger, Kunle Olukotun, Christopher R'e. 2018

114 citations recorded by the source corpus.

Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it. Still, it has been used primarily for inference - not training. Previous low-precision training algorithms suffered from a fundamental tradeoff: as the number of bits of precision is lowered, quantization noise is added to the model, which limits statistical accuracy. To address this issue, we describe a simple low-precision stochastic gradient descent variant called HALP. HALP converges at the same theoretical rate as full-precision algorithms despite the noise introduced by using low precision throughout execution. The key idea is to use SVRG to reduce gradient variance, and to combine this with a novel technique called bit centering to reduce quantization error. We show that on the CPU, HALP can run up to $4 \times$ faster than full-precision SVRG and can match its convergence trajectory. We implemented HALP in TensorQuant, and show that it exceeds the validation performance of plain low-precision SGD on two deep learning tasks.

Subfields: Biomedical Engineering, Computational Mechanics, Radiology, Nuclear Medicine and Imaging, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computational Mechanics

In Radiology, Nuclear Medicine and Imaging

In Optimization & Training

Citations in the corpus

Scaling Neural Machine Translation (2018, methodology): “Previous research considered training and inference with reduced numerical precision for neural networks (Simard and Graf, 1993; Courbariaux et al., 2015; Sa et al., 2018).”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “While previous work [2, 4, 9, 24, 30, 36, 46, 56] had already introduced the idea of functions Qw ,Qa to quantize weights and/or activations in the forward pass, they required the weights W or gradients G, e to be represented in floating-point in the backward pass, in order to optimize accurately.”
Wireless Network Intelligence at the Edge (2018, methodology): “To optimize the compression–distortion tradeoff, HALP training is a viable solution [198].”
Expanding the Reach of Federated Learning by Reducing Client Resource Requirements (2018, methodology): “Popular alternatives include pruning the least useful connections in a network [Han et al., 2016, 2015], weight quantization [Hubara et al., 2016, Lin et al., 2017, De Sa et al., 2018], and model distillation [Hinton et al., 2015].”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, neutral): “Partially initiated by a 1-bit implementation of SGD by Microsoft in [5], a large number of recent studies revisited the idea of low-precision training as a means to reduce communication [13, 14, 15, 16, 17, 18, 19, 17, 20, 21].”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, background): “De Sa et al. [7] describe a simple low-precision stochastic gradient descent variant called HALP, which converges at the same theoretical rate as full-precision algorithms despite the noise introduced by using low precision throughout execution.”
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark (2018, methodology): “…side, large minibatch training [31,38] and reduced precision [20,23,48] can help run iterations of the optimization algorithm faster and speed up “proxy” metrics such as time to process an epoch (“time-per-epoch”), but can prevent models from reaching the same accuracy on unseen data [24, 46, 47].”
OD-SGD (2020, methodology): “As to the term of software, C. R. Aberger et al. [30] introduce a training algorithm called High-Accuracy Low-Precision (HALP) and Micikevicius et al. [31] describe a methodology (Mixed Precision Training, MPT) for neural network training using half-precision ﬂoating point numbers.”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, background): “More recently, researchers demonstrated that effective deep learning inference [3, 32, 34, 38], and even training [8, 51] can rely on numerics in a more limited range and precision than that given by fp32.”
Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs (2019, unknown): “ﬁnally resorting to horizontal scaling that is not sustainable. Many domain-speciﬁc solutions across the stack have been proposed to deal with this memory capacity challenge in deep learning training [36, 37, 38, 39, 18, 5, 40, 8]. We use a set of convolutional neural networks (CNNs) and one recurre”
Mixed Precision Training With 8-bit Floating Point (2019, methodology): “There have been numerous attempts [13],[27],[6],[25],[4] to train deep neural networks at lower precision ( 16-bits) with varying degrees of success.”
SWALP : Stochastic Weight Averaging in Low-Precision Training (2019, methodology): “We make this simplification for the convenience of the theoretical analysis in Sec 4, and following previous theoretical works in this space (Li et al., 2017; De Sa et al., 2018).”
TF-Replicator: Distributed Machine Learning for Researchers (2019, background): “Aside from the growing body of literature on low-precision training (De Sa et al., 2018; Micikevicius et al., 2017), these studies have typically leveraged weak scaling to train in fewer steps with very large batches.”