Scalable Methods for 8-bit Training of Neural Networks

By Ron Banner, Itay Hubara, Elad Hoffer, Daniel Soudry. 2018

375 citations recorded by the source corpus.

Quantized Neural Networks (QNNs) are often used to improve network efficiency during the inference phase, i.e. after the network has been trained. Extensive research in the field suggests many different quantization schemes. Still, the number of bits required, as well as the best quantization scheme, are yet unknown. Our theoretical analysis suggests that most of the training process is robust to substantial precision reduction, and points to only a few specific operations that require higher precision. Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients. Additionally, as QNNs require batch-normalization to be trained at high precision, we introduce Range Batch-Normalization (BN) which has significantly higher tolerance to quantization noise and improved computational complexity. Our simulations show that Range BN is equivalent to the traditional batch norm if a precise scale adjustment, which can be approximated analytically, is applied. To the best of the authors' knowledge, this work is the first to quantize the weights, activations, as well as a substantial volume of the gradients stream, in all layers (including batch normalization) to 8-bit while showing state-of-the-art results over the ImageNet-1K dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “On the AGD-compressed CycleGAN, we further quantize all the weights and activations to 8-bit integers, following (Banner et al., 2018).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “While this survey on quantization is mostly focused on inference, we should emphasize that an important success of quantization has been in NN training [10, 35, 57, 128, 239].”
MLPerf Training Benchmark (2019, background): “With the introduction of systems with varying numerics (Abadi et al., 2016; Banner et al., 2018; Kster et al., 2017; Micikevicius et al., 2018) and performance optimization techniques, ML benchmarks cannot afford to omit accuracy measures.”
Benchmarking TPU, GPU, and CPU Platforms for Deep Learning (2019, methodology): “8-bit training is an example [6].”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, positive): “We might as well assume the bell-shaped gradients can be described by Gaussian distribution, which takes the max absolute value ( | g | max ) as optimal quantization parameter (Banner et al. 2018): In general, the quantization parameter can be denoted by: s = β | g | max , with β ∈ (0 , 1] .”
BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning (2020, methodology): “As a result, dACIQ adopts a simple, yet effective Gaussian ﬁtting method proposed in [4].”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, methodology): “The back propagation is with gradient bifurcation [20].”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, methodology): “This issue is further exacerbated if low precision (4-bit) integers are used to represent weights and activations for inference and in general for training proposals that utilize the same number formats for all tensors (including 8-bit ﬁxed-point[16] or the 5-bit logarithmic format[29]).”
SWALP : Stochastic Weight Averaging in Low-Precision Training (2019, methodology): “, 2018); others explore building specialized layers using lowprecision arithmetic (Wu et al., 2018; Rastegari et al., 2016; Zhou et al., 2016; Courbariaux et al., 2015; Banner et al., 2018).”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, methodology): “However, they too require the ﬁrst and last layers of the model to be in FP32, and similar to (Banner et al., 2018) leverage Stochastic Rounding techniques to maintain model accuracy.”
NICE: Noise Injection and Clamping Estimation for Neural Network Quantization (2018, methodology): “To overcome the accuracy reduction, some works proposed to use a wider network (Zhu et al., 2016; Polino et al., 2018; Banner et al., 2018), which compensates the expressiveness reduction of the quantized networks network.”
Few Shot Network Compression via Cross Distillation (2019, methodology): “To perform data-free network compression, a simple way is to directly apply quantization (Banner et al. 2018) or lowrank factorization (Zhang et al.”
Low-Memory Neural Network Training: A Technical Report (2019, background): “Recently, Banner et al. (2018) showed that all computation during training, aside from accumulation in certain intermediate operations, can actually be done in 8-bit precision; however, they still maintain a 16-bit “master” copy of the weights.”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “Popular quantization methods include post-training quantization [45, 33, 29], quantization-aware training (QAT) [23, 4, 31, 22], and their variations [44, 24, 12].”
NeRV: Neural Representations for Videos (2021, result): “Note that different from many recent works [31, 47–49] that utilize quantization during training, NeRV is only quantized post-hoc (after the training process).”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Therefore, reducing BN’s nonlinearity and quantizing BN operations also matter [180], [194], [195].”
Compact and Computationally Efficient Representation of Deep Neural Networks (2018, background): “Alternatively, one could trivially reduce the numerical precision [31], [32], [41], [42] needed for representing each weight element value down to 7 bits in this case.”