Towards Unified INT8 Training for Convolutional Neural Network

By Feng Zhu, Ruihao Gong, Fisher Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, Junjie Yan. 2019

179 citations recorded by the source corpus.

Recently low-bit (e.g., 8-bit) network quantization has been extensively studied to accelerate the inference. Besides inference, low-bit training with quantized gradients can further bring more considerable acceleration, since the backward process is often computation-intensive. Unfortunately, the inappropriate quantization of backward propagation usually makes the training unstable and even crash. There lacks a successful unified low-bit training framework that can support diverse networks on various tasks. In this paper, we give an attempt to build a unified 8-bit (INT8) training framework for common convolutional neural networks from the aspects of both accuracy and speed. First, we empirically find the four distinctive characteristics of gradients, which provide us insightful clues for gradient quantization. Then, we theoretically give an in-depth analysis of the convergence bound and derive two principles for stable INT8 training. Finally, we propose two universal techniques, including Direction Sensitive Gradient Clipping that reduces the direction deviation of gradients and Deviation Counteractive Learning Rate Scaling that avoids illegal gradient update along the wrong direction. The experiments show that our unified solution promises accurate and efficient INT8 training for a variety of networks and tasks, including MobileNetV2, InceptionV3 and object detection that prior studies have never succeeded. Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, and reduces the training time by 22% on Pascal GPU without too much optimization effort. We believe that this pioneering study will help lead the community towards a fully unified INT8 training for convolutional neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “current min-max or simply min-max, uses the full dynamic range of the tensor (Zhou et al., 2016; Wu et al., 2018b; Zhu et al., 2020);”
Binary Neural Networks: A Survey (2020, background): “The unified framework proposed in [41] proves the possibility of 8-bit training of neural networks from the accuracy and speed aspect.”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, unknown): “In particular, we surpasses UI8 on the large-scale COCO dataset by a large margin (more than 1%).”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, methodology): “Unlike existing work [20, 22], which studies the worst case behavior of the gradient, we adopt a statistical approach.”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “• Training using reduced precision arithmetic on 16-bit floating point or even quantized representations [76, 78, 45] significantly reduces memory [27, 50].”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “…eliminate ﬂoating-point operations from the quantized model, integer-only quantization techniques emulate the full-precision multiplication by 32-bit integer multiplication followed by bit shifting (Jacob et al., 2018; Zhu et al., 2020; Wu et al., 2020; Yao et al., 2021; Kim et al., 2021).”
Forward and Backward Information Retention for Accurate Binary Neural Networks (2019, background): “∗Corresponding author wide variety of computer vision applications such as image classification [30, 46, 47, 49, 50, 56, 63, 53], object detection [17, 16, 42, 45, 34] and semantic segmentation [15, 65].”