Mixed Precision Training of Convolutional Neural Networks using Integer Operations

By Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj D. Kalamkar, Sasikanth Avancha, K. Banerjee, Srinivas Sridharan, K. Vaidyanathan, Bharat Kaul, E. Georganas and 7 more. 2018

169 citations recorded by the source corpus.

The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half-precision

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

1. Yuan Xie
2. Song Han
3. By Lei Deng
4. Guoqi Li
5. Luping P. Shi

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Yi Yang
5. Song Han
6. Junjie Yan
7. Ming-Hsuan Yang
8. Jitendra Malik

Citations in the corpus

• Training Deep Neural Networks with 8-bit Floating Point Numbers (2018, background): “, [16, 4]) to 16 bits significantly impacts the convergence of DNN training (Fig.”
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, unknown): “Several studies have developed techniques such as loss scaling, stochastic rounding, and others to train effectively in 16-bit (Micikevicius et al., 2018; Das et al., 2018; Azim), along with associated hardware support (Markidis et al., 2018).”
• A Study of BFLOAT16 for Deep Learning Training (2019, background): “There are at least three half-precision formats in the domain of mixed precision training of large neural networks: FP16 [26], 16-bit Integer based [8], and BFLOAT16.”
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Half precision floating point (16-bit floating-point, FP16) has been widely used in nVidia GPUs and ASIC accelerators with minimal accuracy loss [51].”
• Norm matters: efficient and accurate normalization schemes in deep networks (2018, methodology): “We additionally demonstrated the benefits of L(1) normalization: it allowed us to perform BN in half-precision floating-point, which was noted to fail in previous works [24, 8] and required full and mixed precision hardware.”
• Towards Unified INT8 Training for Convolutional Neural Network (2019, methodology): “Flexpoint [33], MPT [41] and DFP [9] all use 16-bit floating-point to train DNNs with accuracy comparable to full-precision model.”
• Mixed Precision Training With 8-bit Floating Point (2019, methodology): “Recent studies[21],[5] have shown that, deep neural networks can be trained using 16-bits of precision without any noticeable impact on validation accuracy across a wide range of networks.”
• This paper is included in the Proceedings of the 2022 USENIX Annual Technical Conference. (unknown): “The mixed precision training has shown significant speedup over the single (full) precision training on a variety of NN [2–4], especially with tensor cores (TC) available on some GPU architectures.”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “For example, the quantized networks can be viewed as dense models with shorter bit-width [179]; structurally pruned networks (e.g., vector-wise in RNNs and FM-wise in CNNs) [204], [205], [232], [234] or decomposed networks [139] can be reassembled into smaller dense models to speed up the running.”
• Scalable Methods for 8-bit Training of Neural Networks (2018, background): “While several works [6, 5] have shown that training at 16-bit is sufficient for most networks, more aggressive quantization schemes were also suggested [24, 16, 14, 10].”