Training and Inference with Integers in Deep Neural Networks

By Shuang Wu, Guoqi Li, F. Chen, Luping P. Shi. 2018

419 citations recorded by the source corpus.

Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as "WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Secure Evaluation of Quantized Neural Networks (2019, background): “It is important to notice that the spectrum of research in quantized training of CNNs is much more reduced than in quantized inference, with only a few recent works like [35], [65] and [66].”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, neutral): “However, for the task of training, there are few works that perform all operations in fixed-point [20, 23, 33, 39, 57].”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Early network quantization methods focused on low-bitwidth networks such as BinaryNet (Courbariaux, Bengio, and David 2015; Courbariaux et al. 2016), XOR-Nets (Rastegari et al. 2016), or Ternary networks (Li, Zhang, and Liu 2016; Wu et al. 2018).”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “current min-max or simply min-max, uses the full dynamic range of the tensor (Zhou et al., 2016; Wu et al., 2018b; Zhu et al., 2020);”
Training Deep Neural Networks with 8-bit Floating Point Numbers (2018, background): “WAGE [20] quantizes weights, activations, errors and gradients to 2, 8, 8 and 8 bits respectively.”
Post training 4-bit quantization of convolutional networks for rapid-deployment (2018, background): “The per-channel scale has shown to be important both for inference (Rastegari et al., 2016) and for training (Wu et al., 2018).”
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (2019, background): “While 8-bit training techniques have progressed rapidly, recent work [2, 3, 8, 9] have only demonstrated its applicability on a small subset of deep learning models—focused around convolution networks such as ResNet [10].”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, positive): “Wu et al. [106] proposed an approach to reduce the cost of training and inference by using lower bitwidth integers for both stages of the application.”
Towards Unified INT8 Training for Convolutional Neural Network (2019, background): “WAGE [56] and WAGEUBN [60] quantize gradient to 8-bit integer, but they both incur considerable loss of accuracy (greater than 5%).”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, critical): “Besides quantizing gradient to INT8, WAGE (Wu et al. 2018) and WAGEUBN (Yang et al. 2020) further quantize other training modules (e.g., Error and Update), leading to large loss of accuracy.”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, neutral): “Various 8-bit numerical formats have been proposed, including INT8 [20, 21, 22, 23], FP8 [24, 25], 𝐅 𝑙 (𝐇 𝑙 ,𝚯 𝑙 ) 𝐉 (𝑙) 𝐊 (𝑙) ⊗ ⊗ 𝐇 (𝑙−1) 𝚯 (𝑙) 𝐇 (𝑙) 𝑄 𝑏 (෡∇ 𝐇 (𝑙) ) ෡∇ 𝚯 (𝑙)”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, methodology): “BCQ format (without quantizing activations that require heavy modiﬁcations in training codes and model structure [23], [25]) is also useful to be implemented by LUT-based approaches.”
Fast-SCNN: Fast Semantic Segmentation Network (2019, methodology): “Network Quantization: Since ﬂoating point multiplications are costly compared to integer or binary operations, runtime can be further reduced using quantization techniques for DCNN ﬁlters and activation values [11, 22, 33].”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “We compare with several other schemes from literature including LSQ [7], TWN [17], RQ [19], WAGE [32] and DQ [29].”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, background): “Still other studies examine quantizing neural networks (Wu et al., 2018); in the extreme, Courbariaux et al. (2016) propose binarized networks with both binary weights and binary activations.”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, methodology): “Wu et al. (2018) demonstrate using integers for training LeNet-5 (LeCun et al., 1998) and AlexNet with 8-bits for activations, error and gradients and 2-bits for weights.”
Relaxed Quantization for Discretized Neural Networks (2018, neutral): “Within the body of work that considers quantizing weights and activations fall papers using stochastic rounding (Gupta et al., 2015; Hubara et al., 2016; Gysel et al., 2018; Wu et al., 2018).”
SWALP : Stochastic Weight Averaging in Low-Precision Training (2019, methodology): “Many techniques have been proposed to reduce this performance gap (Zhou et al., 2016; Wu et al., 2018; Banner et al., 2018; Wang et al., 2018).”
Bayesian Bits: Unifying Quantization and Pruning (2020, methodology): “On CIFAR 10 we get an error that is similar to WAGE (Wu et al., 2018), while using only 35% of its GBOPs.”