Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
By Benoit Jacob, Skirmantas Kligys, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, Dmitry Kalenichenko. 2017
4,475 citations recorded by the source corpus.
The rising popularity of intelligent mobile devices and the daunting computational cost of deep learning-based models call for efficient and accurate on-device inference schemes. We propose a quantization scheme that allows inference to be carried out using integer-only arithmetic, which can be implemented more efficiently than floating point inference on commonly available integer-only hardware. We also co-design a training procedure to preserve end-to-end model accuracy post quantization. As a result, the proposed quantization scheme improves the tradeoff between accuracy and on-device latency. The improvements are significant even on MobileNets, a model family known for run-time efficiency, and are demonstrated in ImageNet classification and COCO detection on popular CPUs.
Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Electrical and Electronic Engineering
- Young Geun Kim
- Song Han
- Carole-Jean Wu
- Tim Dettmers
- Luke S. Zettlemoyer
- Quoc V. Le
- Kurt Keutzer
- Raghuraman Krishnamoorthi
In Efficiency & Compression
- Song Han
- Kurt Keutzer
- Dan Alistarh
- Quoc V. Le
- Luke S. Zettlemoyer
- Tijmen Blankevoort
- A. Gholami
- Michael W. Mahoney
Citations in the corpus
- The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “…et al., 2022; Wu et al., 2020; Jain et al., 2020; Nagel et al., 2019; Krishnamoorthi, 2018; Rusci et al., 2020), centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014; Han et al., 2015; Choi et al., 2016; Park et al.,…”
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, methodology): “Here, we estimate the scale of activations channels using the calibration samples from the pre-training dataset (Jacob et al., 2018).”
- Linformer: Self-Attention with Linear Complexity (2020, methodology): “This technique can be further improved through Quantization Aware Training (Jacob et al., 2018; Fan et al., 2020), where the weights are quantized during training and the gradients are approximated with the Straight-Through Estimator.”
- MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, methodology): “Some commonly-used approaches include 1) quantizing the weights and/or activations of a baseline CNN model into lower-bit representations (Han, Mao, and Dally 2015; Jacob et al. 2018), or 2) pruning less important filters (Gordon et al.”
- Q8BERT: Quantized 8Bit BERT (2019, methodology): “Fake quantization is an operation that simulates the rounding effect in Floating Point values as presented by Jacob et al. [5].”
- QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “In fact, Tensorflow offers a lightweight variant to address this goal [19, 27].”
- Towards Federated Learning at Scale: System Design (2019, methodology): “In addition to that, we can modify the training algorithms to obtain models in quantized representation (Jacob et al., 2017), which will have synergetic effect with bandwidth savings and be important for efficient deployment for inference.”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “Moreover, the proposed co-evolution algorithm was built on pruning only, while many other compression techniques, such as quantization (Jacob et al., 2018), knowledge distillation (Polino et al., 2018), and even AutoML (He et al., 2018), were left unexplored.”
- Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, methodology): “This section focuses on fine-tuning the integer-only network with simulated quantization training (a.k.a. fake quantization) (Jacob et al., 2018).”
- Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “There are many effective strategies for improving the parameter efficiency of trained networks: for example, pruning algorithms learn to remove redundant or less important parameters [12,40], and quantization algorithms learn to represent parameters using fewer bits while maintaining task performanc”
- NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, background): “Quantization [11, 12, 17] reduces the complexity by decreasing the computation accuracy.”
- Training with Quantization Noise for Extreme Model Compression (2020, methodology): “A standard solution is to train networks with Quantization Aware Training (Jacob et al., 2018), where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator (Bengio et al.”
- HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, methodology): “Integer-Only [16] 8 8 75.”
- Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “An alternative to PTQ is to train a neural network with the simulated quantization operations in the network, known as quantization-aware training (QAT, Jacob et al. 2018; Gupta et al. 2015; Krishnamoorthi 2018).”