Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss

By Sangil Jung, Changyong Son, Seohyung Lee, JinWoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, Changkyu Choi. 2018

430 citations recorded by the source corpus.

Reducing bit-widths of activations and weights of deep networks makes it efficient to compute and store them in memory, which is crucial in their deployments to resource-limited devices, such as mobile phones. However, decreasing bit-widths with quantization generally yields drastically degraded accuracy. To tackle this problem, we propose to learn to quantize activations and weights via a trainable quantizer that transforms and discretizes them. Specifically, we parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network. This quantization-interval-learning (QIL) allows the quantized networks to maintain the accuracy of the full-precision (32-bit) networks with bit-width as low as 4-bit and minimize the accuracy degeneration with further bit-width reduction (i.e., 3 and 2-bit). Moreover, our quantizer can be trained on a heterogeneous dataset, and thus can be used to quantize pretrained networks without access to their training data. We demonstrate the effectiveness of our trainable quantizer on ImageNet dataset with various network architectures such as ResNet-18, -34 and AlexNet, on which it outperforms existing methods to achieve the state-of-the-art accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, background): “More recent works such as Rastegari et al. (2016); Zhou et al. (2016); Choi et al. (2018); Jung et al. (2018); Zhuang et al. (2018) quantize both weights and activations to reduce the computational cost on CPUs and dedicated hardware accelerators.”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “There are many effective strategies for improving the parameter efficiency of trained networks: for example, pruning algorithms learn to remove redundant or less important parameters [12,40], and quantization algorithms learn to represent parameters using fewer bits while maintaining task performanc”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “To improve accuracies, higher bitwidth quantization methods were studied (Jung et al. 2019; Zhang et al. 2018a; Mishra et al. 2017).”
Learned Step Size Quantization (2019, methodology): “Prior approaches using backpropagation to learn quantization controlling parameters (Choi et al., 2018b;a; Jung et al., 2018) completely remove the round operation when differentiating the forward pass, equivalent in our derivation to removing the round function in equation 5, so that where |v/s| <…”
QKD: Quantization-aware Knowledge Distillation (2019, methodology): “Recent works have tried to improve quantization further by learning the range of weights and activation maps [6, 18, 8, 45].”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “To reduce the quantization error, existing methods [3, 9, 19, 28, 70, 72] explicitly parameterize the quantizer and train it jointly with network parameters.”
A compute-in-memory chip based on resistive random-access memory (2022, background): “A similar phenomenon has been reported in neural-network quantization literature where a model trained with quantization occasionally outperforms a full-precision model 54,55 .”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, positive): “To improve the accuracy, more advanced quantization schemes (Cai et al. 2017; Zhang et al. 2018; Jung et al. 2019) have been proposed.”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “Given the aforementioned limitations, we have chosen non-uniform quantization [9, 25, 34, 72, 83], which allows for a more flexible allocation of the representative values without any constraints.”
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks (2019, methodology): “Therefore, to alleviate the performance degradation, it is also important to find an appropriate clipping range and make a balance between clipping and rounding [7, 20].”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, background): “Due to its practical usefulness, various studies have been proposed for multi-bit linear quantization [13, 14, 35, 15].”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, background): “, 2017), or quantization interval learning (Jung et al., 2019).”
Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks (2019, methodology): “QIL (Jung et al., 2019) introduces a learnable transformer to change the quantization intervals and optimize them based on the task loss.”
Discrimination-Aware Network Pruning for Deep Model Compression (2020, methodology): “To improve the precision of the discrete uniform quantizer, [7], [32] explicitly parameterized and optimized the quantization intervals.”
Any-Precision Deep Neural Networks (2019, background): “(Jung et al. 2019) pointed out that having an uniform quantization pattern across layers is suboptimal and propose a learnable quantizer for each layer to improve the model accuracy.”
Forward and Backward Information Retention for Accurate Binary Neural Networks (2019, background): “Although much progress has been made on bina-rizing DNNs, the existing quantization methods remains a signiﬁcant drop of accuracy compared with the full-precision counterparts [14, 39, 2, 28, 57, 48, 9, 40, 43].”