Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks

By Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tian-Hao Li, Peng Hu, Jiazhen Lin, Fisher Yu, Junjie Yan. 2019

545 citations recorded by the source corpus.

Hardware-friendly network quantization (e.g., binary/uniform quantization) can efﬁciently accelerate the inference and meanwhile reduce memory consumption of the deep neural networks, which is crucial for model deployment on resource-limited devices like mobile phones. However, due to the discreteness of low-bit quantization, existing quantization methods often face the unstable training process and severe performance degradation. To address this problem, in this paper we propose Differentiable Soft Quantization (DSQ) to bridge the gap between the full-precision and low-bit networks. DSQ can automatically evolve during training to gradually approximate the standard quantization. Owing to its differentiable property, DSQ can help pursue the accurate gradients in backward propagation, and reduce the quantization loss in forward process with an appropriate clipping range. Extensive experiments over several popular network structures show that training low-bit neural networks with DSQ can consistently outperform state-of-the-art quantization methods. Besides, our ﬁrst efﬁcient implementation for deploying 2 to 4-bit DSQ on devices with ARM architecture achieves up to 1.7× speed up, compared with the open-source 8-bit high-performance inference framework NCNN [31].

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Versions of 8-bit BERT/RoBERTa include Q8BERT (Zafrir et al., 2019), QBERT (Shen et al., 2020), product quantization with quantization noise (Fan et al., 2020), TernaryBERT (Zhang et al., 2020), and BinaryBERT (Bai et al., 2021).”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “, 2014), or a variable amount of bits (Gong et al., 2019).”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Gong et al. (2019) use a meta-network to provide gradients despite quantization.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…models has also been shown as a promising method to recover the accuracy degradation after binarization/ternarization [33, 173, 189, 252]. c) Improved Training Method: Another interesting branch of work aims for better training methods for binary/ternary models [5, 20, 44, 73, 157, 160, 276, 279].”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “One area uses uniform-precision quantization where the model shares the same precision (Zhou et al., 2018; Wang et al., 2018; Choukroun et al., 2019; Gong et al., 2019; Langroudi et al., 2019; Jin et al., 2020a; Bhalgat et al., 2020; Chen et al., 2020; Yang et al., 2020; Darvish Rouhani et al.,…”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “To reduce the optimization difficulty incurred by the non-differentiable discretization, extensive methods [13, 21, 30, 35, 71] have been proposed to approximate the gradients.”
Loss aware post-training quantization (2019, background): “…of results comparable to a baseline for as low as 2–4 bits per parameter (Baskin et al. 2018; Zhang et al. 2018; Gong et al. 2019; Yang et al. 2019; Jin et al. 2019) and show decent performance even for single bit (binary) parameters (Liu et al. 2019; Peng and Chen 2019; Kim et al. 2020).”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, methodology): “Compared to uniform quantization, N2UQ allows more flexibility in learning input thresholds, and the results demonstrate that N2UQ surpasses uniform methods [8, 12, 14, 50] by a large margin.”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, methodology): “Bi-Real net [18] and DSQ [19] show that replacing STE with high-order polynomial or multiple soft-step functions results in better convergence in low-precision representations.”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, result): “On ImageNet, we also observe in ResNet50 and MobileNetV2 that INT4 in the forward path by itself causes 0.7% and 2.0% loss in accuracy respectively—which we expect can be signiﬁcantly improved using recently published techniques [20, 23].”
QKD: Quantization-aware Knowledge Distillation (2019, methodology): “DSQ [10] shows results on MobileNetV2 at W4A4.”
Forward and Backward Information Retention for Accurate Binary Neural Networks (2019, methodology): “Table 3 lists the performance using different methods on CIFAR-10, including RAD [13] over ResNet-18 (based on [31]), DoReFa-Net [61], LQ-Net [59], DSQ [18] over ResNet-20 (based on [3]) and BNN [27], LAB [25], RAD, XNOR-Net [44] over VGG-Small.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “Activation is more sensitive to numerical precision [70].”
Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks (2019, methodology): “Most of the existing quantization approaches (Cai et al., 2017; Gong et al., 2019) use uniform quantization although non-uniform quantization can usually achieve better”
Binary Neural Networks: A Survey (2020, methodology): “Following the same intuition, Gong et.al. present a Diﬀerential Soft Quantization (DSQ) method [63], replacing the traditional quantization function with a soft quantization function: where k determines the shape of the asymptotic function, s is a scaling factor to make the soft quantization functio”
Towards Unified INT8 Training for Convolutional Neural Network (2019, background): “The successful progress surprisingly shows that the bit-width can be reduced to extremely low such as 4-bit while bringing quite little hurt to the accuracy for inference [15, 57, 13].”
LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, background): “Also, Con-ﬁgurations 1, 3 and 4 outperform existing state-of-the-art methods, namely PACT [5], DSQ [8] and QIL [14].”
Overcoming Oscillations in Quantization-Aware Training (2022, unknown): “Using a constant threshold throughout training, we see that the amount of remaining 3) PACT (Choi et al., 2018) 4/4 61.4 (-10.3) DSQ (Gong et al., 2019) 4/4 64.8 (-6.9) EWGS (J. Lee, 2021) 4/4 70.3 (-1.6) LSQ + BR (Han et al., 2021) 4/4 70.4 (-1.4) LSQ + Dampen (ours) 4/4 70.5 (-1.2) LSQ + Freeze…”