Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks

By Albert Gural, Michael Wu, Chris Dick. 2019

171 citations recorded by the source corpus.

We propose a method of training quantization thresholds (TQT) for uniform symmetric quantizers using standard backpropagation and gradient descent. Contrary to prior work, we show that a careful analysis of the straight-through estimator for threshold gradients allows for a natural range-precision trade-off leading to better optima. Our quantizers are constrained to use power-of-2 scale-factors and per-tensor scaling of weights and activations to make it amenable for hardware implementations. We present analytical support for the general robustness of our methods and empirically validate them on various CNNs for ImageNet classification. We are able to achieve near-floating-point accuracy on traditionally difficult networks such as MobileNets with less than 5 epochs of quantized (8-bit) retraining. Finally, we present Graffitist, a framework that enables automatic quantization of TensorFlow graphs for TQT (available at this https URL ).

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “For example, there are methods associated with blocking and grouping (Park et al., 2022; Wu et al., 2020; Jain et al., 2020; Nagel et al., 2019; Krishnamoorthi, 2018; Rusci et al., 2020), centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering…”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, unknown): “By training quantization thresholds, TQT (Jain et al. 2019) also achieves near-ﬂoating-point accuracy.”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “On the other hand, compared with previous ﬁxed-point quantization (Jain et al., 2019), our approach gives better results.”
QKD: Quantization-aware Knowledge Distillation (2019, methodology): “So, we introduce two trainable parameters for the interval values of each layer’s weight parameters and input activations similar to [8, 17, 45].”
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, methodology): “Particularly, note that for 2-bit weight quantization of TinyBERT, KDLSQ-BERT achieves only about 1.5% accuracy drop on average while obtaining 23.2x compression ratio compared with BERT-base.”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “This has been studied by, for instance, Choi et al. (2019), Jain et al. (2018, 2019), Zhang et al. (2016).”
LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, methodology): “Several recent papers improve over these methods by learning the quantization parameters, e.g. QIL [14], TQT [13] and LSQ [7].”
Bayesian Bits: Unifying Quantization and Pruning (2020, background): “Learning the scale along with the model parameters for a ﬁxed bit width network was independently introduced by (Esser et al., 2020) and Jain et al. (2019).”
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation (2020, background): “This is commonly referred to as “per-channel” weight quantization in literature and we follow that convention [21, 25, 26, 38, 46].”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, methodology): “If still not successful, we need to incorporate other techniques such as quantization aware training [20, 33, 38], equalization [46], or model codesign.”