PACT: Parameterized Clipping Activation for Quantized Neural Networks

By Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan. 2018

1,126 citations recorded by the source corpus.

Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost. To address this cost, a number of quantization schemeshave been proposed - but most of these techniques focused on quantizing weights, which are relatively smaller in size compared to activations. This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation. This technique, PArameterized Clipping acTi-vation (PACT), uses an activation clipping parameter α that is optimized duringtraining to find the right quantization scale. PACT allows quantizing activations toarbitrary bit precisions, while achieving much better accuracy relative to publishedstate-of-the-art quantization schemes. We show, for the first time, that both weights and activations can be quantized to 4-bits of precision while still achieving accuracy comparable to full precision networks across a range of popular models and datasets. We also show that exploiting these reduced-precision computational units in hardware can enable a super-linear improvement in inferencing performance dueto a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, methodology): “(2016)) to quantize weights and PACT (Choi et al. (2018)) to quantize activations.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…and activations in low precision format instead of 32-bit ﬂoating-point numbers (Courbariaux et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhou et al., 2016; Jacob et al., 2018; Choi et al., 2018; Dong et al., 2019; Esser et al., 2020; Chen et al., 2020a; Van Baalen et al., 2020).”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, methodology): “We use PACT [5] as the quantization algorithm.”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “…with the inverse operation: (2) where we compute the scale and bias as: We focus on this uniform rounding scheme instead of other non-uniform schemes (Choi et al., 2018; Li et al., 2019), because it allows for ﬁxed-point arithmetic with implementations in PyTorch and Tensorﬂow (see Appendix).”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, methodology): “For comparison, we test the quantization performance without using the Hessian information, which we refer to as “Direct” method, as well as other methods in the literature including Dorefa [43], PACT [2], LQ-Net [40], and DNAS [36], as shown in Table I.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Another approach is to learn/impose this clipping range during NN training [36, 144, 268, 278].”
HAQ: Hardware-Aware Automated Quantization With Mixed Precision (2018, methodology): “-5 Latency PACT [3] 4 bits 62.44 84.19 45.45 ms 61.39 83.72 52.15 ms 62.44 84.19 57.49 ms 61.39 83.72 74.46 ms Ours flexible 67.40 87.90 45.51 ms 66.99 87.33 52.12 ms 65.33 86.60 57.40 ms 67.01 87.46 73.97 ms our edge device and Xilinx VU9P [29] as our cloud device.”
Training Deep Neural Networks with 8-bit Floating Point Numbers (2018, background): “Recently, precision for weights and activations were scaled down to 1-2 bits ([11, 3]) with a small loss of accuracy, while keeping the gradients and errors in the backward path as well as the weight updates in full-precision.”
I-BERT: Integer-only BERT Quantization (2021, background): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
Learned Step Size Quantization (2019, methodology): “…data that occurs completely prior to the discretization itself (Jung et al., 2018), or estimate the gradient by removing the round operation from the forward equation, algebraically canceling terms, and then differentiating such that ∂ ˆ v / ∂s = 0 where − Q N < v / s < Q P (Choi et al., 2018b;a).”
HAWQV3: Dyadic Neural Network Quantization (2020, background): “As such, many prior quantization methods keep BN parameters in FP32 precision (e.g., (Cai et al., 2020; Chin et al., 2020; Choi et al., 2018; Dong et al., 2020; Park et al., 2018b; Zhang et al., 2018), just to name a few).”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, background): “Quantization techniques generally fall into two categories: quantization-aware training (QAT, which relies on backpropagation to update the quantized weights) [3, 16, 30, 7] and post-training quantization [19, 29, 28] (PTQ, usually training-free).”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, background): “One is quantization aware training (QAT), which tries to obtain through the training process an easy-to-quantize or already-quantized model (cf. [19, 33]).”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “To reduce the quantization error, existing methods [3, 9, 19, 28, 70, 72] explicitly parameterize the quantizer and train it jointly with network parameters.”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, neutral): “Note that the initial quantized weights and clipping levels for activation are determined us-ing conventional QDNN optimization methods (Jung et al. 2019; Choi et al. 2018).”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, methodology): “Also, because our framework applying in QAT is coupled with LSQ+ [12], we show the results of the pure LSQ+, and another canonical quantization approach PACT [41].”
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks (2019, methodology): “Besides, since DSQ can be conveniently inserted to any standard quantization method, we further investigate its performance over the state-of-the-art quantization method PACT [7].”
Discrimination-Aware Network Pruning for Deep Model Compression (2020, methodology): “To improve the precision of the discrete uniform quantizer, [7], [32] explicitly parameterized and optimized the quantization intervals.”