HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

By Zhen Dong, Zi-Mei Yao, Daiyaan Arfeen, A. Gholami, Michael W. Mahoney, Kurt Keutzer. 2019

394 citations recorded by the source corpus.

Quantization is an effective method for reducing memory footprint and inference time of Neural Networks, e.g., for efficient inference in the cloud, especially at the edge. However, ultra low precision quantization could lead to significant degradation in model generalization. A promising method to address this is to perform mixed-precision quantization, where more sensitive layers are kept at higher precision. However, the search space for a mixed-precision quantization is exponential in the number of layers. Recent work has proposed HAWQ, a novel Hessian based framework, with the aim of reducing this exponential search space by using second-order information. While promising, this prior work has three major limitations: (i) HAWQV1 only uses the top Hessian eigenvalue as a measure of sensitivity and do not consider the rest of the Hessian spectrum; (ii) HAWQV1 approach only provides relative sensitivity of different layers and therefore requires a manual selection of the mixed-precision setting; and (iii) HAWQV1 does not consider mixed-precision activation quantization. Here, we present HAWQV2 which addresses these shortcomings. For (i), we perform a theoretical analysis showing that a better sensitivity metric is to compute the average of all of the Hessian eigenvalues. For (ii), we develop a Pareto frontier based method for selecting the exact bit precision of different layers without any manual selection. For (iii), we extend the Hessian analysis to mixed-precision activation quantization. We have found this to be very beneficial for object detection. We show that HAWQV2 achieves new state-of-the-art results for a wide range of tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “In HAWQv2, this method was extended to mixed-precision activation quantization [50], and was shown to be more than 100x faster than RL based mixed-precision methods [238].”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, background): “er output with respect to d, qm and t are given by ▽dxq = sign(x) ⌊(|x|−q s) t d ⌉ − (|x|−q )t d , qs ≤ |x| ≤ qm sign(x) ⌊(q m−q s) t d ⌉− (q m−q s) t d , |x| > qm, 0 otherwise (5) ▽q mxq = ( 0, |x| ≤ qm sign(x)t(qm −qs)t−1, otherwise (6) ▽txq = sign(x)(|x| −qs)t log(|x| −qs), qs ≤ |x”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, methodology): “Recently, the Hessian of the loss function has often been used as a metric of sensitivity against quantization [31, 32, 33].”
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020, background): “Some examples include Hessian based model compression [18, 19, 24, 31], adversarial attacks [68], and studies of the loss landscape topology for different NN architectures [52, 66], to name just a few.”
HAWQV3: Dyadic Neural Network Quantization (2020, methodology): “However, the Pareto frontier method in [14] is not flexible enough to satisfy simultaneously different requirements on hardware.”
Post-Training Quantization for Vision Transformer (2021, methodology): “Inspired by the method in [10], we utilize a Pareto frontier approach to determine the bit-width.”
Bayesian Bits: Unifying Quantization and Pruning (2020, methodology): “NB: we cannot directly compare our results to those of [36], [7; 6] and [38], for reasons outlined in the Appendix, and therefore omit these results in this paper.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “To this end, various DNN compression techniques have been proposed such as [3, 5, 13, 15, 19, 26].”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, background): “To reduce the performance gap between the quantized model and its full precision counterpart, quantization-aware training (QAT) fine-tunes the quantized model on the training dataset [6, 7, 8, 9, 10, 11, 12, 13, 14].”
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (2022, neutral): “While they are many ways for model compression, such as quantization [38, 39, 40] and pruning [41, 42], our current efforts focus on layer reduction through knowledge distillation [43] (KD) – reducing both model size and model computation, and preserving MoE structure at student model.”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “It is possible to avoid this prohibitive search space if we could measure the sensitivity of the model to the quantization of each layer [8, 37, 7].”
SqueezeLLM: Dense-and-Sparse Quantization (2023, background): “The quantization objective is to represent the model weights with low-bit precision with minimal perturbation in the model output [13].”
CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs (2021, background): “Many prior works have attempted to address this accuracy drop with various techniques, such as non-uniform learnable quantizer [39], mixedprecision quantization [7], progressive fine-tuning [42] as well as group-wise [34] and channel-wise quantization [16].”
QuIP: 2-Bit Quantization of Large Language Models With Guarantees (2023, methodology): “[6] Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W.”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “To reduce the performance drop, one may use mixed-precision quantization that assigns different bitwidths according to the quantization sensitivity of each layer [11, 57].”