HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision

By Zhen Dong, Zi-Mei Yao, A. Gholami, Michael W. Mahoney, Kurt Keutzer. 2019

686 citations recorded by the source corpus.

Model size and inference speed/power have become a major challenge in the deployment of neural networks for many applications. A promising approach to address these problems is quantization. However, uniformly quantizing a model to ultra-low precision leads to significant accuracy degradation. A novel solution for this is to use mixed-precision quantization, as some parts of the network may allow lower precision as compared to other layers. However, there is no systematic way to determine the precision of different layers. A brute force approach is not feasible for deep networks, as the search space for mixed-precision is exponential in the number of layers. Another challenge is a similar factorial complexity for determining block-wise fine-tuning order when quantizing the model to a target precision. Here, we introduce Hessian AWare Quantization (HAWQ), a novel second-order quantization method to address these problems. HAWQ allows for the automatic selection of the relative quantization precision of each layer, based on the layer's Hessian spectrum. Moreover, HAWQ provides a deterministic fine-tuning order for quantizing layers. We show the results of our method on Cifar-10 using ResNet20, and on ImageNet using Inception-V3, ResNet50 and SqueezeNext models. Comparing HAWQ with state-of-the-art shows that we can achieve similar/better accuracy with 8× activation compression ratio on ResNet20, as compared to DNAS, and up to 1% higher accuracy with up to 14% smaller models on ResNet50 and Inception-V3, compared to recently proposed methods of RVQuant and HAQ. Furthermore, we show that we can quantize SqueezeNext to just 1MB model size while achieving above 68% top1 accuracy on ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “4 Quantization stores model weights in low precision formats to (1) accelerate operations when using hardware with reduced precision support and (2) reduce overall memory footprint (Han et al., 2016; Dong et al., 2019).”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For example, using Hessian information (Dong et al., 2019), step-size quantization (Esser et al., 2019), soft quantization (Gong et al., 2019), mixed-precision via linear programming optimization (Yao et al., 2021), and other learned quantization methods (Zhang et al., 2018; Gholami et al., 2021).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “In [8], a Hessian AWare Quantization (HAWQ) is developed for mixed-bits assignments.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…and activations in low precision format instead of 32-bit ﬂoating-point numbers (Courbariaux et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhou et al., 2016; Jacob et al., 2018; Choi et al., 2018; Dong et al., 2019; Esser et al., 2020; Chen et al., 2020a; Van Baalen et al., 2020).”
Differentiable Model Compression via Pseudo Quantization Noise (2021, neutral): “, 2021), second-order optimization (Dong et al., 2019; 2020; Yao et al., 2021), and differentiable quantization methods (Uhlich et al.”
The hardware lottery (2020, background): “, 2019) and weight specific quantization (Zhen et al., 2019) are very successful compression techniques in deep neural networks but are incompatible with current hardware and compilations kernels.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “Finally, it is possible to assign different bit-widths for different layers or parts of the network, a technique known as mixed precision (Lin et al., 2016; Wu et al., 2018a; Zhou et al., 2018; Dong et al., 2019; Wang et al., 2019; van Baalen et al., 2020).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Different than these exploration and regularization-based approaches, HAWQ [51] introduces an automatic way to ﬁnd the mixed-precision settings based on second-order sensitivity of the model.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, background): “…(Liu et al., 2019; Wang et al., 2020), common approaches of mixed-precision quantization are primarily based on differentiable search (Wu et al., 2018a; Li et al., 2020b), reinforcement learning (Wu et al., 2018b; Wang et al., 2019), or simply loss curvatures (Dong et al., 2019; Shen et al., 2020).”
I-BERT: Integer-only BERT Quantization (2021, methodology): “One promising method to tackle this challenge is quantization [15, 27, 28, 65, 66, 72], a procedure which compresses models into smaller size by representing parameters and/or activations with low bit precision, e.”
Any-Precision Deep Neural Networks (2019, methodology): “[9] presented an novel second-order quantization method to select the bit-width of each layer as well as the ﬁne-tuning order of layers, based on the layer’s Hessian spectrum.”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, methodology): “…quantization has leveraged the largest Hessian eigenvalue or the Hessian trace to assess which layers are most sensitive to quantization [8, 9]; however, due to the computational challenges of computing the full Hessian or even estimating Hessian eigenvalues, we instead leverage the Fisher…”
Pareto-Optimal Quantized ResNet Is Mostly 4-bit (2021, methodology): “To address the quality degradation, quantization-aware training (QAT) has been proposed and applied in several papers [6, 8, 32] and was also our method of choice.”
Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers (2019, methodology): “The HAWQ [5] method relies on a second order Hessian metric to deﬁne prioritization of tensor’s bit precision to reduce, but without choosing the optimal per-tensor quantization level.”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “It is possible to avoid this prohibitive search space if we could measure the sensitivity of the model to the quantization of each layer [8, 37, 7].”