HAQ: Hardware-Aware Automated Quantization With Mixed Precision

By Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han. 2018

1,120 citations recorded by the source corpus.

Model quantization is a widely used technique to compress and accelerate deep neural network (DNN) inference. Emergent DNN hardware accelerators begin to support mixed precision (1-8 bits) to further improve the computation efficiency, which raises a great challenge to find the optimal bitwidth for each layer: it requires domain experts to explore the vast design space trading off among accuracy, latency, energy, and model size, which is both time-consuming and sub-optimal. There are plenty of specialized hardware for neural networks, but little research has been done for specialized neural network optimization for a particular hardware architecture. Conventional quantization algorithm ignores the different hardware architectures and quantizes all the layers in a uniform way. In this paper, we introduce the Hardware-Aware Automated Quantization (HAQ) framework which leverages the reinforcement learning to automatically determine the quantization policy, and we take the hardware accelerator's feedback in the design loop. Rather than relying on proxy signals such as FLOPs and model size, we employ a hardware simulator to generate direct feedback signals (latency and energy) to the RL agent. Compared with conventional methods, our framework is fully automated and can specialize the quantization policy for different neural network architectures and hardware architectures. Our framework effectively reduced the latency by 1.4-1.95x and the energy consumption by 1.9x with negligible loss of accuracy compared with the fixed bitwidth (8 bits) quantization. Our framework reveals that the optimal policies on different hardware architectures (i.e., edge and cloud architectures) under different resource constraints (i.e., latency, energy and model size) are drastically different. We interpreted the implication of different quantization policies, which offer insights for both neural network architecture design and hardware architecture design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, background): “It proves to be effective for various convolutional neural works (CNNs) (Han et al., 2016; Jacob et al., 2018; Nagel et al., 2019; Wang et al., 2019; Lin et al., 2020) and transformers (Shen et al., 2020; Kim et al., 2021; Liu et al., 2021; Wang et al., 2020; Bondarenko et al., 2021).”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “2017); (ii) automated neural architecture search (NAS) with reward function set as latency or model size (Wang et al. 2019); (iii) pruning based methods to reduce model size of existing architectures (LeCun, Denker, and Solla 1990; Hassibi, Stork, and Wolff 1994); (iv) knowledge distillation from a ”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “For example, one might want to use a more resource-efficient architecture for deployment on edge inference devices [44, 66, 76], or to adapt the network to the inference hardware [9, 63, 71], or for continually learning to classify new image classes [29, 34], etc.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “Finally, it is possible to assign different bit-widths for different layers or parts of the network, a technique known as mixed precision (Lin et al., 2016; Wu et al., 2018a; Zhou et al., 2018; Dong et al., 2019; Wang et al., 2019; van Baalen et al., 2020).”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “While many works [6, 7, 8, 9, 10] aim at reducing the memory footprint of classifiers and feature extractors, inference speed is equally important, with high throughput corresponding to better energy efficiency.”
AdaBits: Neural Network Quantization With Adaptive Bit-Widths (2019, methodology): “This suggests we might be able to employ the quantized models with adaptive bit-widths to search for bit-width in each layer or channel, for the purpose of mixed-precision quantization [10, 44, 42, 41, 26].”
Any-Precision Deep Neural Networks (2019, methodology): “[30] proposed to use reinforcement learning to determine the quantization policy.”
GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, background): “Extensive attention has been paid to hardware-efﬁcient deep learning for various real-world applications [18, 17, 68, 56, 16].”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, neutral): “The former, e.g. AMC [15] or HAQ [41], search for a layer-wise compression policy directly via, for example, reinforcement learning or genetic programming [43], whereas the latter, e.g. HAWQv3 [44] or AdaQuant [19], construct a relaxed version of the overall problem that is then solved exactly.”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, methodology): “Instead, HAQ would end-to-end search for each hardware performance threshold from scratch.”
Point-Voxel CNN for Efficient 3D Deep Learning (2019, background): “For instance, researchers have proposed to reduce the memory access cost by pruning and quantizing the models [7,8,9,24,39,49] or directly designing the compact models [11, 12, 14, 25, 34, 48].”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, background): “Arbitrary bit-widths used in some literature [29][30] are more difficult to exploit for power or computational savings unless dedicated silicon or FPGAs are employed.”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, background): “Most of the many academic works on quantization focus on computer vision [25, 31, 33, 38, 46, 53, 57] or natural language processing models [15, 57, 61, 62].”
Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help? (2019, background): “choice is to first train a network and then squeeze or compress it for a target deployment device [2, 9, 12, 37, 40].”