Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search

By Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Péter Vajda, Kurt Keutzer. 2018

313 citations recorded by the source corpus.

Recent work in network quantization has substantially reduced the time and space complexity of neural network inference, enabling their deployment on embedded and mobile devices with limited computational and memory resources. However, existing quantization methods often represent all weights and activations with the same precision (bit-width). In this paper, we explore a new dimension of the design space: quantizing different layers with different bit-widths. We formulate this problem as a neural architecture search problem and propose a novel differentiable neural architecture search (DNAS) framework to efficiently explore its exponential search space with gradient-based optimization. Experiments show we surpass the state-of-the-art compression of ResNet on CIFAR-10 and ImageNet. Our quantized models with 21.1x smaller model size or 103.9x lower computational cost can still outperform baseline quantized or even full precision models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Neural Architecture Search & AutoML

Citations in the corpus

FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search (2018, methodology): “The technique of using Gumbel Softmax for neural architecture search is also proposed in [25, 1].”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, methodology): “Following [36, 5, 27], we only search and quantize the res-blocks, excluding the first convolutional layer and the last fully-connected layer.”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, methodology): “In particular, we showed results for ResNet20 on Cifar-10, where we can achieve similar testing performance as [39], but with 8× higher compression ratio for activations.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “…of the most common range estimators include: current min-max or simply min-max , uses the full dynamic range of the tensor (Zhou et al., 2016; Wu et al., 2018b; Zhu et al., 2020); running min-max uses exponential moving average of the min and max over multiple batches (Krishnamoorthi, 2018);…”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “It is important to consider this fact for achieving optimal beneﬁts through hardware-aware quantization [86, 90, 238, 242, 246, 248, 257, 259].”
I-BERT: Integer-only BERT Quantization (2021, methodology): “One promising method to tackle this challenge is quantization [15, 27, 28, 65, 66, 72], a procedure which compresses models into smaller size by representing parameters and/or activations with low bit precision, e.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “Many neural nets have been targeted to achieve eﬃciency, including SqueezeNet [16,10,54], MobileNets [12,39,11], ShiftNet [55,61], ShuﬄeNet [63,28], FBNet [53,57], ChamNet [7], MnasNet [44], and EﬃcientNet [45].”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, background): “Mixed precision quantization [32, 36, 43] and multi-stage quantization [41] have been proposed to solve/alleviate this problem.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Recent approaches quantize both weights and activations through using uniform quantization that uses identical bit-width for all layers [66, 8, 39], or mixed precision quantization that uses different bit-widths for different layers or even channels [51, 4, 56].”
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (2019, methodology): “…exponential function of the number of blocks in a given network, we propose HAWQ-V2 to generate precision settings automatically based on Hessian information instead of using simple search methods [22, 23]. setting with good generalization and hardware performance (in terms of latency and power).”
Bayesian Bits: Unifying Quantization and Pruning (2020, result): “ made in doing so can be found in the appendix. We note that the bit widths of all quantizers in our network are learned, contrary to common practice in quantization literature (e.g.Choi et al.(2018);Wu et al. (2019) which keeps the ﬁrst and last layers of the networks in full precision or a higher ”
HAWQV3: Dyadic Neural Network Quantization (2020, methodology): “HAQ (Wang et al., 2019) proposes to search this space by applying a Reinforcement Learning algorithm, while (Wu et al., 2018) uses a Differentiable Neural Architecture Search.”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “A promising ap-3 proach to address this is to perform mixed-precision quantization [8, 7, 40, 47, 39], where different bit-precision is used for different layers.”
AdaBits: Neural Network Quantization With Adaptive Bit-Widths (2019, methodology): “This suggests we might be able to employ the quantized models with adaptive bit-widths to search for bit-width in each layer or channel, for the purpose of mixed-precision quantization [10, 44, 42, 41, 26].”