AdaBits: Neural Network Quantization With Adaptive Bit-Widths

By Qing Jin, Linjie Yang, Zhenyu A. Liao. 2019

152 citations recorded by the source corpus.

Deep neural networks with adaptive configurations have gained increasing attention due to the instant and flexible deployment of these models on platforms with different resource budgets. In this paper, we investigate a novel option to achieve this goal by enabling adaptive bit-widths of weights and activations in the model. We first examine the benefits and challenges of training quantized model with adaptive bit-widths, and then experiment with several approaches including direct adaptation, progressive training and joint training. We discover that joint training is able to produce comparable performance on the adaptive model as individual models. We also propose a new technique named Switchable Clipping Level (S-CL) to further improve quantized models at the lowest bit-width. With our proposed techniques applied on a bunch of models including MobileNet V1/V2 and ResNet50, we demonstrate that bit-width of weights and activations is a new option for adaptively executable deep neural networks, offering a distinct opportunity for improved accuracy-efficiency trade-off as well as instant adaptation according to the platform constraints in real-world applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alan L. Yuille
2. Shiji Song
3. Gao Huang
4. Jianfei Cai
5. Cihang Xie
6. Yulin Wang
7. Song Bai
8. Yingwei Li

In Computer Vision and Pattern Recognition

1. Alan L. Yuille
2. Jianfei Cai
3. Luc Van Gool
4. Dacheng Tao
5. Shiji Song
6. Yi Yang
7. Xiaogang Wang
8. Gao Huang

In Efficiency & Compression

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Bohan Zhuang
5. Jianfei Cai
6. Jing Liu
7. Le Yang
8. Yizeng Han

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “Instead of either skipping a layer, or executing it thoroughly with a full numerical precision, a line of work [68], [69] studies adaptive bit-width for different layers conditioned on the resource budget.”
• A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “As such, there is a large body of work that has proposed different solutions to address this [18, 25, 47, 52, 77, 82, 91, 92, 118, 120, 122, 127, 129, 133, 139, 147, 152, 157, 190, 192, 210, 241, 243, 252, 254, 279, 281].”
• F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, background): “…quantization where the model shares the same precision (Zhou et al., 2018; Wang et al., 2018; Choukroun et al., 2019; Gong et al., 2019; Langroudi et al., 2019; Jin et al., 2020a; Bhalgat et al., 2020; Chen et al., 2020; Yang et al., 2020; Darvish Rouhani et al., 2020; Oh et al., 2021).”
• Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “To tackle this problem, we adopt any-precision recognition [27, 65] that makes a single model be flexible to any numerical precision during the inference.”
• Neural Architecture Search for Lightweight Non-Local Networks (2020, background): “Recently, it has received much attention to use neural architecture search (NAS) to design efficient network architectures for various applications [34, 13, 23, 43, 21].”
• Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Specifically, we obtain the quantized models of different bitwidths by performing quantization on the shared full-precision weights w following AdaBits [25].”