Quantization Networks
By Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, Xiansheng Hua. Computer Vision and Pattern Recognition, 2019
426 citations recorded by the source corpus.
Although deep neural networks are highly effective, their high computational and memory costs severely hinder their applications to portable devices. As a consequence, lowbit quantization, which converts a full-precision neural network into a low-bitwidth integer version, has been an active and promising research topic. Existing methods formulate the low-bit quantization of networks as an approximation or optimization problem. Approximation-based methods confront the gradient mismatch problem, while optimizationbased methods are only suitable for quantizing weights and can introduce high computational cost during the training stage. In this paper, we provide a simple and uniform way for weights and activations quantization by formulating it as a differentiable non-linear function. The quantization function is represented as a linear combination of several Sigmoid functions with learnable biases and scales that could be learned in a lossless and end-to-end manner via continuous relaxation of the steepness of Sigmoid functions. Extensive experiments on image classification and object detection tasks show that our quantization networks outperform state-of-the-art methods. We believe that the proposed method will shed new lights on the interpretation of neural network quantization.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/86284d298539db8dc499a99dcd2b633b035dc713
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “To reduce the optimization difficulty incurred by the non-differentiable discretization, extensive methods (Ding et al., 2019; Yang et al., 2019; Gong et al., 2019; Lee et al., 2021; Kim et al., 2021a) have been proposed to approximate the gradients.”
- A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Some work in the literature has also explored non-uniform quantization [25, 38, 62, 74, 78, 98, 117, 123, 150, 156, 175, 184, 185, 230, 240, 248, 256, 258, 268, 275], where quantization steps as well as quantization levels are allowed to be non-uniformly spaced.”
- Binary Neural Networks: A Survey (2020, methodology): “A similar method [93] also provides a simple and uniform way for weight and activation quantization by formulating it as a differentiable non-linear function.”
- Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, positive): “Net (60.8) ResNet18 (70.3) ResNet34 (73.6) Method AlexNet (60.8) ResNet18 (70.3) DoReFay(Zhou et al. 2016) 46.4 62.6 - BalancedQ (Zhou et al. 2017) 56.5 61.1 QIL (Jung et al. 2019) 58.1 65.7 70.6 QN (Yang et al. 2019) - 65.7 PACT SWAB (Choi et al. 2019) 57.2 67.0 - DoReFay(Zhou et al. 2016) 54.1 66.”
- Towards Unified INT8 Training for Convolutional Neural Network (2019, background): “The successful progress surprisingly shows that the bit-width can be reduced to extremely low such as 4-bit while bringing quite little hurt to the accuracy for inference [15, 57, 13].”