Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks

By Yuhang Li, Xin Dong. International Conference on Learning Representations, 2019

334 citations recorded by the source corpus.

We propose Additive Powers-of-Two~(APoT) quantization, an efficient non-uniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks. By constraining all quantization levels as the sum of Powers-of-Two terms, APoT quantization enjoys high computational efficiency and a good match with the distribution of weights. A simple reparameterization of the clipping function is applied to generate a better-defined gradient for learning the clipping threshold. Moreover, weight normalization is presented to refine the distribution of weights to make the training more stable and consistent. Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models, demonstrating the effectiveness of our proposed APoT quantization. For example, our 4-bit quantized ResNet-50 on ImageNet achieves 76.6% top-1 accuracy without bells and whistles; meanwhile, our model reduces 22% computational cost compared with the uniformly quantized counterpart.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/973404c13a5d50469182b76f299860bf71c12fb3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “For instance, many non-uniform quantization methods have been designed for bell-shaped distributions of the weights and activations that often involve long tails [12, 25, 61, 114, 145, 175].”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, methodology): “Quantization can be further classified to uniform quantization [7, 8, 15, 21, 22, 52] and nonuniform quantization [28, 36, 48, 49].”
Deep Neural Networks are Surprisingly Reversible: A Baseline for Zero-Shot Inversion (2021, background): “Albeit being important, DNNs’ invertibility remains an ill-posed and challenging problem [19]: it highly depends on model architecture [20], weights distribution [21], and learning objectives [22].”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Following APOT [37], we use weight normalization before quantization.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “Model quantization: Besides clustering and regularization methods, model quantization can also reduce the model size, and training-time quantization techniques have been developed to improve the accuracy of quantized models (Li et al., 2019; Zhao et al., 2019).”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, background): “Apart from uniform quantization, some works like Li et al. (2019) argue that non-uniform quantization has better performance than uniform quantization while keeping its efficiency.”