Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks
By Aojun Zhou, Anbang Yao, Kuan Wang, Yurong Chen. 2018
93 citations recorded by the source corpus.
Benefiting from tens of millions of hierarchically stacked learnable parameters, Deep Neural Networks (DNNs) have demonstrated overwhelming accuracy on a variety of artificial intelligence tasks. However reversely, the large size of DNN models lays a heavy burden on storage, computation and power consumption, which prohibits their deployments on the embedded and mobile systems. In this paper, we propose Explicit Loss-error-aware Quantization (ELQ), a new method that can train DNN models with very low-bit parameter values such as ternary and binary ones to approximate 32-bit floating-point counterparts without noticeable loss of predication accuracy. Unlike existing methods that usually pose the problem as a straightforward approximation of the layer-wise weights or outputs of the original full-precision model (specifically, minimizing the error of the layer-wise weights or inner products of the weights and the inputs between the original and respective quantized models), our ELQ elaborately bridges the loss perturbation from the weight quantization and an incremental quantization strategy to address DNN quantization. Through explicitly regularizing the loss perturbation and the weight approximation error in an incremental way, we show that such a new optimization method is theoretically reasonable and practically effective. As validated with two mainstream convolutional neural network families (i.e., fully convolutional and non-fully convolutional), our ELQ shows better results than state-of-the-art quantization methods on the large scale ImageNet classification dataset. Code will be made publicly available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Similarity-Preserving Knowledge Distillation (2019, background): “Quantized networks [8, 15, 17, 30, 43, 46], such as fixed-point, binary, ternary, and arbitrary-bit networks, encode weights and/or activations using a small number of bits, or at lower precision.”
- A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Some work in the literature has also explored non-uniform quantization [25, 38, 62, 74, 78, 98, 117, 123, 150, 156, 175, 184, 185, 230, 240, 248, 256, 258, 268, 275], where quantization steps as well as quantization levels are allowed to be non-uniformly spaced.”
- HAQ: Hardware-Aware Automated Quantization With Mixed Precision (2018, methodology): “[32] binarized each convolution filter into {−w,+w}; Zhu et al.”
- Relaxed Quantization for Discretized Neural Networks (2018, unknown): “We compare against multiple works that employ fixed-point quantization: SR+DR (Gupta et al., 2015; Gysel et al., 2018), LR Net (Shayer et al., 2018), Jacob et al. (2017), TWN (Li et al., 2016), INQ (Zhou et al., 2017), BWN (Rastegari et al., 2016), XNOR-net (Rastegari et al., 2016), DoReFa (Zhou et a”
- MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning (2019, background): “There are extensive studies on compressing and accelerating neural networks, such as quantization [54, 43, 37, 23, 56, 57], pruning [22, 30, 16] and compact network design [24, 46, 55, 39, 29].”
- Fully Quantized Network for Object Detection (2019, background): “Most network quantization research has focused on object classification, including BNN [13], QNN [14], XNOR-Net [24], DoReFa-Net [35], INQ [33], ELQ [34], LQ-Nets [31] etc.”
- MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization (2019, background): “Although over-performance, ELQ is a combination of a series of previous quantization methods and tricks on incremental quantization.”