NICE: Noise Injection and Clamping Estimation for Neural Network Quantization

By Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Giryes, A. Mendelson, A. Bronstein. 2018

68 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) are very popular in many fields including computer vision, speech recognition, natural language processing, etc. Though deep learning leads to groundbreaking performance in those domains, the networks used are very computationally demanding and are far from being able to perform in real-time applications even on a GPU, which is not power efficient and therefore does not suit low power systems such as mobile devices. To overcome this challenge, some solutions have been proposed for quantizing the weights and activations of these networks, which accelerate the runtime significantly. Yet, this acceleration comes at the cost of a larger error unless spatial adjustments are carried out. The method proposed in this work trains quantized neural networks by noise injection and a learned clamping, which improve accuracy. This leads to state-of-the-art results on various regression and classification tasks, e.g., ImageNet classification with architectures such as ResNet-18/34/50 with as low as 3 bit weights and activations. We implement the proposed solution on an FPGA to demonstrate its applicability for low-power real-time applications. The quantization code will become publicly available upon acceptance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Compared with the NICE (Baskin et al. 2018a) and UNIQ (Baskin et al.”
Learned Step Size Quantization (2019, background): “…PACT (Choi et al., 2018b), Regularization (Choi et al., 2018c), and NICE (Baskin et al., 2018). stochastic gradient descent to learn a quantizer that minimizes task loss (Zhu et al., 2016; Mishra & Marr, 2017; Choi et al., 2018b;a; Jung et al., 2018; Baskin et al., 2018; Polino et al., 2018).”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, methodology): “More recently, NICE [24], PQ [25], and DiffQ [26] have proposed alternative QAT approaches that ﬁnetune the network with pseudo quantization noise (PQN) [27] instead of using STE.”
Loss aware post-training quantization (2019, methodology): “Following the convention [2, 24], we do not quantize the first and the last layer.”