Resiliency of Deep Neural Networks under Quantization

By Wonyong Sung, Sungho Shin, Kyuyeon Hwang. 2015

171 citations recorded by the source corpus.

The complexity of deep neural network algorithms for hardware implementation can be much lowered by optimizing the word-length of weights and signals. Direct quantization of floating-point weights, however, does not show good performance when the number of bits assigned is small. Retraining of quantized networks has been developed to relieve this problem. In this work, the effects of retraining are analyzed for a feedforward deep neural network (FFDNN) and a convolutional neural network (CNN). The network complexity is controlled to know their effects on the resiliency of quantized networks by retraining. The complexity of the FFDNN is controlled by varying the unit size in each hidden layer and the number of layers, while that of the CNN is done by modifying the feature map configuration. We find that the performance gap between the floating-point and the retrain-based ternary (+1, 0, -1) weight neural networks exists with a fair amount in 'complexity limited' networks, but the discrepancy almost vanishes in fully complex networks whose capability is limited by the training data, rather than by the number of connections. This research shows that highly complex DNNs have the capability of absorbing the effects of severe weight quantization through retraining, but connection limited networks are less resilient. This paper also presents the effective compression ratio to guide the trade-off between the network size and the precision when the hardware resource is limited.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Multiple different metrics have been proposed to ﬁnd the best range, including minimizing Mean Squared Error (MSE) between original unquantized weight distribution and the corresponding quantized values [40, 214, 221, 273].”
Learned Step Size Quantization (2019, methodology): “This approach has been explored in a number of works as means to improve training of quantized networks (Sung et al., 2015; Zhou et al., 2016; Mishra & Marr, 2017; McKinstry et al., 2018).”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, background): “Therefore, to evaluate the performance of the quantization algorithm, the best choice might be employing a small network which is located in the under-parameterized region (Sung et al., 2015; Boo et al., 2019).”
Compact Deep Convolutional Neural Networks With Coarse Pruning (2016, methodology): “Network pruning is one promising techique that first learns a function with a suficiently large sized network followed by removing less important connections Yu et al. (2012); Han et al. (2015b); Anwar et al. (2015b).”
FINN: A Framework for Fast, Scalable Binarized Neural Network Inference (2016, methodology): “A tradeoff between network size, precision and accuracy exists [27] so if one would like to achieve a certain classification accuracy for a particular problem, which approach leads to the most efficient solution? 1) A regular ANN with floating point precision? 2) A larger network, but a BNN? To gain”
QKD: Quantization-aware Knowledge Distillation (2019, background): “Quantization Reducing the precision of neural networks [7, 25, 53, 32, 43] has been studied extensively due to its computational and storage benefits.”
Improving Neural Network Quantization without Retraining using Outlier Channel Splitting (2019, methodology): “This method chooses a clip threshold which minimizes the mean squared error (MSE) or L2-norm between the ﬂoating-point and quantized values (Sung et al., 2015; Shin et al., 2016).”
YodaNN: An Ultra-Low Power Convolutional Neural Network Accelerator Based on Binary Weights (2016, methodology): “Sev-1 eral reduced-precision implementations have been proposed recently, relying on 16-bit, 12-bit or 10-bit of accuracy for both operands and weights [10], [11], [12], [13], [14], exploiting the intrinsic resiliency of CNNs to quantization and approximation [15], [16].”
LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, methodology): “To overcome this, we propose initializing the scale and offset parameters per layer by optimizing the MSE minimization problem, similar to [24, 25]: where ˆ x is given by (2).”
FPGA-Based Accelerators of Deep Learning Networks for Learning and Classification: A Review (2019, background): “In line with the current trends towards compressed neural networks, with dramatically reduced weights and activations bit-width using 1-bit or 2-bit quantization [176]–[180],”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “Deep neural networks are also robust to noise [ Sung et al. , 2015; Merolla et al. , 2016 ] .”
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (2017, methodology): “Schemes like INQ, work in Sung et al. (2015) and Mellempudi et al. (2017) do ﬁne-tuning to quantize the network weights.”
A comprehensive survey on model compression and acceleration (2020, background): “In another study, Sung et al. (2015) observed some performance gap between floating-point and retraining-based ternary networks.”
YodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration (2016, methodology): “It was shown that state-of-the-art results can be achieved quantizing the weights and activations of each layer separately [33], while lowering precision down to 2-bit ( − 1, 0, + 1) and increasing the network size [20].”
Post-Training Quantization for Vision Transformer (2021, methodology): “Post-training quantization [24] is a kind of efficient model compression technique, which can directly quantize neural network models without fine-tuning.”
WRPN: Wide Reduced-Precision Networks (2017, background): “Schemes like INQ, [18] and [14] do ﬁne-tuning to quantize the network weights and do not sacriﬁce accuracy as much but are not applicable for training networks from scratch.”