Deep neural networks are robust to weight binarization and other non-linear distortions

By Paul A. Merolla, Rathinakumar Appuswamy, John V. Arthur, Steven K. Esser, Dharmendra S. Modha. 2016

97 citations recorded by the source corpus.

Recent results show that deep neural networks achieve excellent performance even when, during training, weights are quantized and projected to a binary representation. Here, we show that this is just the tip of the iceberg: these same networks, during testing, also exhibit a remarkable robustness to distortions beyond quantization, including additive and multiplicative noise, and a class of non-linear projections where binarization is just a special case. To quantify this robustness, we show that one such network achieves 11% test error on CIFAR-10 even with 0.68 effective bits per weight. Furthermore, we find that a common training heuristic--namely, projecting quantized weights during backpropagation--can be altered (or even removed) and networks still achieve a base level of robustness during testing. Specifically, training with weight projections other than quantization also works, as does simply clipping the weights, both of which have never been reported before. We confirm our results for CIFAR-10 and ImageNet datasets. Finally, drawing from these ideas, we propose a stochastic projection rule that leads to a new state of the art network with 7.64% test error on CIFAR-10 using no data augmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Merolla et al. (2016) showed that DNN can be robust to more than just weight binarization.”
Bayesian Compression for Deep Learning (2017, background): “Several studies show that this has little to no effect on network accuracy when using 16bit weights [Merolla et al., 2016, Gupta et al., 2015, Courbariaux et al., 2014, Venkatesh et al., 2016, Chai et al., 2017].”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “Recent research efforts (Courbariaux et al., 2014; Kim & Smaragdis, 2016; Rastegari et al., 2016; Li & Liu, 2016; Merolla et al., 2016) have considerably reduced both model size and computation complexity by using low bitwidth weights and low bitwidth activations.”
Soft Weight-Sharing for Neural Network Compression (2017, background): “Merolla et al. (2016) demonstrates that neural networks are robust against certain amounts of low precision; indeed several groups have exploited this and showed that decreasing the weight encoding precision has little to no effect on the accuracy loss (Gupta et al., 2015; Courbariaux et al., 2014;…”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, background): “Similarly, imposing constraints on the network weights through weight clipping has also been shown to improve robustness [188], [189].”
YodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration (2016, unknown): “Similarly to the scaling by the batch normalization, Merolla et al. [36] evaluated different weight projection functions where the accuracy could even be improved from 89% to 92% on Cifar-10 when binarizing weights and scaling every output channel by the maximum-absolute value of all contained…”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, background): “…of activations (e.g., Hubara et al., 2016), modiﬁcations of regularizers (Tang et al., 2017), gradual transitions from soft to hard binarizations (Wu et al., 2015), adding noise on activations and weights (Merolla et al., 2016), and knowledge distillation (Wu et al., 2016; Polino et al., 2018).”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “It has been shown that the backward propagation fine-tunes (retrains) a quantized network and can recover losses in accuracy caused by the quantization process [161].”
Stein Neural Sampler (2018, unknown): “If we use Gaussian kernel or other popular kernels, we can still ensure weak convergence if we enforce uniformly tightness (Merolla et al., 2016).”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, background): “The systematic study in [16] showed that networks trained with backpropagation could be robust against (robust against or resilient to) specific weight distortions, including binary weights.”
A Survey of Model Compression and Acceleration for Deep Neural Networks (2017, background): “The systematic study in [15] showed that networks trained with back propogation could be resilient to speciﬁc weight distortions, including binary weights.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “Deep neural networks are also robust to noise [ Sung et al. , 2015; Merolla et al. , 2016 ] .”
Accurate deep neural network inference using computational phase-change memory (2019, methodology): “In fact, quantization of the weights or activations 30 , and injecting noise on the weights 31 , activations 32 or gradients 33 have been widely used as DNN regularizers during training to reduce overﬁtting on the training dataset 34,35 .”