WRPN: Wide Reduced-Precision Networks

By Asit K. Mishra, Eriko Nurvitadhi, Jeffrey J. Cook, Debbie Marr. 2017

278 citations recorded by the source corpus.

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

Citations in the corpus

Model compression via distillation and quantization (2018, background): “We also note that, in line with previous work on this dataset (Zhu et al., 2016; Mishra et al., 2017), we do not quantize the first and last layers of the models, as this can hurt accuracy.”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “To improve accuracies, higher bitwidth quantization methods were studied (Jung et al. 2019; Zhang et al. 2018a; Mishra et al. 2017).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…each branch, and we refer the interested readers to [191] for more details. a) Quantization Error Minimization: The ﬁrst branch of solutions aims to minimize the quantization error, i.e., the gap between the real values and the quantized values [19, 34, 62, 102, 149, 155, 160, 165, 174, 211, 240].”
Quantizing deep convolutional networks for efficient inference: A whitepaper (2018, background): “Note that this comparison allows us to evaluate a depth vs quantization tradeoff (see [31]).”
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (2017, methodology): “Further, the memory and compute requirements during training and inference are quite different (Mishra et al., 2017).”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “…most widely-used techniques for neural network compression (Courbariaux et al., 2015; Han et al., 2015; Zhu et al., 2016; Zhou et al., 2016; 2017; Mishra et al., 2017; Park et al., 2017; Banner et al., 2018), with two types of training strategies: Post-Training Quantization directly quantizes a…”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “In particular, increasing the number of parameters in CNNs reduces the quantization sensitivity (Mishra et al. 2017).”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, background): “Nevertheless, networks with binary activations usually show degraded test performance, which can partly be compensated by modifying the network structure, e.g., increasing their width (Mishra et al., 2017).”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Binarization not only reduces the network size by up-to 32×, but also drastically reducesmemory usage resulting in significantly lower energy consumption [164, 106].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Although more bits [144], [147], [169], [182] and wider layers [170], [174] can help to improve accuracy, they will increase parameters and operations as well.”
Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications (2018, background): “…losses from quantization by making DL models wider is an example of hardware cost aware trade-offs: int8 multiplication consumes more than 5 × less energy and area compared to fp16 multiplication, hence there is a big room to recover the accuracy while maintaining the energy savings [14, 44].”
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (2019, background): “In addition, for low-precision inference, ﬁxed point techniques involving costly retraining of networks for ultra-short bit-widths [11, 12, 13] as well as post-training quantization for simpler deployment of the INT8/INT4 inference models [7, 14, 15, 16] have been extensively explored, but the…”
AdaBits: Neural Network Quantization With Adaptive Bit-Widths (2019, methodology): “Initially, uniform precision quantization is adopted inside the whole network, where all layers share the same bit-width [17, 19, 28, 31, 32, 33, 46, 52].”
Scalable Methods for 8-bit Training of Neural Networks (2018, methodology): “Recently, Mishra et al. [20] showed that this accuracy loss can be prevented by merely increasing the number of ﬁlter maps in each layer, thus suggesting that quantized neural networks (QNNs) do not possess an inherent convergence problem.”
And the Bit Goes Down: Revisiting the Quantization of Neural Networks (2019, unknown): “Those approaches include training with binary or ternary weights [22, 23, 24, 25], learning a combination of binary bases [26] and quantizing the activations [27, 28, 29].”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “In [Mishra et al. , 2017], the authors proposed wide reduced-precision networks (WRPN) to quantize activation and weights.”
QKD: Quantization-aware Knowledge Distillation (2019, unknown): “The original Ap-prentice uses WRPN scheme [31].”
Binary Neural Networks: A Survey (2020, methodology): “With a 3 × wider structure, the accuracy loss of ResNet-34 in [75] is lower than 1%.”