Forward and Backward Information Retention for Accurate Binary Neural Networks

By Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fisher Yu, Jingkuan Song. Computer Vision and Pattern Recognition, 2019

399 citations recorded by the source corpus.

Weight and activation binarization is an effective approach to deep neural network compression and can accelerate the inference by leveraging bitwise operations. Although many binarization methods have improved the accuracy of the model by minimizing the quantization error in forward propagation, there remains a noticeable performance gap between the binarized model and the full-precision one. Our empirical study indicates that the quantization brings information loss in both forward and backward propagation, which is the bottleneck of training accurate binary neural networks. To address these issues, we propose an Information Retention Network (IR-Net) to retain the information that consists in the forward activations and backward gradients. IR-Net mainly relies on two technical contributions: (1) Libra Parameter Binarization (Libra-PB): simultaneously minimizing both quantization error and information loss of parameters by balanced and standardized weights in forward propagation; (2) Error Decay Estimator (EDE): minimizing the information loss of gradients by gradually approximating the sign function in backward propagation, jointly considering the updating ability and accurate gradients. We are the first to investigate both forward and backward processes of binary networks from the unified information perspective, which provides new insight into the mechanism of network binarization. Comprehensive experiments with various network structures on CIFAR-10 and ImageNet datasets manifest that the proposed IR-Net can consistently outperform state-of-the-art quantization methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/c304745c22b4cc70578eb1b86714d358565ca102

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions (2020, background): “65 666 667 668 669 670 671 672 673 674 ECCV #2106 ECCV #2106 ECCV-20 submission ID 2106 15 Methods OPs (! 108)Top-1Acc(%) Real-to-Binary Baseline [3]1.63 60.9 Our ReAct Baseline Net 0.87 61.1 XNORNet [29]1.67 51.2 Bi-RealNet [23]1.63 56.4 Real-to-Binary [3]1.65 65.4 MeliusNet29 [2]2.14 65.8 Our ReAc”
Binary Neural Networks: A Survey (2020, methodology): “ierentiable Soft Quantization [63] - A - X BCGD [91] - - - X ProxQuant [92] - A - X Quantization Networks [93] - S - X Self-Binarizing Networks [94] - A - X Improved Training BNN [95] - A - X IR-Net [96] - S X X * Tech. = Technology. Tricks: ST = Structure Transformation, OP = Optimizer, AQ = Asympt”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “As such, there is a large body of work that has proposed different solutions to address this [18, 25, 47, 52, 77, 82, 91, 92, 118, 120, 122, 127, 129, 133, 139, 147, 152, 157, 190, 192, 210, 241, 243, 252, 254, 279, 281].”