Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm

By Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei David Liu, Kwang-Ting Cheng. European Conference on Computer Vision, 2018

648 citations recorded by the source corpus.

In this work, we study the 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While being efficient, the classification accuracy of the current 1-bit CNNs is much worse compared to their counterpart real-valued CNN models on the large-scale dataset, like ImageNet. To minimize the performance gap between the 1-bit and real-valued CNN models, we propose a novel model, dubbed Bi-Real net, which connects the real activations (after the 1-bit convolution and/or BatchNorm layer, before the sign function) to activations of the consecutive block, through an identity shortcut. Consequently, compared to the standard 1-bit CNN, the representational capability of the Bi-Real net is significantly enhanced and the additional cost on computation is negligible. Moreover, we develop a specific training algorithm including three technical novelties for 1-bit CNNs. Firstly, we derive a tight approximation to the derivative of the non-differentiable sign function with respect to activation. Secondly, we propose a magnitude-aware gradient with respect to the weight for updating the weight parameters. Thirdly, we pre-train the real-valued CNN model with a clip function, rather than the ReLU function, to better initialize the Bi-Real net. Experiments on ImageNet show that the Bi-Real net with the proposed training algorithm achieves 56.4% and 62.2% top-1 accuracy with 18 layers and 34 layers, respectively. Compared to the state-of-the-arts (e.g., XNOR Net), Bi-Real net achieves up to 10% higher top-1 accuracy with more memory saving and lower computational cost.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4bd23d951846832bdf550df574cdec07bc08dec1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets (2020, background): “Low-bit quantization [18, 61, 36, 30, 8, 12, 19] represents weights or activations in neural networks using low-bit values.”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, methodology): “Meanwhile, the widely-adopted deterministic binarization function in forward pass [4,9, 28, 30,31] can be attained via setting a hard threshold on the probability (i.”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, positive): “Bi-Real net [18] and DSQ [19] show that replacing STE with high-order polynomial or multiple soft-step functions results in better convergence in low-precision representations.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “To mitigate drawbacks of low bit width networks, reduced information quality, and reduced network capacity, MeliusNet used a combination of dense block [22] which increases the network channels by concatenating derived channels from the input to improve capacity and improvement block [151] which imp”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, positive): “Many literatures (Jacob et al. 2018; Wang et al. 2018b; Zhuang et al. 2018; Liu et al. 2018; Jain et al. 2019) have demonstrated that low bit-width quantization can be utilized to speed up the inference stage (weight or activation) with minor hurt of accuracy.”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, methodology): “We use linear learning rate decay following other binary network training protocol (Martinez et al., 2020; Liu et al., 2018).”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Bi-Real Net [139] is proposed to minimize the accuracy degradation of 1-bit CNN incorporating real activations after a 1-bit convolution operation.”
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration (2021, methodology): “Promisingly, it can be observed that our results are even comparable to some recently proposed supervised methods, such as Bi-RealNet-18 [22], CI-BCNN [39] while only containing about 1/2 OPs to them.”
Binary Neural Networks: A Survey (2020, methodology): “Liu et.al. creatively proposed Bi-Real Net, which added additional shortcuts (Bi-Real) to reduce the information loss caused by binarization [62].”
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions (2020, methodology): “Following the insights highlighted in [23], we adopt blocks with identity shortcuts which bypass 1-bit vanilla convolutions to replace the convolutions in MobileNetV1.”
MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization (2019, methodology): “[12] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “While STE is the mainstream approach [218, 280], other approaches have also been explored in the literature [2, 25, 31, 59, 142, 160].”
Any-Precision Deep Neural Networks (2019, background): “(Liu et al. 2018b) also attacked the gradient mismatch problem by introducing a piecewise polynomial function to approximate the sign function.”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “Evaluation is on ResNet-18 and ResNet-34 [7], as common practice in previous quantization works (e.g. [5, 27, 16, 36, 31]).”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, methodology): “Model compression techniques for DNN models, including element-wise pruning [7, 21, 38], structural pruning [33, 25, 20], quantization [23, 31], and factorization [15, 39, 36, 35], have been extensively studied.”
Forward and Backward Information Retention for Accurate Binary Neural Networks (2019, methodology): “To better deal with the discreteness, different approximations of binarization for backward propagation have been studied [8, 38, 11, 32, 7], mainly categorized into either improving the updating ability or reducing the mismatching area between sign function and the approximation one.”
GhostNet: More Features From Cheap Operations (2019, methodology): “Specifically, binarization methods [20, 40, 35] with only 1-bit values can extremely accelerate the model by efficient binary operations.”