ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions

By Zechun Liu, Zhiqiang Shen, Marios Savvides, Kwang-Ting Cheng. European Conference on Computer Vision, 2020

434 citations recorded by the source corpus.

In this paper, we propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts, bypassing all the intermediate convolutional layers including the downsampling layers. This baseline network strikes a good trade-off between accuracy and efficiency, achieving superior performance than most of existing binary networks at approximately half of the computational cost. Through extensive experiments and analysis, we observed that the performance of binary networks is sensitive to activation distribution variations. Based on this important observation, we propose to generalize the traditional Sign and PReLU functions, denoted as RSign and RPReLU for the respective generalized functions, to enable explicit learning of the distribution reshape and shift at near-zero extra cost. Lastly, we adopt a distributional loss to further enforce the binary network to learn similar output distributions as those of a real-valued network. We show that after incorporating all these ideas, the proposed ReActNet outperforms all the state-of-the-arts by a large margin. Specifically, it outperforms Real-to-Binary Net and MeliusNet29 by 4.0% and 3.6% respectively for the top-1 accuracy and also reduces the gap to its real-valued counterpart to within 3.0% top-1 accuracy on ImageNet dataset. Code and models are available at: this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1fab639219c601ddaf73430249035c827fbc1d7f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Fast Knowledge Distillation Framework for Visual Recognition (2021, methodology): “We employ ReActNet [22] and ResNet-50 [14] as the target/student backbones in these experiments.”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Moreover, when it comes to binary quantization, some studies [2, 37, 38, 41] show that Adam [28] performs better than SGD [48] for optimization.”
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration (2021, methodology): “The summary of our transfer results is provided in Table 5, all the network structures in this table are MobileNet-like ReActNet.”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, background): “The student is the state-of-the-art ReActNet (Liu et al., 2020) with ResNet-18 backbone.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, background): “…binarization approaches can be categorized into quantization error minimization (Rastegari et al., 2016; Hou et al., 2017; Zhang et al., 2018), improving training objectives (Martinez et al., 2020; Bai et al., 2020) and reduction of gradient mismatch (Bai et al., 2018; Liu et al., 2018, 2020).”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, methodology): “Meanwhile, the widely-adopted deterministic binarization function in forward pass [4,9, 28, 30,31] can be attained via setting a hard threshold on the probability (i.”