How to Train a Compact Binary Neural Network with High Accuracy?

By Wei-Yao Tang, Gang Hua, Liang Wang. 2017

259 citations recorded by the source corpus.

How to train a binary neural network (BinaryNet) with both high compression rate and high accuracy on large scale dataset? We answer this question through a careful analysis of previous work on BinaryNets, in terms of training strategies, regularization, and activation approximation. Our findings first reveal that a low learning rate is highly preferred to avoid frequent sign changes of the weights, which often makes the learning of BinaryNets unstable. Secondly, we propose to use PReLU instead of ReLU in a BinaryNet to conveniently absorb the scale factor for weights to the activation function, which enjoys high computation efficiency for binarized layers while maintains high approximation accuracy. Thirdly, we reveal that instead of imposing L2 regularization, driving all weights to zero which contradicts with the setting of BinaryNets, we introduce a regularization term that encourages the weights to be bipolar. Fourthly, we discover that the failure of binarizing the last layer, which is essential for high compression rate, is due to the improper output range. We propose to use a scale layer to bring it to normal. Last but not least, we propose multiple binarizations to improve the approximation of the activations. The composition of all these enables us to train BinaryNets with both high compression rate and high accuracy, which is strongly supported by our extensive empirical study.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, background): “Other efforts speciﬁ-cally train very sparse networks via L1 penalties (Zhang, Lee, and Jordan 2016) or even binary neural networks (Tang, Hua, and Wang 2017; Rastegari et al. 2016) with the goal of faster computation.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “To further improve the quantizer, more recent work [77, 226, 250] formulates non-uniform quantization as an optimization problem.”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, background): “, 2016), modifications of regularizers (Tang et al., 2017), gradual transitions from soft to hard binarizations (Wu et al.”
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (2017, methodology): “Our experiments show that distillation is an overkill for AlexNet and one can get comparable accuracies using techniques proposed in (Tang et al., 2017; Mishra et al., 2017).”
Direct Training for Spiking Neural Networks: Faster, Larger, Better (2018, methodology): “We found similar coding scheme is also leveraged by previous work on ANN compression (Tang, Hua, and Wang 2017; Hubara, Soudry, and Ran 2016).”
Double Forward Propagation for Memorized Batch Normalization (2018, methodology): “…such as recurrent neural networks(RNNs) (Laurent et al. 2016; Amodei et al. 2016; Cooijmans et al. 2016) and successfully been applied on many challenging tasks (Ba, Kiros, and Hinton 2016; Du-moulin, Shlens, and Kudlur 2017; Li et al. 2016; Tang, Hua, and Wang 2017; Zhang, Zheng, and Qi 2017).”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Therefore, a small learning rate (e.g., 10 − 3 ) is widely adopted in quantized training, especially for extreme quantization such as binarization and ternarization [147], [158], [162].”
SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks (2018, methodology): “Multiple binarizations and a scaling layer were described in [28] to improve accuracy and binarize the last layer.”
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions (2020, background): “In addition to architectural design, studies on 1-bit CNNs expand from training algorithms [34,1,44,3] and binary optimizer design [13], to better approximation of binary weights and activations [28,11,35].”
Cellular, Wide-Area, and Non-Terrestrial IoT: A Survey on 5G Advances and the Road Toward 6G (2021, background): “The accuracy of binary nets is not impressive and needs further improvement [358].”
Heterogeneous Bitwidth Binarization in Convolutional Neural Networks (2018, methodology): “…varied, but the typical approaches include linearly or logarithmically placing the quantization points between 0 and 1, clamping values below a threshold distance from zero to zero (Li et al., 2016), and computing higher bits by measuring the residual error from lower bits (Tang et al., 2017).”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, background): “…prior works also identified that the difficulty of training binarized networks mostly resides in training with insufficient number of iterations or having too large learning rates, which can lead to frequent sign changes of the weights that make the learning of binarized networks unstable [50].”
Towards Accurate Post-training Network Quantization via Bit-Split and Stitching (2020, methodology): “…(Gupta et al., 2015), data representation (Miyashita et al., 2016), multi-bit quantization (Lin et al., 2017b; Zhu et al., 2019; Li et al., 2017; Tang et al., 2017), ﬁxed-point factorization (Wang & Cheng, 2017; Hu et al., 2018a), learnable quantization (Faraone et al., 2018; Wang et al.,…”
Any-Precision Deep Neural Networks (2019, methodology): “Early works [27] show that by adding a layer-wise real-value scaling factor s could largely help reduce the output range variation and hence achieve better model accuracy.”
Efficient Interaction-based Neural Ranking with Locality Sensitive Hashing (2019, background): “The acceleration of neural network computation has been studied for image classification with complex neural network structure (e.g. [12, 26, 28] for binarization).”