Towards Accurate Binary Convolutional Neural Network

By Xiaofan Lin, Cong Zhao, Wei Pan. 2017

685 citations recorded by the source corpus.

We introduce a novel scheme to train binary convolutional neural networks (CNNs) -- CNNs with weights and activations constrained to {-1,+1} at run-time. It has been known that using binary weights and activations drastically reduce memory size and accesses, and can replace arithmetic operations with more efficient bitwise operations, leading to much faster test-time inference and lower power consumption. However, previous works on binarizing CNNs usually result in severe prediction accuracy degradation. In this paper, we address this issue with two major innovations: (1) approximating full-precision weights with the linear combination of multiple binary weight bases; (2) employing multiple binary activations to alleviate information loss. The implementation of the resulting binary CNN, denoted as ABC-Net, is shown to achieve much closer performance to its full-precision counterpart, and even reach the comparable prediction accuracy on ImageNet and forest trail datasets, given adequate binary weight bases and activations.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Expanding the Reach of Federated Learning by Reducing Client Resource Requirements (2018, methodology): “Popular alternatives include pruning the least useful connections in a network [Han et al., 2016, 2015], weight quantization [Hubara et al., 2016, Lin et al., 2017, De Sa et al., 2018], and model distillation [Hinton et al., 2015].”
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “In this section, we compare the performance of our quantization method with existing methods including TWN [29], TTQ [54], BNN [21], BWN [39], XNORNet [39], DoReFa-Net [53], HWQG [3] and ABC-Net [33], with various network architectures tested on CIFAR-10 and ImageNet classification tasks.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, positive): “…each branch, and we refer the interested readers to [191] for more details. a) Quantization Error Minimization: The ﬁrst branch of solutions aims to minimize the quantization error, i.e., the gap between the real values and the quantized values [19, 34, 62, 102, 149, 155, 160, 165, 174, 211, 240].”
Training Compact Neural Networks with Binary Weights and Low Precision Activations (2018, methodology): “Note that the Layerwise approach have similarities between tensor approximation methods in [7, 22, 23] and the diﬀerences are described in Sec.”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, background): “Therefore, the community has raised increasing attention on model compression approaches such as model pruning [7, 21, 34], model quantization [14, 19, 23] and knowledge distillation [13, 30, 33, 37, 38].”
3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning (2018, background): “Note that the quantization and sparsification we discuss in this paper differ from model compression (Han et al., 2016; Jouppi et al., 2017; Lin et al., 2017; Venkatesh et al., 2017).”
Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss (2018, background): “Initialization Good initialization of parameters improves the accuracy of trained low bit-width networks [17, 19].”
Post training 4-bit quantization of convolutional networks for rapid-deployment (2018, background): “Learning quantization for numerical precision of 4-bits and below has long been shown to be effective (Lin et al., 2017; McKinstry et al., 2018; Zhou et al., 2016; Choi et al., 2018).”
And the Bit Goes Down: Revisiting the Quantization of Neural Networks (2019, methodology): “…benchmark for ResNet-18 and ResNet-50 architectures and compare our results to the following methods: Trained Ternary Quantization (TTQ) [23], LR-Net [22], ABC-Net [26], Binary Weight Network (XNOR-Net or BWN) [25], Deep Compression (DC) [15] and Hardware-Aware Automated Quantization (HAQ) [41].”
Binary Neural Networks: A Survey (2020, methodology): “Taking ResNet-18 in ABC-Net [71] on ImageNet dataset as an example, there is only about 7% accuracy loss after applying binarization to weights but there is addition 20% loss after the activations are binarized.”
Quantization Networks (2019, background): “1 ABC-Net [23] - 68.”
Deep Learning for Generic Object Detection: A Survey (2018, neutral): “ネットワークの冗長性を低減または除去するため,コンパクトで軽量なネットワークの設計 [29, 4, 119, 112, 169, 300] やネットワークの高速化 [44, 122, 253, 155, 158, 282]を行う研究への関心が高まっている.”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Existing quantization methods can be roughly divided into two categories according to the quantization bitwidth, namely, fixed-point quantization [4,7,9,19,22,25,28,32,46,72,75,77] and binary quantization [1, 26, 41, 45, 47, 59, 60].”
Forward and Backward Information Retention for Accurate Binary Neural Networks (2019, methodology): “Table 4 shows a number of SOTA quantization methods over ResNet-18 and ResNet-34, including BWN [44], HWGQ [35], TWN [33], LQ-Net [59], DoReFa-Net [61], ABC-Net [36], Bi-Real [38], XNOR++ [6], BWHN [26], SQ-BWN and SQ-TWN [1].”
An Empirical study of Binary Neural Networks' Optimisation (2018, methodology): “…in their implementations: Bina-ryConnect uses ADAM (Kingma & Ba, 2015) for CIFAR-10 and vanilla SGD for MNIST (although in their released source code they used ADAM for both datasets), DoReFa-Net and XNOR-Net use ADAM in their experiments and ABC-Net (Lin et al., 2017) uses SGD with momentum.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, neutral): “, 2021), ABC-Net (Lin et al., 2017), BWN (Rastegari et al.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Besides the basic quantization form in (23), the quantizer itself can also be learned with adaptive dynamic range and quantization levels for different layers [142]–[146].”
Low-bit Quantization of Neural Networks for Efficient Inference (2019, result): “In contrary to other training methods where kernels are blindly added over all the network layers [26], our approach allows minimal computational overhead.”