Training Compact Neural Networks with Binary Weights and Low Precision Activations

By Bohan Zhuang, Chunhua Shen, Ian David Reid. 2018

14 citations recorded by the source corpus.

In this paper, we propose to train a network with binary weights and low-bitwidth activations, designed especially for mobile devices with limited power consumption. Most previous works on quantizing CNNs uncritically assume the same architecture, though with reduced precision. However, we take the view that for best performance it is possible (and even likely) that a different architecture may be better suited to dealing with low precision weights and activations. Specifically, we propose a "network expansion" strategy in which we aggregate a set of homogeneous low-precision branches to implicitly reconstruct the full-precision intermediate feature maps. Moreover, we also propose a group-wise feature approximation strategy which is very flexible and highly accurate. Experiments on ImageNet classification tasks demonstrate the superior performance of the proposed model, named Group-Net, over various popular architectures. In particular, with binary weights and activations, we outperform the previous best binary neural network in terms of accuracy as well as saving more than 5 times computational complexity on ImageNet with ResNet-18 and ResNet-50.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kurt Keutzer
2. Bichen Wu
3. Péter Vajda
4. Yuandong Tian
5. Peizhao Zhang
6. Song Han
7. Yuan Xie
8. Guoqi Li

In Computer Vision and Pattern Recognition

1. Kurt Keutzer
2. Song Han
3. Bichen Wu
4. Péter Vajda
5. Luc Van Gool
6. Yuandong Tian
7. Peizhao Zhang
8. Quoc V. Le

Citations in the corpus

• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, background): “GroupNet uses binary weights and 2-bit activations, but its blocks contain 5 parallel branches.”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Although more bits [144], [147], [169], [182] and wider layers [170], [174] can help to improve accuracy, they will increase parameters and operations as well.”