Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation

By Zhiyong Cheng, Daniel Soudry, Zexi Mao, Zhenzhong Lan. 2015

52 citations recorded by the source corpus.

Compared to Multilayer Neural Networks with real weights, Binary Multilayer Neural Networks (BMNNs) can be implemented more efficiently on dedicated hardware. BMNNs have been demonstrated to be effective on binary classification tasks with Expectation BackPropagation (EBP) algorithm on high dimensional text datasets. In this paper, we investigate the capability of BMNNs using the EBP algorithm on multiclass image classification tasks. The performances of binary neural networks with multiple hidden layers and different numbers of hidden units are examined on MNIST. We also explore the effectiveness of image spatial filters and the dropout technique in BMNNs. Experimental results on MNIST dataset show that EBP can obtain 2.12% test error with binary weights and 1.66% test error with real weights, which is comparable to the results of standard BackPropagation algorithm on fully connected MNNs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Soudry et al. (2014) and Cheng et al. (2015) proved the contrary by showing that good performance could be achieved even if all neurons and weights are binarized to ±1 .”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, background): “[37, 38] do not train their DNN with Backpropagation (BP) but with a variant called Expectation Backpropagation (EBP).”
Assumed Density Filtering Methods for Learning Bayesian Neural Networks (2016, methodology): “Interestingly, on MNIST both EBP and PBP perform quite well and are better than (Cheng et al. 2015) while being comparable to (Blundell et al. 2015).”
Neural Networks with Few Multiplications (2015, background): “Similar to that, Cheng et al. (2015) proves deep neural networks with binary weights can be trained to distinguish between multiple classes with expectation back propagation.”
Training Quantized Nets: A Deeper Understanding (2017, methodology): “Bayesian learning has also been applied to train binary networks [19, 20].”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “EBP infers networkswith quantizedweights by updating the posterior distributions over the weights.”
Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM (2017, methodology): “When coming to extremely low bit networks, i.e. only one or two bits are used to represent weights [12, 13, 14], they only work well on simple datasets (e.g. MNIST and CIFAR10), and usually incur a large loss on challenging datasets like ImageNet.”
A comprehensive survey on model compression and acceleration (2020, methodology): “BinaryConnect was different from Soudry et al. (2014) and Cheng et al. (2015), where authors use EBP to train the network instead of BP and does not works with convolutional networks.”
Binarized Neural Networks (2016, background): “In BinaryConnect, the real-valued version of the weights is saved and used as a key reference for the binarization process.”
Backpropagation for Energy-Efficient Neuromorphic Computing (2015, background): “88%) [15], and iii) we demonstrate the network running in real-time on the TrueNorth chip [7], achieving by far the best published power efficiency for digit recognition (4 μJ per classification at 95% accuracy running 1000 images per second) compared to other low power approaches (6 mJ per classifi”