Bitwise Neural Networks

By Minje Kim, Paris Smaragdis. 2016

226 citations recorded by the source corpus.

Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.

Subfields: Artificial Intelligence, Efficiency & Compression, Neural Architecture Search & AutoML, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Neural Architecture Search & AutoML

In Signal Processing

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Kim and Smaragdis (2016) retrained neural networks with binary weights and activations.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “[ 41] retraine a previously trained neural network with binary weights and binary inputs.”
Recent advances in convolutional neural networks (2015, background): “weights, and binary output activations. Full binarization requires all the three components are binarized, and the cases with one or two components are considered as partial binarization. Kim et al . [171] consider full binarization with a predetermined portion of the synapses having zero weight, an”
Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding (2019, methodology): “A speech coding system can be formulated by DNN as an autoencoder (AE) with a code layer discretized by vector quantization (VQ) [9] or bitwise network techniques [10], etc.”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “Recent research efforts (Courbariaux et al., 2014; Kim & Smaragdis, 2016; Rastegari et al., 2016; Li & Liu, 2016; Merolla et al., 2016) have considerably reduced both model size and computation complexity by using low bitwidth weights and low bitwidth activations.”
Neural Networks with Few Multiplications (2015, background): “Published as a conference paper at ICLR 2016 Kim & Paris (2015) introduces a completely Boolean network, which simpliﬁes the test time computation at an acceptable performance hit.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “As such, there is a large body of work that has proposed different solutions to address this [18, 25, 47, 52, 77, 82, 91, 92, 118, 120, 122, 127, 129, 133, 139, 147, 152, 157, 190, 192, 210, 241, 243, 252, 254, 279, 281].”
Training Quantized Nets: A Deeper Understanding (2017, background): “The most extreme scenario of quantization is binarization, in which only 1-bit (two states) is used for weight representation [10, 5, 1, 3, 11, 12].”
FINN: A Framework for Fast, Scalable Binarized Neural Network Inference (2016, background): “[12] M. Kim and P. Smaragdis.”
Online Continual Learning for Embedded Devices (2022, background): “…2016; LeCun et al., 1989; Li et al., 2016; Louizos et al., 2018; Srinivas & Babu, 2015), quantization (Courbariaux et al., 2016; Jacob et al., 2018; Kim & Smaragdis, 2016), model compression (Jacob et al., 2018; Krishnamoorthi, 2018; Wu et al., 2016; Soudry et al., 2014; Zhou et al., 2017a; 2016;…”
LCNN: Lookup-Based Convolutional Neural Network (2016, background): “[23] retrains a previously trained neural network with binary weights and binary inputs.”
A comprehensive survey on model compression and acceleration (2020, methodology): “In addition, Bitwise NN proposed by Kim and Smaragdis (2016) uses binary values to represent everything, i.e. inputs, biases, weights and intermediate hidden outputs.”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, background): “There are some other techniques that have been proposed in last few years [271,272,273].”
The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning (2016, unknown): “Another line of research concerns inference and model compression of a pre-trained model [44, 13, 18, 28, 22, 23, 46].”
FINN-R (2018, background): “al Networks - Accuracy On smaller image classification benchmarks such as MNIST, SVHN and CIFAR-10, QNNs have been demonstrated [13, 72] to achieve nearly state-of-the-art accuracy. Kim and Smaragdis [29] consider full binarization (where weights, inputs and outputs are binarized) with a predetermin”