Neural Networks with Few Multiplications

By Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, Yoshua Bengio. 2015

347 citations recorded by the source corpus.

For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Sequence-Level Knowledge Distillation (2016, background): “, 2016), sparsity-inducing regularizers (Murray and Chiang, 2015), binarization of weights (Courbariaux et al., 2016; Lin et al., 2016), and weight sharing (Chen et al.”
FastText.zip: Compressing text classification models (2016, background): “Recently, several research efforts have been conducted to compress the parameters of architectures involved in computer vision, namely for state-of-the-art Convolutional Neural Networks (CNNs) (Han et al., 2016; Lin et al., 2015).”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Lin et al. (2015a) carried over the work of Courbariaux et al. (2015a) to the back-propagation process by quantizing the representations at each layer of the network, to convert some of the remaining multiplications into binary shifts by restricting the neurons’ values to be power-of-two integers.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, background): “[34] proposed a back-propagation process by quantizing the representations at each layer of the network.”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, background): “In addition, an emerging area of research is low precision networks where the parameters and/or activations are quantized to 4 bits or fewer (Courbariaux et al., 2015; Lin et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhu et al., 2016).”
Fixed Point Quantization of Deep Convolutional Networks (2015, background): “This may be attributed to the regularization effect of the added quantization noise (Lin et al., 2015; Luo & Yang, 2014).”
Machine Learning Models that Remember Too Much (2017, background): “required to achieve high performance in machine learning models [29, 48, 64].”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “From the perspective of inference acceleration, sparse and quantized Deep Neural Networks (DNNs) have been widely studied, such as [17][18][19][20][21][22][23][24][25].”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “(Lin et al., 2015) makes a step further towards low bitwidth gradients by converting some multiplications to bit-shift.”
Trained Ternary Quantization (2016, background): “For BNN with binary connections, only quantized binary values are needed for inference.”
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “Notwithstanding the promising results achieved by the existing quantizationbased methods [6, 34, 29, 54, 52, 21, 39, 53, 22, 3, 55, 9, 18], there is still a sizeable accuracy gap between the quantized DNNs and their full-precision counterparts, especially when quantized with extremely low bit-widths”
Compression of Neural Machine Translation Models via Pruning (2016, methodology): “, 2015), or even using binary weights (Lin et al., 2016).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Performing the backward pass with ﬂoating point is important, as accumulating the gradients in quantized precision can result in zero-8 gradient or gradients that have high error, especially in low-precision [42, 79, 80, 106, 156, 181, 198, 223].”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, background): “Weight ternarization is pioneered in ternaryconnect (Lin et al., 2016) where the ternarized values can take {−1, 0, 1} represented by 2 bits.”
Adding New Tasks to a Single Network with Weight Trasformations using Binary Masks (2018, methodology): “Due to the low overhead required by the task-specific parameters introduced by our method, our work is linked to recent works on binarization [6,17,56,40,29] and quantization [18,28] of network parameters.”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, background): “…(Sindhwani et al., 2015; Ding et al., 2018; Wang et al., 2018; Ding et al., 2017), quantization (Vanhoucke et al., 2011; Zhu et al., 2016; Lin et al., 2015; Hubara et al., 2017; He et al., 2016a; Hubara et al., 2016; Courbariaux & Bengio, 2016; Gope et al., 2019) and tensor decomposition…”
Structured Probabilistic Pruning for Convolutional Neural Network Acceleration (2017, methodology): “As the extreme form of quantization, binarized networks were proposed to learn binary value of weights or activation functions in CNN training and testing [4, 20, 24].”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, background): “The work in [18] significantly reduced the time on float point multiplication in the training stage by stochastically binarizing weights and converting multiplications in the hidden state computation to sign changes.”