Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

By Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio. 2016

2,112 citations recorded by the source corpus.

We introduce a method to train Quantized Neural Networks (QNNs) -- neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At traintime the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves 51% top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Mixed Precision Training (2017, background): “Hubara et al. (2016b) propose another approach to quantize RNNs without altering their structure.”
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, background): “Training techniques to overcome these effects are beyond the scope of this work ( e.g ., quantized gradients [18], Block-Momentum SGD [6]).”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, methodology): “To do this, we perform weight quantization [26]: given a trained network with weights θ, we force each weight to be one of only 256 different values, so each parameter can be represented in 8 bits.”
Fast Inference from Transformers via Speculative Decoding (2022, background): “Numerous techniques try to make inference more efﬁcient for all tokens, e.g. distillation (Hinton et al., 2015), sparciﬁcation (Jaszczur et al., 2021), quantization (Hubara et al., 2016), and architecture modiﬁcation (So et al., 2021; Shazeer, 2019).”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “Here we use uniform quantization function, where the range of floating point values in a tensor is equally split [12, 42] and then represented by unsigned integers in { 0, .”
Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “Figure 16: Understanding the learning behaviors of FedDF on CIFAR-10 with ResNet-8 for α=1.”
Secure Evaluation of Quantized Neural Networks (2019, methodology): “r ﬂoating-point counterparts. It is important to notice that the spectrum of research in quantized training of CNNs is much more reduced than in quantized inference, with only a few recent works like [35], [65] and [66]. Moreover, it is not clear yet how compatible these methods are with MPC or FHE ”
Natural Compression for Distributed Deep Learning (2019, methodology): “Despite the optimization effort, we identify non-negligible 10 ∼ 15% overheads in doing random number generation used in stochastic rounding, which was also reported in [22].”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, background): “In addition, an emerging area of research is low precision networks where the parameters and/or activations are quantized to 4 bits or fewer (Courbariaux et al., 2015; Lin et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhu et al., 2016).”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “As such, the deep learning community has seen a focus on model compression techniques, including knowledge distillation (Ba and Caruana, 2014; Hinton et al., 2015), network pruning (Li et al., 2016; Han et al., 2016) and quantization (Gupta et al., 2015; Hubara et al., 2016).”
Model compression via distillation and quantization (2018, methodology): “The first direction is the work on training quantized neural networks, e.g. Courbariaux et al. (2015); Rastegari et al. (2016); Hubara et al. (2016); Wu et al. (2016a); Mellempudi et al. (2017); Ott et al. (2016); Zhu et al. (2016), which showed that neural networks can converge to good task…”
Network Pruning via Transformable Architecture Search (2019, methodology): “A variety of techniques have been proposed, such as low-rank decomposition [44], weight pruning [11, 26, 10, 9], channel pruning [15, 30], dynamic computation [7, 5] and quantization [20, 1].”
Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations (2017, background): “It is worth noting that many recent works target quantization of the DNN model parameters and possibly the feature representation to speed up DNN evaluation on hardware with low-precision arithmetic, see, e.g. , [15, 23, 38, 43].”
A 64-mW DNN-Based Visual Navigation Engine for Autonomous Nano-Drones (2018, methodology): “Then, we retrained the network from scratch replacing the native convolutional layers from Keras to make them “quantization-aware”, using the methodology proposed by Hubara et al. [37].”
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017, background): “2: ResNet on ImageNet: Accuracy under various quantization schemes, including binary weight networks (BWN [21, 15]), ternary weight networks (TWN [21, 22]), incremental network quantization (INQ [33]) and fine-grained quantization (FGQ [26])”
In-place Activated BatchNorm for Memory-Optimized Training of DNNs (2017, background): “Such works include (partially) binarized weights/activations/gradients [6, 13, 14], which however typically lead to degraded overall performance.”
Trained Ternary Quantization (2016, background): “Hubara et al. (2016) and Li & Liu (2016) propose ternary weight networks to trade off between model size and accuracy.”