DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

By Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, Yuheng Zou. 2016

2,283 citations recorded by the source corpus.

We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1\% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Mixed Precision Training (2017, background): “Zhou et al. (2016) quantize weights activations and gradients to different bit counts to further improve result accuracy.”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “Those results are presently state-of-the-art, surpassing those obtained by the DoReFa net (Zhou et al., 2016).”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “Here we solely focus on quantization [4, 6, 8, 14, 17, 24, 40, 42].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, background): “Quantization [31, 27, 39, 45, 44] and factorization [22, 16, 18, 37] are proposed in literature to reduce redundancy in calculations to speed up inference.”
Secure Evaluation of Quantized Neural Networks (2019, methodology): “…1990s [23, 4, 61, 48], and this extensive research body have enabled modern quantized neural networks to have essentially the same accuracy as their full-precision counterparts [12, 68, 26, 30, 52], even with large CNN architectures like AlexNet [43], VGGNet [59], GoogleNet [60] and ResNet [31].”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Another promising line of research addressing the communication bottleneck of large scale training is to use quantization [31, 32, 33] or more aggressive sparsification [34, 35, 36] of gradients.”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, neutral): “DoReFa-Net (Zhou et al., 2016) uses 1-bit weights with 2-bit gradients.”
Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, background): “For example, dropout [25], Batch Normalization (BN) [26] and quantization [27, 28] all serve the purpose of model regularization.”
Recent advances in convolutional neural networks (2015, methodology): “ BNNs on the ImageNet dataset with topologies inspired by AlexNet, ResNet and GoogLeNet, reporting top-1 accuracies of up to 51.2% for full binarization and 65.5% for partial binarization. DoReFa-Net [149] explores reducing precision during the forward pass as well as the backward pass. Both partial”
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, methodology): “We follow DoReFa-Net (Zhou et al. (2016)) to quantize weights and PACT (Choi et al. (2018)) to quantize activations.”
Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…and activations in low precision format instead of 32-bit ﬂoating-point numbers (Courbariaux et al., 2015; Hubara et al., 2016; Rastegari et al., 2016; Zhou et al., 2016; Jacob et al., 2018; Choi et al., 2018; Dong et al., 2019; Esser et al., 2020; Chen et al., 2020a; Van Baalen et al., 2020).”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, methodology): “For the same purpose, quantizing gradients to low-precision values with smaller bit width has also been extensively studied [22][28][29][30].”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, background): “Several papers considered the problem of reducing the precision of gradient by using fewer bits to represent floating pointing numbers [29, 44, 37, 2, 40, 8] or only transmitting coordinates [1, 24].”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, methodology): “Following [36, 5, 27], we only search and quantize the res-blocks, excluding the first convolutional layer and the last fully-connected layer.”
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017, methodology): “This methodology, embraced by approaches such as Ternary weight networks (TWN [22]), Binary Neural Networks (BNN [14]), XNOR-net [27], and more [8, 21, 26, 33, 34, 35], is the focus of our investigation.”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, neutral): “Partially initiated by a 1-bit implementation of SGD by Microsoft in [5], a large number of recent studies revisited the idea of low-precision training as a means to reduce communication [13, 14, 15, 16, 17, 18, 19, 17, 20, 21].”
Searching for MobileNetV3 (2019, background): “Quantization [23, 25, 47, 41, 51, 52, 37] is another important complementary effort to improve the network efficiency through reduced precision arithmetic.”
Trained Ternary Quantization (2016, methodology): “N ET Zhou et al. (2016) proposed DoReFa-Net which quantizes weights, activations and gradients of neural networks using different widths of bits.”
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “As for quantizing both weights and activations, our results are significantly better than DoReFa-Net and HWGQ when using very low bit-widths (1 bit for weights and few for activations).”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, methodology): “For comparison, we test the quantization performance without using the Hessian information, which we refer to as “Direct” method, as well as other methods in the literature including Dorefa [43], PACT [2], LQ-Net [40], and DNAS [36], as shown in Table I.”