Training Quantized Nets: A Deeper Understanding

By Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, Tom Goldstein. 2017

230 citations recorded by the source corpus.

Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized networks, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of these algorithms for non-convex problems, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the difficulty of training using low-precision arithmetic.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “While previous work [2, 4, 9, 24, 30, 36, 46, 56] had already introduced the idea of functions Qw ,Qa to quantize weights and/or activations in the forward pass, they required the weights W or gradients G, e to be represented in floating-point in the backward pass, in order to optimize accurately.”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, background): “Theoretically justified gradient compression using unbiased stochastic quantizers has been proposed and analyzed in [6], [8], [13], [15].”
Dynamic Model Pruning with Feedback (2020, methodology): “One variant of this approach is implemented in the Binary Connect algorithm (BC) (Courbariaux et al., 2015) with prominent results, see also (Li et al., 2017) for further insights and discussion.”
Model compression via distillation and quantization (2018, neutral): “Li et al. (2017) also examines these dynamics in detail.”
Training and Inference with Integers in Deep Neural Networks (2018, background): “Besides, when it comes to ImageNet dataset, the gradient accumulation, i.e., the bit width of gradients ( k G ) and batch normalization become more important (Li et al., 2017) since samples in training set are so variant.”
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (2019, background): “We study this round-off residual scheme on a wide range of DNN applications and show that it does not impact convergence (consistent with the rich body of theoretical work in this space [31, 32]).”
Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets (2019, methodology): “In particular, weight quantization of DNN have been extensively studied in the literature; see for examples (Li et al., 2016; Zhu et al., 2016; Li et al., 2017; Yin et al., 2016; 2018a; Hou & Kwok, 2018; He et al., 2018; Li & Hao, 2018).”
Training Student Networks for Acceleration with Conditional Adversarial Networks (2018, methodology): “Low precision methods use limited number of bits to store and operate the network weights [20, 26], which often achieve conceptual acceleration because mainstream GPUs have limited support for low precision computation.”
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks (2017, methodology): “Low precision methods use limited number of bits to store and operate the network weights, and the extreme case is binary networks that only use 1-bit to represent each number [28, 21].”
Reborn Filters: Pruning Convolutional Neural Networks with Limited Data (2020, background): “For example, quantization (Courbariaux, Bengio, and David 2015; Rastegari et al. 2016; Li et al. 2017) attempts to reduce the redundancy on numerical precision, and sparse connection (Han, Mao, and Dally 2015; Wen et al. 2016; Alvarez and Salzmann 2016) or tensor factorization (Lebedev et al. 2014;…”
Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (2023, background): “These optimizers consistently result in higher accuracy (Rastegari et al., 2016; Courbariaux et al., 2016) and avoid local minima (Li et al., 2017).”
Binary Neural Networks: A Survey (2020, background): “[98] further investigated training methods for quantized neural networks from a theoretical viewpoint, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the diﬃculty of…”
Towards Unified INT8 Training for Convolutional Neural Network (2019, result): “As the prior studies [1, 34, 22, 61] do, we first make the following assumptions:”
SWALP : Stochastic Weight Averaging in Low-Precision Training (2019, methodology): “To understand whether SWALP can achieve a better bound than SGD-LP, we need to answer the following question: can SGD-LP algorithm potentially achieve a better bound than the O(δ) bound proved in Li et al. (2017)?”
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks (2019, background): “Experiments and analysis in [26, 8, 24] have shown that the gradient error caused by quantization and STE greatly harms the accuracy of quantized models.”
Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks (2018, methodology): “According to the results shown in Table 6, we can find that our ELQ obtains much better results than the latest BC-ADAM method [24].”
A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (2018, methodology): “[26] studied different training methods for quantized neural network.”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, methodology): “Hubara et al. (2016) achieve good results on classiﬁcation tasks by modifying the binarization scheme of activations a from spikes ( a ∈ { 0, 1 } ) to a ∈ {− 1, 1 } , thereby improving the convergence properties (Li et al., 2017).”