A Statistical Framework for Low-bitwidth Training of Deep Neural Networks

By Jianfei Chen, Yujie Gai, Zi-Mei Yao, Michael W. Mahoney, Joseph E. Gonzalez. 2020

73 citations recorded by the source corpus.

Fully quantized training (FQT), which uses low-bitwidth hardware by quantizing the activations, weights, and gradients of a neural network model, is a promising approach to accelerate the training of deep neural networks. One major challenge with FQT is the lack of theoretical understanding, in particular of how gradient quantization impacts convergence properties. In this paper, we address this problem by presenting a statistical framework for analyzing FQT algorithms. We view the quantized gradient of FQT as a stochastic estimator of its full precision counterpart, a procedure known as quantization-aware training (QAT). We show that the FQT gradient is an unbiased estimator of the QAT gradient, and we discuss the impact of gradient quantization on its variance. Inspired by these theoretical results, we develop two novel gradient quantizers, and we show that these have smaller variance than the existing per-tensor quantizer. For training ResNet-50 on ImageNet, our 5-bit block Householder quantizer achieves only 0.5% validation accuracy loss relative to QAT, comparable to the existing INT8 baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Zhu
2. Chongxuan Li
3. Jian Ren
4. Sergey Tulyakov
5. Qing Jin
6. Kai-Min Yang
7. Zhiyu Chen
8. Zhengang Li

In Computer Vision and Pattern Recognition

1. Jun Zhu
2. Chongxuan Li
3. Jian Ren
4. Jianfei Chen
5. Sergey Tulyakov
6. Qing Jin
7. Kai-Min Yang
8. Zhiyu Chen

Citations in the corpus

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “With this goal in mind, 8-bit quantization methods for transformers have been developed (Chen et al., 2020; Lin et al., 2020; Zafrir et al., 2019; Shen et al., 2020).”
• Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…community, various likelihood-based generative models have been developed, including normalizing flows (Dinh et al. (2017); Ho et al. (2019a); Chen et al. (2020b); Lu et al. (2021), NFs), variational auto-encoders (Kingma & Welling (2013); Rezende et al. (2014), VAEs), auto-regressive…”
• A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “For instance, in [29, 78], the Int operator in Eq.”
• F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, background): “…quantization where the model shares the same precision (Zhou et al., 2018; Wang et al., 2018; Choukroun et al., 2019; Gong et al., 2019; Langroudi et al., 2019; Jin et al., 2020a; Bhalgat et al., 2020; Chen et al., 2020; Yang et al., 2020; Darvish Rouhani et al., 2020; Oh et al., 2021).”