Training Deep Neural Networks with 8-bit Floating Point Numbers

By Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, Kailash Gopalakrishnan. 2018

566 citations recorded by the source corpus.

The state-of-the-art hardware platforms for training deep neural networks are moving from traditional single precision (32-bit) computations towards 16 bits of precision - in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of deep neural networks using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of deep learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4 times improved throughput over today's systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Stable and low-precision training for large-scale vision-language models (2023, background): “Moreover, while int8 and int4 are used for inference [17, 66, 16], and there is earlier work exploring 8 bit training for convnets [61, 79, 10], these formats are not commonly used for training transformer models at scale.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For example, a common variant of this data type has 5 bits for the exponent and 2 bits for the fraction (Wang et al., 2018; Sun et al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) and uses either no scaling constants or zeropoint scaling.”
8-bit Optimizers via Block-wise Quantization (2021, background): “…Brown et al., 2020) and RoBERTa pretraining (Liu et al., 2019) on English CC-100 + RoBERTa corpus (Nagel, 2016; Gokaslan and Cohen, 2019; Zhu et al., 2015; Wenzek et al., 2020), finetuning the pretrained masked language model RoBERTa (Liu et al., 2019)3 on GLUE (Wang et al., 2018a), ResNet-50 v1.”
Secure Evaluation of Quantized Neural Networks (2019, background): “It is important to notice that the spectrum of research in quantized training of CNNs is much more reduced than in quantized inference, with only a few recent works like [35], [65] and [66].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “While this survey on quantization is mostly focused on inference, we should emphasize that an important success of quantization has been in NN training [10, 35, 57, 128, 239].”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, unknown): “Various 8-bit numerical formats have been proposed, including INT8 [20, 21, 22, 23], FP8 [24, 25], 𝐅 𝑙 (𝐇 𝑙 ,𝚯 𝑙 ) 𝐉 (𝑙) 𝐊 (𝑙) ⊗ ⊗ 𝐇 (𝑙−1) 𝚯 (𝑙) 𝐇 (𝑙) 𝑄 𝑏 (෡∇ 𝐇 (𝑙) ) ෡∇ 𝚯 (𝑙)”
TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning (2020, background): “Similarly, [32] proposes to reduce the bitwidth of training activations by introducing new reduced-precision floating-point formats.”
Tiny Transfer Learning: Towards Memory-Efficient On-Device Learning (2020, background): “Similarly, [46] proposes to reduce the bitwidth of training activations by introducing new reduced-precision ﬂoating-point formats.”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, methodology): “While 8-bit ﬂoating point formats appear to be sufﬁcient for training [14, 19], 4-bit gradient representations appear challenging from a quantization error (rounding), precision and dynamic range perspective.”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, unknown): “Many literatures (Jacob et al. 2018; Wang et al. 2018b; Hubara et al. 2016; Choi et al. 2019; Jain et al. 2019) have demonstrated that low bit-width quantization can be utilized to speed up the inference stage (weight or activation) with minor hurt of accuracy.”
Compact and Computationally Efficient Representation of Deep Neural Networks (2018, background): “Alternatively, one could trivially reduce the numerical precision [31], [32], [41], [42] needed for representing each weight element value down to 7 bits in this case.”
Towards Unified INT8 Training for Convolutional Neural Network (2019, background): “RangeBN [2] and FP8 training [54] achieve accuracy comparable to full-precision models, but they both use ﬂoating-point number in gradients, which is not beneﬁcial for hardware optimization to boost the speed.”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “(Banner et al., 2018; Wang et al., 2018a) , where θ t denotes the weights at the t -th iteration of training.”
FP8 Formats for Deep Learning (2022, unknown): “[22] proposed using 5-bit exponent format for training neural networks, confirming their methodology on the convolutional neural networks (CNNs) for image classification on CIFAR-10 and ILSVRC12 ar X iv :2 20 9.”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “• Training using reduced precision arithmetic on 16-bit floating point or even quantized representations [76, 78, 45] significantly reduces memory [27, 50].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Recently, Wang et al. [184] further pushed the multiplications, accumulations, and updated in training to 8, 16, and 16-bit ﬂoating-point numbers, respectively. b) RNN quantization: Table 9 lists most existing quantization works for RNNs.”
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (2019, methodology): “Therefore, loss scaling has been adopted to enable gradients to become large enough to be representable using the (1-5-2) format [2].”