Mixed Precision Training With 8-bit Floating Point

By Naveen Mellempudi, Sudarshan M. Srinivasan, Dipankar Das, Bharat Kaul. 2019

80 citations recorded by the source corpus.

Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, deep learning training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attempts to train DNNs at 8-bit precision have met with significant challenges because of the higher precision and dynamic range requirements of back-propagation. In this paper, we propose a method to train deep neural networks using 8-bit floating point representation for weights, activations, errors, and gradients. In addition to reducing compute precision, we also reduced the precision requirements for the master copy of weights from 32-bit to 16-bit. We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-1K, WMT16) and a broader set of workloads (Resnet-18/34/50, GNMT, Transformer) than previously reported. We propose an enhanced loss scaling method to augment the reduced subnormal range of 8-bit floating point for improved error propagation. We also examine the impact of quantization noise on generalization and propose a stochastic rounding technique to address gradient noise. As a result of applying all these techniques, we report slightly higher validation accuracy compared to full precision baseline.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For example, a common variant of this data type has 5 bits for the exponent and 2 bits for the fraction (Wang et al., 2018; Sun et al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) and uses either no scaling constants or zeropoint scaling.”
8-bit Optimizers via Block-wise Quantization (2021, background): “The most common combination is 5 bits for the exponent and 2 bits for the fraction (Wang et al., 2018b; Sun et al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) with either no normalization or min-max normalization.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, critical): “While several interesting works exist in this area [10, 26, 121, 135, 169], the proposed methods often require a lot of hyperparameter tuning, or they only work for a few NN models on relatively easy learning tasks.”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, methodology): “…also of activations and weights • S2FP8 does not require keeping the ﬁrst and last layer in FP32 precision, which is needed for other approaches (Mellempudi et al., 2019), however maintains the master weights and accumulations inside the matrix multipliers in FP32 We demonstrate across image…”
A domain-specific supercomputer for training deep neural networks (2020, methodology): “˲ Wider data: Quantized arithmetic—8-bit integer instead of 32-bit floating point (FP)—can work for inference like in TPUv1 but reduced-precision training is an active research area.(21,25) The challenge is sufficiently capturing the SGD sum of many small weight updates to preserve the accuracy of u”
FP8 Formats for Deep Learning (2022, unknown): “Obtaining results for large models is imperative as previous studies have identified different numerical behavior for models of different sizes (for example, R18 and R50 in [12]).”