Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks

By Léopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oguz H. Elibol, Hanlin Tang. 2020

54 citations recorded by the source corpus.

Training with larger number of parameters while keeping fast iterations is an increasingly adopted strategy and trend for developing better performing Deep Neural Network (DNN) models. This necessitates increased memory footprint and computational requirements for training. Here we introduce a novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Reduced bit precision allows for a larger effective memory and increased computational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We show that, unlike previous 8-bit precision training methods, the proposed method works out-of-the-box for representative models: ResNet-50, Transformer and NCF. The method can maintain model accuracy without requiring fine-tuning loss scaling parameters or keeping certain layers in single precision. We introduce two learnable statistics of the DNN tensors - shifted and squeezed factors that are used to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in information due to quantization.

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For example, a common variant of this data type has 5 bits for the exponent and 2 bits for the fraction (Wang et al., 2018; Sun et al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) and uses either no scaling constants or zeropoint scaling.”
8-bit Optimizers via Block-wise Quantization (2021, background): “The most common combination is 5 bits for the exponent and 2 bits for the fraction (Wang et al., 2018b; Sun et al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) with either no normalization or min-max normalization.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, critical): “While several interesting works exist in this area [10, 26, 121, 135, 169], the proposed methods often require a lot of hyperparameter tuning, or they only work for a few NN models on relatively easy learning tasks.”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, unknown): “𝑄 𝑏 (෡∇ 𝐇 (𝑙−1) ) 𝐅 𝑙 (𝐇 𝑙 ,𝚯 𝑙 ) 𝐉 (𝑙) 𝐊 (𝑙) ⊗ ⊗ 𝐇 (𝑙−1) 𝚯 (𝑙) 𝐇 (𝑙) ∇ 𝐇 (𝑙) ∇ 𝚯 (𝑙) 𝐅 𝑙 (𝐇 𝑙 ,𝚯 𝑙 ) 𝐉 (𝑙) 𝐊 (𝑙) ⊗ ⊗ 𝐇 (𝑙−1) 𝚯 (𝑙) 𝐇 (𝑙) ∇ 𝐇 (𝑙) ∇ 𝚯 (𝑙) ℓ block ﬂoating point [26], FP8 with learnable parameters [27], and adaptive precision [28].”