FP8 Formats for Deep Learning

By Paulius Micikevicius, Dusan Stosic, N. Burgess, Marius Cornea, Pradeep K. Dubey, R. Grisenthwaite, Sangwon Ha, A. Heinecke, Patrick Judd, John Kamalu and 5 more. 2022

285 citations recorded by the source corpus.

FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for representatio of special values, E4M3’s dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions. Our study covers the main modern neural network architectures - CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters unchanged from the 16-bit baseline training sessions. Our training experiments include large, up to 175B parameter, language models. We also examine FP8 post-training-quantization of language models trained using 16-bit formats that resisted fixed point int8 quantization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Ali Farhadi
3. Ludwig Schmidt
4. Tim Dettmers
5. Mitchell Wortsman
6. Percy Liang
7. Mohit Bansal
8. Hannaneh Hajishirzi

In Computer Vision and Pattern Recognition

1. Ali Farhadi
2. Luke S. Zettlemoyer
3. Luc Van Gool
4. Mitchell Wortsman
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Ludwig Schmidt
8. Dacheng Tao

In Electrical and Electronic Engineering

1. Tim Dettmers
2. Luke S. Zettlemoyer
3. Ali Farhadi
4. Ludwig Schmidt
5. Tri Dao
6. Dan Alistarh
7. Elias Frantar

In Efficiency & Compression

1. Luke S. Zettlemoyer
2. Tim Dettmers
3. Mitchell Wortsman
4. Ali Farhadi
5. Ludwig Schmidt
6. Ari S. Morcos
7. Tri Dao
8. Dan Alistarh

In Optimization & Training

1. Luke S. Zettlemoyer
2. Tim Dettmers
3. Ali Farhadi
4. Ludwig Schmidt
5. Tri Dao
6. Ari S. Morcos
7. Mitchell Wortsman
8. Torsten Hoefler

Citations in the corpus

• Stable and low-precision training for large-scale vision-language models (2023, methodology): “Besides int8 training, we also study large-scale 8-bit float (fp8) [39] training.”
• The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “Since we use 3-8 bit precision in our work, we take the FP8 data type as a reference (Micikevicius et al., 2022).”
• AI and Memory Wall (2024, unknown): “A recent promising trend is to use a mixture of FP8 and FP16 (and even most recently FP4) [27].”
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, background): “One typically use per-tensor scaling [37] by keeping one scalar per tensor (e.g., one for Q , for K , and for V ).”