A Study of BFLOAT16 for Deep Learning Training

By Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, K. Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen and 9 more. 2019

492 citations recorded by the source corpus.

This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 is attractive for Deep Learning training for two reasons: the range of values it can represent is the same as that of IEEE 754 floating-point format (FP32) and conversion to/from FP32 is simple. Maintaining the same range as FP32 is important to ensure that no hyper-parameter tuning is required for convergence; e.g., IEEE 754 compliant half-precision floating point (FP16) requires hyper-parameter tuning. In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16. We have implemented a method to emulate BFLOAT16 operations in Tensorflow, Caffe2, IntelCaffe, and Neon for our experiments. Our results show that deep learning training using BFLOAT16 tensors achieves the same state-of-the-art (SOTA) results across domains as FP32 tensors in the same number of iterations and with no changes to hyper-parameters.

Subfields: Artificial Intelligence, Computational Mathematics, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “The NVIDIA A100 GPUs that we ultimately had access to support the bfloat16 format (Wang and Kanwar, 2019; Kalamkar et al., 2019), which has the same dynamic range as float32.”
Multi-lingual Evaluation of Code Generation Models (2022, methodology): “Our training pipeline is based on PyTorch Lightning § and we use bfloat16 (Kalamkar et al., 2019) and DeepSpeed (Rasley et al.”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, background): “However, 8-bit precision training remains an open challenge (Johnson, 2018; Kalamkar et al., 2019).”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, result): “As an example, BFLOAT16 [47] has 16 bits to represent a parameter, and thus, saves memory footprint and area overhead of arithmetic units compared to FP32.”
Large Language Models in Finance: A Survey (2023, background): “In Section 5, we provide a more detailed discussion of which option is more favorable under different circumstances.”
FP8 Formats for Deep Learning (2022, unknown): “Common floating point types for training include IEEE single precision, TF32 mode for single precision [19], IEEE half precision [14], and bfloat16 [9].”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, methodology): “The cluster includes a set of four highly flexible FPUs shared between all nine cores, capable of FP32 and FP16 computation [57].”
A domain-specific supercomputer for training deep neural networks (2020, methodology): “Beyond our experience that it works for training production applications, a recent Intel study corroborated its benefits.(21) However, fp16 requires adjustments to training software (loss scaling) to deliver convergence and efficiency.”
The Deep Learning Compiler: A Comprehensive Survey (2020, unknown): “Besides, low-precision training and inference, including FP32, FP16, INT8, and non-IEEE floating-point format bfloat16 [45] are also supported.”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “• Training using reduced precision arithmetic on 16-bit floating point or even quantized representations [76, 78, 45] significantly reduces memory [27, 50].”