Mixed Precision Training

By Paulius Micikevicius, Sharan Narang, Jonah Alben, G. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh and 1 more. 2017

2,457 citations recorded by the source corpus.

Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulates the gradients after each optimizer step. This single-precision copy is rounded to half-precision format during training. Secondly, we propose scaling the loss appropriately to handle the loss of information with half-precision gradients. We demonstrate that this approach works for a wide variety of models including convolution neural networks, recurrent neural networks and generative adversarial networks. This technique works for large scale models with more than 100 million parameters trained on large datasets. Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x. In future processors, we can also expect a significant computation speedup using half-precision hardware units.

Subfields: Computational Theory and Mathematics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “We train with mixed precision ﬂoating point arithmetic on DGX-1 machines, each with 8 × 32GB Nvidia V100 GPUs interconnected by In-ﬁniband (Micikevicius et al., 2018).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Mixed-precision (Micikevicius et al., 2017) was used to accelerate training and save memory.”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “To train our models efﬁciently we utilize mixed precision training with dynamic loss scaling to take advantage of the V100’s Tensor Cores (Micikevicius et al., 2017; NVIDIA, 2018).”
Generating Long Sequences with Sparse Transformers (2019, methodology): “We store network weights in single-precision ﬂoating-point, but otherwise compute network activations and gradients in half-precision, as in (Micikevicius et al., 2017).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “FAIRSEQ implements dynamic loss scaling (Micikevicius et al., 2018) in order to avoid underflows for activations and gradients because of the limited precision offered by FP16.”
Scaling Neural Machine Translation (2018, background): “This however requires scaling the loss to achieve comparable model quality (Narang et al., 2018).”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “Mixed precision training [41] typically stores weights and gradients in half precision formats (i.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “FP16 Training: Typical mixed-precision training recipes require storing model weights in both 16-bit (FP16) and 32-bit (FP32), as well as storing optimizer state in 32-bit to preserve accurate weight updates (Micikevicius et al., 2018; Ott et al., 2018).”
ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, methodology): “en mixed-precision training is applied. Mixed-Precision Training The state-of-the-art approach to train large models on the current generation of NVIDIA GPUs is via mixed precision (fp16/32) training [23], where parameters and activations are stored as fp16, enabling the use of the high throughput t”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “We use a dropout of 0.1 (including embedding dropout) and clip gradient norms to 1.0, and use dynamic loss scaling to prevent underflows (Micikevicius et al., 2018).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “On the other hand, bfloat16 still has much lower precision, which motivated our use of mixed-precision training (Micikevicius et al., 2018).”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “To avoid under-ﬂows, we used dynamic loss scaling, as described in Micikevicius et al. (2017).”
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023, methodology): “In terms of parameter management, it adheres to the standard mixed precision technique, which maintains both low and full precision copies of parameters [17].”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “We implement our models using PyTorch [97] with native automatic mixed-precision [98].”
Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “[19] proposes to store all parameters and activations in FP16 and use FP16 to compute gradients.”
Zero-Shot Text-to-Image Generation (2021, background): “Standard loss scaling (Micikevicius et al., 2017) is able to avoid underflow when the range spanned by the smallest and largest activation gradients (in absolute value) fits within the exponent range of the 16-bit format.”
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019, methodology): “We trained with mixed precision floating point arithmetic on two DGX1 machines, each with eight 16GB Nvidia V100 GPUs interconnected by Infiniband (Micikevicius et al., 2018).”