Efficient Large-Scale Language Model Training on GPU Clusters

By D. Narayanan, Mohammad Shoeybi, J. Casper, Patrick LeGresley, M. Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro and 3 more. 2021

108 citations recorded by the source corpus.

Subfields: Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Efficient Large Scale Language Modeling with Mixtures of Experts (2021, positive): “We count the number of floating-point operations (FLOPs) analytically following Narayanan et al. (2021).”
The Efficiency Misnomer (2021, methodology): “FLOPs: A widely used metric as the proxy for the computational cost of a model is the number of ﬂoating-point multiplication-and-addition operations (Johnson, 2018; Kim et al., 2021; Arnab et al., 2021; Tay et al., 2021a;b; Narayanan et al., 2021; Liu et al., 2021).”
Pre-Trained Models: Past, Present and Future (2021, background): “Megatron-LM (Shoeybi et al., 2019; Narayanan et al., 2021) and Deep Speed (Rajbhandari et al., 2020, 2021) target at training large-scale NLP PTMs. Insight Face (ins, 2021) trains large-scale face recognition models.”
Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training (2021, methodology): “Megatron-LM (Shoeybi et al., 2019) created a tensor-parallel implementation of GPT-2 and T5 based on PyTorch, and added pipeline parallelism in later work (Narayanan et al., 2021b).”
FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models (2022, methodology): “Megatron-LM [20] is a dedicated training system for pre-training, with an insightful model parallelism method for transformer blocks.”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, background): “PTD-P [34] scheduled with interleaved stages in a fine-grained way to reduce the size of the pipeline bubble further.”
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining (2021, background): “The scale of foundation models across domains, including natural language processing, computer vision, and cross-modality representation learning, have been growing tremendously from millions to trillions of parameters [9; 26; 27; 29; 36; 40; 4; 32; 17; 13; 19; 50; 49; 45; 47; 41] thanks to the conc”
Software-hardware co-design for fast and scalable training of deep learning recommendation models (2021, background): “Frameworks that support combinations of these strategies are generally designed for specific DNN applications [16, 22, 41, 50].”
Scalable and Efficient MoE Training for Multitask Multilingual Models (2021, methodology): “For example, the GPT-3 175B parameter model takes 168 days to train on 256 NVIDIA A100 GPUs based on the latest optimizations (Narayanan et al., 2021), while a model with a trillion parameters will take over a year even with 512 NVIDIA A100 GPUs, making these models prohibitively expensive.”
Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models (2022, background): “recent years [35]: the Megatron-Turing natural language generation model [37] with a dense architecture reaches 530 billion parameters; the recent GLaM [20] with a sparse mixture-of-experts architecture has 1.”
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch (2021, background): “Megatron-LMs (Shoeybi et al., 2020; Narayanan et al., 2021) and DeepSpeed (dee, 2021; Rajbhandari et al., 2021; 2020) support model parallelism for pre-training large NLP models.”
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass (2025, unknown): “Fast3R predicts local and global pointmaps X L and X G , and corresponding confidence maps Σ L and Σ G (of shape Like MASt3R, the global pointmap X G is in the coordinate frame of the first camera and the X L is in the coordinate frame of the viewing camera, as shown in Figure 2 3”