Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
By Mohammad Shoeybi, M. Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro. 2019
2,986 citations recorded by the source corpus.
Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).
Subfields: Artificial Intelligence, Efficiency & Compression, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Yoshua Bengio
- Yejin Choi
In Efficiency & Compression
- Kurt Keutzer
- Song Han
- Dan Alistarh
- Zi-Mei Yao
- Luke S. Zettlemoyer
- A. Gholami
- Michael W. Mahoney
- Tri Dao
In Optimization & Training
- Quoc V. Le
- Yoshua Bengio
- Torsten Hoefler
- Tal Ben-Nun
- Frank Hutter
- Michael W. Mahoney
- Luke S. Zettlemoyer
- Kurt Keutzer
Citations in the corpus
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “, 2019), MegatronLM (Shoeybi et al., 2019), and T5 (Raffel et al.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “To optimize for latency, we host our largest models using 8 A100s in a single node with tensor parallelism (Shoeybi et al., 2019).”
- Experience Grounds Language (2020, background): “For example, in the LAM-BADA dataset (Paperno et al., 2016) (designed to capture human intuition), GPT2 (Radford et al., 2019) (1.5B), Megatron-LM (Shoeybi et al., 2019) (8.3B), and TuringNLG (Rosset, 2020) (17B) perform within a few points of each other and very far from perfect ( 68%).”
- Adversarial Training for Large Neural Language Models (2020, methodology): “The training code is based on Megatron, implemented in PyTorch (Shoeybi et al., 2019) 4 .”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “In fact, the challenge in the MLM task is not overfitting , but instead, it is fitting the data—even 8 billion parameter models do not overfit to large pretraining corpora (Shoeybi et al., 2019).”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “Existing solutions to the aforementioned problems include model parallelization (Shoeybi et al., 2019) and clever memory management (Chen et al., 2016; Gomez et al., 2017).”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “Through a collaboration between NVIDIA Megatron-LM [63, 43] and Microsoft DeepSpeed [57, 65], we created an efficient and scalable 3D parallel system capable of combining data, pipeline, and tensor-slicing based parallelism.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “…et al., 2020), enabling training of models with more than one trillion parameters using only data parallelism without the added complexity introduced by model parallel training approaches like pipeline or tensor parallelism (Huang et al., 2019; Shoeybi et al., 2019; Narayanan et al., 2021).”
- Compressive Transformers for Long-Range Sequence Modelling (2019, background): “…about a step-change in state-of-the-art performance, within machine translation (Vaswani et al., 2017), language modelling (Dai et al., 2019; Shoeybi et al., 2019), video captioning (Zhou et al., 2018), and a multitude of language understanding benchmarks (Devlin et al., 2018; Yang et al.,…”
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, methodology): “To share concrete insights, we perform the communication volume analysis in the context of transformer based models implemented using SOTA MP approach, Megatron-LM.”
- Scaling Up Models and Data with t5x and seqio (2022, background): “These options correspond to previously described parallelism techniques: 2D parameter partitioning is also known as ZeRO-3 (Rajbhandari et al., 2020) or fully sharded data parallelism; 1D activation partitioning is also known as Megatron (Shoeybi et al., 2019) and is the default in the Mesh TensorFl”
- GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, methodology): “This sharding configuration combines the benefit of data parallelism, weight-update sharding [30, 40] and in-layer model parallelism [36], and only requires 7 annotations in the model.”