Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

By Shaden Smith, M. Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Anand Korthikanti and 10 more. 2022

859 citations recorded by the source corpus.

Pretrained general-purpose language models can achieve state-of-the-art accuracies in various natural language processing domains by adapting to downstream tasks via zero-shot, few-shot and fine-tuning techniques. Because of their success, the size of these models has increased rapidly, requiring high-performance hardware, software, and algorithmic techniques to enable training such large models. As the result of a joint effort between Microsoft and NVIDIA, we present details on the training of the largest monolithic transformer based language model, Megatron-Turing NLG 530B (MT-NLG), with 530 billion parameters. In this paper, we first focus on the infrastructure as well as the 3D parallelism methodology used to train this model using DeepSpeed and Megatron. Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a key ingredient to the success of the model. Finally, we discuss various evaluation results, as well as other interesting observations and new properties exhibited by MT-NLG. We demonstrate that MT-NLG achieves superior zero-, one-, and few-shot learning accuracies on several NLP benchmarks and establishes new state-of-the-art results. We believe that our contributions will help further the development of large-scale training infrastructures, large-scale language models, and natural language generations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “, 2021) 280 Billion 300 Billion MT-NLG 530B (Smith et al., 2022) 530 Billion 270 Billion Chinchilla 70 Billion 1.”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Megatron-Turing NLG 530B (Smith et al., 2022) was trained on 2240 A100 GPUs using a combination of model, data, and pipeline parallelism, and Gopher (Rae et al.”
Solving Quantitative Reasoning Problems with Language Models (2022, background): “In particular, large language models have achieved excellent performance across a variety of natural language tasks including common-sense reasoning, question answering, and summarization [1, 2, 3, 4, 5].”
Scaling Up Models and Data with t5x and seqio (2022, background): “…from other research groups include model libraries such as fairseq (Ott et al., 2019), large-scale parallelism libraries such as FairScale (Baines et al., 2021), and libraries that include both kinds of functionality such as DeepSpeed (Rasley et al., 2020) and Megatron (Smith et al., 2022).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Ultimately, we performed final training in bfloat16 mixed precision, which proved to solve the instability problem (in line with previous observation by Smith et al., 2022).”
Reducing Activation Recomputation in Large Transformer Models (2022, methodology): “To quantify this, let’s consider GPT-3 [4] and MT-NLG [20] models, some of the largest models that have been trained so far.”
Efficiently Scaling Transformer Inference (2022, neutral): “We benchmark the Megatron 530B model (Smith et al., 2022) and the similarly-sized PaLM 540B model, which has architectural optimizations including multiquery attention and parallel attention/feedforward layers (full list of differences in Table F.1).”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…a surge of research and development in autoregressive language models (Radford et al., 2019; Lewis et al., 2019; Raffel et al., 2019; Brown et al., 2020; Rae et al., 2021; Artetxe et al., 2021; Smith et al., 2022), with ever increasing size and natural language understanding (NLU) capabilities.”
Large Language Models are Zero-Shot Reasoners (2022, background): “…GPT-3, etc Reasoning Ability of LLMs Several studies have shown that pre-trained models usually are not good at reasoning [Brown et al., 2020, Smith et al., 2022, Rae et al., 2021], but its ability can be substantially increased by making them produce step-by-step reasoning, either by…”
Predictability and Surprise in Large Generative Models (2022, background): “0 Titan [81], FLAN [82], Gopher [62], GPT-3 [11], HyperClova [46], Jurassic-1-Jumbo [51], Megatron Turing NLG [72], LaMDA [78], Pan Gu [89], Yuan 1 .”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Later, despite many 100B-scale LLMs (Lieber et al., 2021; Thoppilan et al., 2022; Rae et al., 2021; Smith et al., 2022; Chowdhery et al., 2022; Wu et al., 2021; Zeng et al., 2021; Wang et al., 2021) in both English and Chinese, they are not available to public or only accessible via limited APIs.”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “More importantly they need retraining or ﬁnetuning the full model to recover the accuracy, and such compute cost on extra-large models, like [57, 11], can be hardly aﬀordable for most research labs or practitioners.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, methodology): “We do not take into account memory use: typical use cases are compute-bound by the available GPU/TPU-hours.”