DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

By Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, Yuxiong He. 2020

2,160 citations recorded by the source corpus.

Explore new techniques in Microsoft's open source library called DeepSpeed, which advances large model training by improving scale, speed, cost, and usability, unlocking the ability to train 100-billion-parameter models. DeepSpeed is compatible with PyTorch. One piece of our library, called ZeRO, is a new parallelized optimizer that greatly reduces the resources needed for model and data parallelism while massively increasing the number of parameters that can be trained. Researchers have used these breakthroughs to create Turing Natural Language Generation (Turing-NLG), which at the time of its release was the largest publicly known language model at 17 billion parameters. In addition we will also go over our latest transformer kernel advancements that led the DeepSpeed team to achieve the world fastest BERT pretraining record. The Zero Redundancy Optimizer (ZeRO) is a novel memory optimization technology for large-scale distributed deep learning. ZeRO can train deep learning models with over 100 billion parameters on the current generation of GPU clusters at three to five times the throughput of the current best system. It also presents a clear path to training models with trillions of parameters, demonstrating an unprecedented leap in deep learning system technology. DeepSpeed brings state-of-the-art training techniques, such as ZeRO, optimized kernels, distributed training, mixed precision, and checkpointing, through lightweight APIs compatible with PyTorch. With just a few lines of code changes to your PyTorch model, you can leverage DeepSpeed to address underlying performance challenges and boost the speed and scale of your training.

Subfields: Artificial Intelligence, Computer Networks and Communications, Hardware and Architecture, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Hardware and Architecture

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “Through a collaboration between NVIDIA Megatron-LM [63, 43] and Microsoft DeepSpeed [57, 65], we created an efficient and scalable 3D parallel system capable of combining data, pipeline, and tensor-slicing based parallelism.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Recent developments in model parallelism infrastructure (Shazeer et al., 2018b; Huang et al., 2019; Xu et al., 2021; Shoeybi et al., 2019; Rasley et al., 2020) allow efficient training of models up to trillions of parameters.”
Scaling Up Models and Data with t5x and seqio (2022, background): “, 2021), and libraries that include both kinds of functionality such as DeepSpeed (Rasley et al., 2020) and Megatron (Smith et al.”
Pathways: Asynchronous Distributed Dataflow for ML (2022, methodology): “System designers have adopted inge-nious techniques to execute pipelined (Narayanan et al., 2021; Rasley et al., 2020; Narayanan et al., 2019; Huang et al., 2019) and homogeneous MoE (Lepikhin et al., 2020; Fedus et al., 2021) models on MPI-style systems, but as we argue in detail later, the MPI…”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “It consists of two parts: Megatron-LM21 (Shoeybi et al., 2019) provides the Transformer implementation, tensor parallelism, and data loading primitives, whereas DeepSpeed22 (Rasley et al., 2020) provides the ZeRO optimizer, model pipelining, and general distributed training components.”
Reducing Activation Recomputation in Large Transformer Models (2022, methodology): “An alternative to model parallelism is to combine a number of training techniques along with data parallelism to enable large scale model training [14, 17, 15, 16].”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “, 2020) and DeepSpeed (Rasley et al., 2020) to facilitate efficient and straightforward training of large language models with tens of billions of parameters.”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “In addition, significant research has gone into algorithmic approaches to efficiently scale training [59], [60] by reducing communication cost via compression [61], [62], pipelining [63], and sharding [64], [65].”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “All models are trained using DeepSpeed’s ZeRO-3 (Rasley et al., 2020).”
PEER: A Collaborative Language Model (2022, methodology): “We use DeepSpeed (Rasley et al., 2020) to enable more efficient multi-GPU training.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “2021) implementation from DeepSpeed (Rasley et al., 2020) to train GLM-130B with a relative big global batch size (4,224) to reduce time and GPU memory wasting.”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, methodology): “GPT-3-style Models All GPT-3-style models used in the paper are trained using DeepSpeed [52] and Megatron-DeepSpeed Library 3 .”
Measuring Coding Challenge Competence With APPS (2021, methodology): “We use DeepSpeed and its implementation of the ZeRO optimizer to reduce memory consumption while training large models (Rasley et al., 2020; Rajbhandari et al., 2020).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “To handle redundant activation states, gradient check pointing methods (Rasley et al., 2020) have been used to save memory by storing only a part of the activation states after forward pass.”
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021, methodology): “Our pretraining implementation is based on Megatron-LM (Shoeybi et al., 2019) and Deep-Speed (Rasley et al., 2020).”
Multi-lingual Evaluation of Code Generation Models (2022, background): “Our training pipeline is based on PyTorch Lightning § and we use bfloat16 (Kalamkar et al., 2019) and DeepSpeed (Rasley et al., 2020) for training optimization.”