GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

By Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Z. Chen. 2018

2,195 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “…research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise [Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b].”
Scaling Laws for Neural Language Models (2020, methodology): “Deep models can be trained using pipelining [HCC + 18], which splits parameters depth-wise between devices, but eventually requires increased batch sizes as more devices are used.”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “, 2018) or alternatively separating layers of the models across accelerators and then pipe-lining activations between the stages (Huang et al., 2019).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “For example, GPipe (Huang et al., 2018) and Mesh-Tensorﬂow (Shazeer et al., 2018) provide frameworks for model parallelism of different kinds.”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “This is further exacerbated by the increasing model capacity of state-of-the-art NMT models (Shazeer et al., 2018; Bapna et al., 2018; Huang et al., 2018); full fine-tuning is just too parameter inefficient.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Other than the systems and engineering challenges (Jouppi et al., 2017; Huang et al., 2018; Shazeer et al., 2018), training deep and high capacity neural networks poses signiﬁcant trainabil-ity challenges (Montavon et al., 2018).”
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “‡ DAT denotes domain adaptive transfer learning (Ngiam et al., 2018).”
Unified Scaling Laws for Routed Language Models (2022, neutral): “There are still more variations of model parallelism, notably Pipeline Parallelism [Narayanan et al., 2019, Huang et al., 2019], which we do not use.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “Pipeline Model Parallelism Pipeline model parallelism (or, pipeline parallelism ) divides the layers of the model into stages that can be processed in parallel [23, 42].”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “…et al., 2020), enabling training of models with more than one trillion parameters using only data parallelism without the added complexity introduced by model parallel training approaches like pipeline or tensor parallelism (Huang et al., 2019; Shoeybi et al., 2019; Narayanan et al., 2021).”
ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, background): “When a model does not fit in the device memory, model parallelism (MP) [7], [4], [11] and pipeline parallelism (PP) [12], [13] split the model among processes, in vertical and horizontal way respectively.”
Pathways: Asynchronous Distributed Dataflow for ML (2022, methodology): “System designers have adopted inge-nious techniques to execute pipelined (Narayanan et al., 2021; Rasley et al., 2020; Narayanan et al., 2019; Huang et al., 2019) and homogeneous MoE (Lepikhin et al., 2020; Fedus et al., 2021) models on MPI-style systems, but as we argue in detail later, the MPI…”
GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, background): “…buffer to pass data across stages, additional stages - 1 iterations to wait for the last stage to are equivalent to the bubbles in earlier work [18] that describe With the help of vmap or vectorized_map , this loop implementation OneStageCompute and turn it to a pipelined computation.…”
PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “The second regime is pipeline model parallelism [24, 25, 26, 27, 28], which partitions the total layers to stages, and then places stages to different devices, as shown in Figure 4(c).”
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, methodology): “Activation recomputation [12, 18, 20, 21] is an optional technique that trades off an increase in the number of compute operations performed for additional memory footprint, by running the forward pass a second time just before the backward pass (and stashing only the input activations for a given p”
ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “To be more specific, distributed parallel DL training technologies such as pipeline parallelism [10], model parallelism [28], and ZeRO [21] (Zero Redundancy Optimizer) allow transcending the memory boundaries of single GPU/accelerator device by splitting the model states (parameters, gradients and o”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “To that effect, we leverage progress in scaling (Kaplan et al., 2020; Arora et al., 2018) to train models that are over 50 times larger than current bilingual models with model parallelism (Huang et al., 2019; Shoeybi et al., 2019).”