PipeDream: Fast and Efficient Pipeline Parallel DNN Training

By Aaron Harlap, D. Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, G. R. Ganger, Phillip B. Gibbons. 2018

317 citations recorded by the source corpus.

PipeDream is a Deep Neural Network(DNN) training system for GPUs that parallelizes computation by pipelining execution across multiple machines. Its pipeline parallel computing model avoids the slowdowns faced by data-parallel training when large models and/or limited network bandwidth induce high communication-to-computation ratios. PipeDream reduces communication by up to 95% for large DNNs relative to data-parallel training, and allows perfect overlap of communication and computation. PipeDream keeps all available GPUs productive by systematically partitioning DNN layers among them to balance work and minimize communication, versions model parameters for backward pass correctness, and schedules the forward and backward passes of different inputs in round-robin fashion to optimize "time to target accuracy". Experiments with five different DNNs on two different clusters show that PipeDream is up to 5x faster in time-to-accuracy compared to data-parallel training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, background): “When a model does not fit in the device memory, model parallelism (MP) [7], [4], [11] and pipeline parallelism (PP) [12], [13] split the model among processes, in vertical and horizontal way respectively.”
ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “Two prominent examples of scale out training is model parallelism [5, 28] and pipeline parallelism [7,10], both partitioning the model states and the residual states across multiple GPUs.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “For graph-level partitioning, sophisticated algorithms [15, 24] are needed to reduce the overhead”
Memory-Efficient Pipeline-Parallel DNN Training (2020, background): “For example, GPipe (Huang et al., 2019) and PipeDream (Harlap et al., 2018; Narayanan et al., 2019) push multiple inputs in sequence through a series of workers that each manage one model partition, allowing different workers to process different inputs in parallel.”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “[2019] in combination with pipeline parallelism [Harlap et al., 2018] to distribute the model across GPUs.”
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023, background): “Pipeline parallelism [4, 7] involves partitioning model parameters and their activations across multiple devices through the division of models into pipeline stages.”
A Review of Sparse Expert Models in Deep Learning (2022, background): “…include data parallelism (model weights replicated, but data sharded), tensor model-parallelism (Shazeer et al., 2018) (data and weight tensors are split across devices), and pipeline parallelism (Harlap et al., 2018; Huang et al., 2019) (entire layers or groups of layers are split across devices).”
8-bit Optimizers via Block-wise Quantization (2021, background): “Such approaches are model parallelism (Krizhevsky et al., 2009), pipeline parallelism (Krizhevsky et al., 2009; Huang et al., 2018; Harlap et al., 2018), and operator parallelism (Lepikhin et al., 2020).”
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models (2021, methodology): “Harlap et al. (2018) propose using an asynchronous training algorithm to mitigate the effect of pipeline bubbles in microbach-based pipeline parallel training, but asynchronous training introduces uncertainty in model accuracy and is thus not widely adopted for training DNNs. Wavefront parallelism…”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, methodology): “Furthermore, [13] may suffer from staleness of the weight update, which is linear in the number of stages.”
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale (2022, methodology): “Pipeline parallelism [17], [18], [19] splits a model vertically into pipeline stages and use micro-batching to hide pipeline bubbles.”
DAPPLE: a pipelined data parallel approach for training large models (2020, background): “Model Parallelism [50] partitions DNN models among GPUs to mitigate communication overhead and memory bottlenecks for distributed training [10], [14], [39], [40], [51]–[54].”
Training Large Neural Networks with Constant Memory using a New Execution Algorithm (2020, background): “GPipe and PipeDream both have overheads related during the start of the pipeline, and both approaches require the number of devices to scale with the model depth and not just the layer size.”
PipeMare: Asynchronous Pipeline Parallel DNN Training (2019, methodology): “PipeDream (Harlap et al., 2018) is a pipeline parallel distributed training technique used to reduce high communication-to-computation ratios.”
Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, methodology): “If weight stashing (Harlap et al., 2018) is used to mitigate weight inconsistency the resulting update is: Weight stashing requires the overhead of storing parameter versions along with the activations.”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, background): “[61] design PipeDream, a system that supports pipelined training and automatically determines how to systematically split a given model across the available computing nodes.”