PipeMare: Asynchronous Pipeline Parallel DNN Training

By Bowen Yang, Jian Zhang, Christopher R'e, Christopher R. Aberger, Christopher De Sa. 2019

136 citations recorded by the source corpus.

Recently there has been a flurry of interest around using pipeline parallelism while training neural networks. Pipeline parallelism enables larger models to be partitioned spatially across chips and within a chip, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve statistical efficiency, existing pipeline-parallelism techniques sacrifice hardware efficiency by introducing bubbles into the pipeline and/or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices are necessary. Theoretically, we derive a simple but robust training method, called PipeMare, that tolerates asynchronous updates during pipeline-parallel execution. Using this, we show empirically, on a ResNet network and a Transformer network, that PipeMare can achieve final model qualities that match those of synchronous training techniques (at most 0.9% worse test accuracy and 0.3 better test BLEU score) while either using up to 2.0X less weight and optimizer memory or being up to 3.3X faster than other pipeline parallel training techniques. To the best of our knowledge we are the first to explore these techniques and fine-grained pipeline parallelism (e.g. the number of pipeline stages equals to the number of layers) during neural network training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “For example, several researchers might concurrently ﬁne-tune (Houlsby et al., 2019; Zhang et al., 2021) a foundation model for different tasks, using the same accelerators to hold the ﬁxed foundation model layers.”
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, methodology): “Pipeline parallelism can also be implemented with relaxed semantics: PipeDream-2BW [30] maintains two weight versions and guarantees 1-stale weight updates without expensive flushes, while PipeMare [45] and Kosson et al. [23] use asynchoronous pipeline parallelism.”
Memory-Efficient Pipeline-Parallel DNN Training (2020, methodology): “PipeMare (Yang et al., 2019) uses asynchronous pipeline parallelism to provide high throughput (no pipeline ﬂushes) with asynchronous weight update semantics.”
DAPPLE: a pipelined data parallel approach for training large models (2020, neutral): “Model Parallelism [50] partitions DNN models among GPUs to mitigate communication overhead and memory bottlenecks for distributed training [10], [14], [39], [40], [51]–[54].”
Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models (2022, methodology): “Pipeline model parallelism [24, 29, 33, 34, 53] is another technique that have been proposed for large deep learning models to address the scaling challenges.”
Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, methodology): “Discrepancy correction (Yang et al., 2019) can be seen as a form of weight prediction.”
vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training (2022, methodology): “Existing pipeline parallel systems [14], [33], [59] carefully designed various strategies to orchestrate (add or reduce) the staleness on parameter updates for higher training accuracy or through-put on speciﬁc DNNs.”