DAPPLE: a pipelined data parallel approach for training large models

By Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia and 3 more. 2020

336 citations recorded by the source corpus.

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are still several tricky issues to address: improving computing efficiency while ensuring convergence, and reducing memory usage without incurring additional computing costs. We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategies of data and pipeline parallelism. We also propose a new runtime scheduling algorithm to reduce device memory usage, which is orthogonal to re-computation approach and does not come at the expense of training throughput. Experiments show that DAPPLE planner consistently outperforms strategies generated by PipeDream's planner by up to 3.23× speedup under synchronous training scenarios, and DAPPLE runtime outperforms GPipe by 1.6× speedup of training throughput and saves 12% of memory consumption at the same time.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Hardware and Architecture, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Hardware and Architecture

In Efficiency & Compression

Citations in the corpus

GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, methodology): “Pipelining algorithms [13, 15, 22, 24, 36] focus on one type of parallelism, while GSPMD can be used either to express similar ideas with the help of vectorization (Section 3.”
PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “The second regime is pipeline model parallelism [24, 25, 26, 27, 28], which partitions the total layers to stages, and then places stages to different devices, as shown in Figure 4(c).”
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, methodology): “[41], and DAPPLE [14]), but instead suggest heuristics (in §3) that we found work well in practice.”
Memory-Efficient Pipeline-Parallel DNN Training (2020, methodology): “PipeDream, DAPPLE (Fan et al., 2021), and FlexFlow (Jia et al., 2018) use planning algorithms to partition operator graphs over multiple accelerators to maximize throughput.”
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models (2021, background): “, gradient accumulation (Fan et al., 2020), rematerialization (Chen et al.”
This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. (neutral): “With this conventional view, prior explorations [17, 25, 55, 60] of auto-parallelization are limited to combining data parallelism with at most one model parallelism approach, which misses substantial performance opportunities.”
FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models (2022, background): “Another approach to save memory is pipeline parallelism [19], and it can be mixed with data parallelism [4].”