HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism
By Jay H. Park, Gyeongchan Yun, Chang Yi, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Sam H. Noh, Young-ri Choi. 2020
174 citations recorded by the source corpus.
Deep Neural Network (DNN) models have continuously been growing in size in order to improve the accuracy and quality of the models. Moreover, for training of large DNN models, the use of heterogeneous GPUs is inevitable due to the short release cycle of new GPU architectures. In this paper, we investigate how to enable training of large DNN models on a heterogeneous GPU cluster that possibly includes whimpy GPUs that, as a standalone, could not be used for training. We present a DNN training system, HetPipe (Heterogeneous Pipeline), that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance. We also propose a novel parameter synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to accommodate both PMP and DP for virtual workers, and provide convergence proof of WSP. Our experimental results on a given heterogeneous setting show that with HetPipe, DNN models converge up to 49% faster compared to the state-of-the-art DP technique.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohammad Shoeybi
- Bryan Catanzaro
- Matei A. Zaharia
- M. Patwary
- D. Narayanan
- Qun Liu
- Amar Phanishayee
- Jared Casper
In Computer Vision and Pattern Recognition
- Matei A. Zaharia
- D. Narayanan
- Mohammad Shoeybi
- Yonghong Tian
- Yaowei Wang
- M. Patwary
- Bryan Catanzaro
- Jared Casper
In Neurology
In Efficiency & Compression
- Mohammad Shoeybi
- Matei A. Zaharia
- Bryan Catanzaro
- M. Patwary
- D. Narayanan
- Qun Liu
- Xin Jiang
- Vijay Anand Korthikanti
Citations in the corpus
- PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “The second regime is pipeline model parallelism [24, 25, 26, 27, 28], which partitions the total layers to stages, and then places stages to different devices, as shown in Figure 4(c).”
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, methodology): “HetPipe [31] uses a combination of pipeline and data parallelism on a set of heterogeneous accelerators.”