Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
By Chi-Chung Chen, Chia-Lin Yang, Hsiang-Yun Cheng. 2018
114 citations recorded by the source corpus.
The training process of Deep Neural Network (DNN) is compute-intensive, often taking days to weeks to train a DNN model. Therefore, parallel execution of DNN training on GPUs is a widely adopted approach to speed up the process nowadays. Due to the implementation simplicity, data parallelism is currently the most commonly used parallelization method. Nonetheless, data parallelism suffers from excessive inter-GPU communication overhead due to frequent weight synchronization among GPUs. Another approach is pipelined model parallelism, which partitions a DNN model among GPUs, and processes multiple mini-batches concurrently. This approach can significantly reduce inter-GPU communication cost compared to data parallelism. However, pipelined model parallelism faces the weight staleness issue; that is, gradients are computed with stale weights, leading to training instability and accuracy loss. In this paper, we present a pipelined model parallel execution method that enables high GPU utilization while maintaining robust training accuracy via a novel weight prediction technique, SpecTrain. Experimental results show that our proposal achieves up to 8.91x speedup compared to data parallelism on a 4-GPU platform while maintaining comparable model accuracy.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohammad Shoeybi
- Bryan Catanzaro
- M. Patwary
- Raul Puri
- Vitaliy Chiley
- Atli Kosson
- Fei Wu
- Nguyen T. Nguyen
In Computer Vision and Pattern Recognition
- Mohammad Shoeybi
- Joel Hestness
- Raul Puri
- M. Patwary
- Fei Wu
- Vitaliy Chiley
- Atli Kosson
- Nguyen T. Nguyen
In Optimization & Training
- Mohammad Shoeybi
- Bryan Catanzaro
- M. Patwary
- Vitaliy Chiley
- Atli Kosson
- Fei Wu
- Nguyen T. Nguyen
- Abhinav Venigalla
Citations in the corpus
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “Some approaches (Harlap et al., 2018; Chen et al., 2018) use a parameter server (Li et al., 2014) in conjunction with pipeline parallelism.”
- DAPPLE: a pipelined data parallel approach for training large models (2020, methodology): “[53], [55], [56] make further optimization based on PipeDream.”
- Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training (2021, background): “The works in (Chen et al., 2018) and (Narayanan et al.”
- Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, methodology): “SpecTrain (Chen et al., 2018) uses a form of weight prediction to mitigate both stale gradients and inconsistent weights.”
- HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “A weight prediction technique is proposed to address the staleness issue in pipelined model parallelism [5].”