vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training

By Shixiong Zhao, Fanxin Li, Xusheng Chen, Xiuxian Guan, Jianyu Jiang, Dong Huang, Yuhao Qing, Sen Wang, Peng Wang, Gong Zhang and 3 more. 2022

52 citations recorded by the source corpus.

The increasing computational complexity of DNNs achieved unprecedented successes in various areas such as machine vision and natural language processing (NLP), e.g., the recent advanced Transformer has billions of parameters. However, as large-scale DNNs significantly exceed GPU’s physical memory limit, they cannot be trained by conventional methods such as data parallelism. Pipeline parallelism that partitions a large DNN into small subnets and trains them on different GPUs is a plausible solution. Unfortunately, the layer partitioning and memory management in existing pipeline parallel systems are fixed during training, making them easily impeded by out-of-memory errors and the GPU under-utilization. These drawbacks amplify when performing neural architecture search (NAS) such as the evolved Transformer, where different network architectures of Transformer needed to be trained repeatedly. vPipe is the first system that transparently provides dynamic layer partitioning and memory management for pipeline parallelism. vPipe has two unique contributions, including (1) an online algorithm for searching a near-optimal layer partitioning and memory management plan, and (2) a live layer migration protocol for re-balancing the layer distribution across a training pipeline. vPipe improved the training throughput of two notable baselines (Pipedream and GPipe) by 61.4-463.4 percent and 24.8-291.3 percent on various large DNNs and training settings.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Neurology, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Quoc V. Le
3. Dacheng Tao
4. Yi Yang
5. Andrew Zisserman
6. Jitendra Malik
7. Xiaogang Wang
8. Ming-Hsuan Yang

In Electrical and Electronic Engineering

1. Ruoming Pang

In Neurology

1. Ruoming Pang

In Efficiency & Compression

1. Yonghui Wu
2. Joseph Dean
3. Aakanksha Chowdhery
4. Ruoming Pang
5. Hyeontaek Lim
6. Quoc V. Le
7. Dacheng Tao
8. Mingxing Tan

In Optimization & Training

1. Michael Isard
2. Aakanksha Chowdhery
3. Ruoming Pang
4. Ryan Sepassi
5. Yonghui Wu
6. Joseph Dean
7. Hyeontaek Lim

Citations in the corpus

• Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “To increase utilization, some ML hardware resource management researchers (Xiao et al., 2020; Bai et al., 2020; Yu and Chowdhury, 2020; Wang et al., 2021; Lim et al., 2021; Zhao et al., 2022; Weng et al., 2022) a r X i v : 2203 .”