Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

By Sina Lin, Charles Ruizhongtai Qi, A. Aiken. 2018

131 citations recorded by the source corpus.

The past few years have witnessed growth in the size and computational requirements for training deep convolutional neural networks. Current approaches parallelize the training process onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, this design results in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an individual parallelization strategy. We jointly optimize how each layer is parallelized by solving a graph search problem. Our experiments show that layer-wise parallelism outperforms current parallelization approaches by increasing training speed, reducing communication costs, achieving better scalability to multiple GPUs, while maintaining the same network accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Mesh-TensorFlow: Deep Learning for Supercomputers (2018, methodology): “[20, 19] implemented a framework that uses cost modeling to pick the best parallelization strategy, including how to partition work for each operation.”
Beyond Data and Model Parallelism for Deep Neural Networks (2018, methodology): “However, as reported in (Jia et al., 2018), parallelizing an operator through different dimensions can result in dif- ferent task computation time.”
Supporting Very Large Models using Automatic Dataflow Graph Partitioning (2018, methodology): “Prior work [13–15] manually discovers the partition strategies for a few common DNN layers.”
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping (2020, background): “[22, 23, 50] propose to auto-mate the model parallelism and reduce the communication with dataflow graph analysis.”
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform (2018, background): “Jia et al. (Jia et al., 2019; 2018) partition the tensor along multiple dimensions and search for best parallelization strategy.”
MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud (2022, methodology): “FlexFlow [15] and OptCNN [39] use heuristic search for parallel strategies including tensor MP and device placement MP.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “Besides data parallelism, other parallelization strategies include model parallelism [9, 11], and hybrids of model and data parallelism [26, 27].”
AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators (2020, methodology): “Recursive methods [74,118] are proposed for tensor partitioning on multiple devices and dynamic programming methods [116,118] are proposed for tensor partitioning layer-wisely.”
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch (2021, methodology): “OptCNN (Jia et al., 2018) parallelizes CNN model training by splitting operations along batch and channel dimensions on homogeneous devices.”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “In running FlexFlow for AlexNet on Cluster-B (not shown in Figure 14b), we observe that PipeDream is 1 .”