On Model Parallelization and Scheduling Strategies for Distributed Machine Learning

By Seunghak Lee, Jin Kyu Kim, Xun Zheng, Qirong Ho, Garth A. Gibson, Eric P. Xing. 2014

129 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Eric P. Xing
2. Quoc V. Le
3. Willie Neiswanger
4. Ruslan Salakhutdinov
5. Sergey Levine
6. Yanping Huang
7. Mohit Bansal
8. Z. Chen

In Computer Vision and Pattern Recognition

1. Quoc V. Le
2. Eric P. Xing
3. Luc Van Gool
4. Xiaogang Wang
5. Dacheng Tao
6. Ming-Hsuan Yang
7. Yi Yang
8. Trevor Darrell

In Hardware and Architecture

1. Quoc V. Le
2. Wei Dai
3. Eric P. Xing
4. Qirong Ho
5. Jiquan Ngiam
6. Yanping Huang
7. Ion Stoica
8. Z. Chen

Citations in the corpus

• Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some relevant references are Lee et al. (2014); Ma and Takáč (2015); Smith et al. (2016); Chen et al. (2016b); Fang and Klabjan (2018).”
• HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “For training large DNN models, data parallelism [4, 28, 29, 45], which employs multiple workers using parameter servers or AllReduce communication, and model parallelism [10,25,27], which divides the network layers of a DNN model into multiple partitions and assigns each partition to a different GPU”
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (2018, background): “The core idea of model parallelism involves partitioning a network into different computational units, which are then placed on different devices [39, 40, 41, 42].”
• Parallel and Distributed Block-Coordinate Frank-Wolfe Algorithms (2014, methodology): “The bound is proportional to κ when we allow the delay to grow unboundedly, and proportional to√ κ when the delay is bounded by a small κmax.”
• Strategies and Principles of Distributed Machine Learning on Big Data (2015, background): “This suggests that computation may be better prioritized toward parameters that are still non-zero, by executing ΔLasso more frequently on them—and such a strategy indeed reduces the time taken by the ML program to finish [39].”
• FDML: A Collaborative Machine Learning Framework for Distributed Features (2019, background): “, DistBelief [6] and STRADS [16], which aims to train a big model by letting each worker be responsible for updating a subset of model parameters.”
• Wavelet: Efficient DNN Training with Tick-Tock Scheduling (2021, methodology): “The most popular distributed approaches are data parallelism (Li et al., 2014; Chen et al., 2015) and model parallelism (Dean et al., 2012; Lee et al., 2014).”